O envio do Spark falha com java.lang.NoSuchMethodError: scala.Predef $. $ Conforms () Lscala / Predef $$ menos $ colon $ less;
Estou usando a versão de pré-construção do spark 1.3.1 spark-1.3.1-bin-hadoop2.6.tgz
Exceção no encadeamento "main" java.lang.NoSuchMethodError: scala.Predef $. $ Conforms () Lscala / Predef $ menos $ colon $ less; em org.apache.spark.util.Utils $ .getSystemProperties (Utils.scala: 1418) em org.apache.spark.SparkConf. (SparkConf.scala: 58) em org.apache.spark.SparkConf. (SparkConf.scala: 52) em com.zoho.zbi.Testing.test (Testing.java:43) em com.zoho.zbi.Testing.main (Testing.java:39) Usando o perfil padrão do log4j do Spark: org / apache / spark / log4j- defaults.properties
Estou tentando um aplicativo de demonstração simples para salvar em cassandra
SparkConf batchConf= new SparkConf()
.setSparkHome(sparkHome)
.setJars(jars)
.setAppName(ZohoBIConstants.getAppName("cassandra"))//NO I18N
.setMaster(master).set("spark.cassandra.connection.host", "localhost");
JavaSparkContext sc = new JavaSparkContext(batchConf);
// here we are going to save some data to Cassandra...
List<Person> people = Arrays.asList(
Person.newInstance(1, "John", new Date()),
Person.newInstance(2, "Anna", new Date()),
Person.newInstance(3, "Andrew", new Date())
);
// Person test = Person.newInstance(1, "vini", new Date())''
System.out.println("Inside Java API Demo : "+people);
JavaRDD<Person> rdd = sc.parallelize(people);
System.out.println("Inside Java API Demo rdd : "+rdd);
javaFunctions(rdd).writerBuilder("test", "people", mapToRow(Person.class)).saveToCassandra();
System.out.println("Stopping sc");
sc.stop();
quando eu enviar usando spark enviar seu trabalho
bin/spark-submit --class "abc.efg.Testing" --master spark://xyz:7077 /home/test/target/uber-Cassandra-0.0.1-SNAPSHOT.jar
Aqui está o meu pom
dependências
<dependencies>
<!-- Scala -->
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-compiler</artifactId>
<version>${scala.version}</version>
</dependency>
<!-- END Scala -->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>18.0</version>
</dependency>
<dependency>
<groupId>com.yammer.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<version>3.1.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>2.1.5</version>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20090211</version>
</dependency>
<!-- Cassandra Spark Connector dependency -->
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.10</artifactId>
<version>1.2.0</version>
</dependency>
<!-- Cassandra java Connector dependency -->
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector-java_2.10</artifactId>
<version>1.2.0</version>
</dependency>
<!-- Spark Core dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.3.1</version>
</dependency>
<!-- Spark dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>1.3.1</version>
</dependency>
<!-- Spark dependency -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka_2.10</artifactId>
<version>1.3.1</version>
</dependency>
</dependencies>
e eu construo usando
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<finalName>uber-${project.artifactId}-${project.version}</finalName>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>
mas quando eu envio através do código não está funcionando, qualquer ajuda é muito apreciada .. Eu tentei adicionar scala2.10.4 prop em pom ainda sem sorte
Estou executando no eclipse como executado como aplicativo com todos os mestres, spark home e jars configurados para sparkConf o erro é exibido exatamente no sparkConf
Minha versão do scala é
scala -version Scala code runner versão 2.11.2 - Copyright 2002-2013, LAMP / EPFL
isso tem algo a ver com o problema?
Como mudar para uma versão mais antiga do scala? No documento, diz que o spark1.3.1 suporta as versões do scala 2.10.x, informe-me como corrigir isso