scala Spark rdd 错误：java.lang.ClassCastException：无法分配 java.lang.invoke.SerializedLambda 的实例

问题描述 投票：0回答：1

我是 Scala 和 Spark 的初学者。

scala 版本：2.12.10

火花版本：3.0.1

我正在 scala 中尝试一个非常简单的 Spark rdd 函数。

但我收到错误。

(1) 构建.sbt


scalaVersion := "2.12.10"


name := "hello-world"
organization := "ch.epfl.scala"
version := "1.0"


libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
libraryDependencies +=        "org.apache.spark" %% "spark-sql" % "3.0.1" 
libraryDependencies +=        "org.apache.spark" %% "spark-core" % "3.0.1"

(2) Main.scala

import org.apache.spark.sql.SparkSession
object Main extends App {
  println("Hello, World!")

  implicit val spark = SparkSession.builder()
        .master("spark://centos-master:7077")
        // .master("local[*]")
        .appName("spark-api")
        .getOrCreate()


  val inputrdd = spark.sparkContext.parallelize(Seq(("arth",10), ("arth", 20), ("samuel", 60), ("jack", 65)))
      println("inputrdd : ", inputrdd)
  val mapped = inputrdd.mapValues(x => (x, 1))
  println("mapped : ", mapped)
  mapped.collect.foreach(println)
}

(3) 发生错误时

似乎是mapped.collect.foreach(println)部分发生了错误。

(4)错误内容

21/04/17 20:54:19 INFO DAGScheduler: Job 0 failed: collect at Main.scala:16, took 6.083947 s
[error] (run-main-0) org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, 192.168.0.220, executor 0):

java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda 
to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in 
instance of org.apache.spark.rdd.MapPartitionsRDD

[error]         at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
[error]         at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
[error]         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
[error]         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error]         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error]         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] 
[error] Driver stacktrace:
21/04/17 20:54:19 INFO TaskSetManager: Lost task 1.3 in stage 0.0 (TID 6) on 192.168.0.220, executor 0: java.lang.ClassCastException (cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 7]
21/04/17 20:54:19 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
[error] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 7, 192.168.0.220, executor 0): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
[error]         at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
[error]         at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
[error]         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
[error]         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error]         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error]         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] 
[error] Driver stacktrace:
[error]         at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
[error]         at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[error]         at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[error]         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[error]         at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
[error]         at scala.Option.foreach(Option.scala:407)
[error]         at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
[error]         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
[error]         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
[error]         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
[error]         at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
[error]         at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2139)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2164)
[error]         at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
[error]         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[error]         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[error]         at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
[error]         at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
[error]         at Main$.delayedEndpoint$Main$1(Main.scala:16)
[error]         at Main$delayedInit$body.apply(Main.scala:2)
[error]         at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]         at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]         at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]         at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]         at scala.collection.immutable.List.foreach(List.scala:392)
[error]         at scala.App.main(App.scala:80)
[error]         at scala.App.main$(App.scala:78)
[error]         at Main$.main(Main.scala:2)
[error]         at Main.main(Main.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error] Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
[error]         at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2301)
[error]         at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1431)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:502)
[error]         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:460)
[error]         at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
[error]         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error]         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error]         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)

如果我需要更多的库或者代码错误（但它在spark-shell中工作正常）....

如何解决？

scala apache-spark rdd
1个回答
0
投票

您需要将 jar 提交到 Spark，以便您的代码可以在其中运行。 Spark-Shell 在幕后向您隐藏了所有这些。

这个答案提供了更好的细节https://stackoverflow.com/a/28367602/1810962和背景。

您可以使用 bin/spark-submit 作为解决方法，并使用 

--class
、
--jars
和 
--driver-class-path

提供本地类路径
最新问题
© www.soinside.com 2019 - 2024. All rights reserved.