无法在 conda windows 中启动 Spark 会话

问题描述 投票:0回答:1

我无法在 Windows conda 虚拟 python 环境中启动 Spark 会话。但是,几天前我能够成功执行此语句,在本地驱动器中安装最新的 java 版本并将路径保存在系统环境变量中后没有任何问题。

from pyspark.sql import SparkSession 
spark = SparkSession.builder.appName("survival").getOrCreate()

今天执行代码时。我收到以下错误。

Py4JJavaError Traceback (most recent call last)

Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
    at java.base/javax.security.auth.Subject.getSubject(Subject.java:347)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
    at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2416)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2416)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:329)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
    at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
    at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:501)
    at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:485)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
    at py4j.Gateway.invoke(Gateway.java:238)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.base/java.lang.Thread.run(Thread.java:1575)
python apache-spark-sql
1个回答
0
投票

检查Spark和Java版本。某些功能在较新的 Java 版本中可能已被弃用。

这是兼容性矩阵 - https://community.cloudera.com/t5/Community-Articles/Spark-and-Java-versions-Supportability-Matrix/ta-p/383669

最新问题
© www.soinside.com 2019 - 2025. All rights reserved.