Pyspark 错误：“EMR 7.0.0 中未找到类 org.apache.hadoop.fs.s3a.S3AFileSystem”

Question

我使用的是EMR 7.0.0版本，AWS中有python 3.9、spark 3.5.0、Hadoop 3.3.6。 我收到错误：

  File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 740, in csv
  File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
  File "/usr/local/lib/python3.9/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o45.csv.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

我搜索了解决方案，并建议

org.apache.hadoop:hadoop-aws = 3.2.0

。 然而，它仍然不断失败。构建 Spark 的代码

builder = pyspark.sql.SparkSession.builder.appName("stock-daily-ingestion") \
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.0") \
    .config('spark.sql.shuffle.partitions', 2500) \
    .config("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
    .config("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")
sc = builder.getOrCreate()

有人遇到过这个错误吗？帮助将不胜感激。谢谢。

Answer 1

问题已解决。在spark中运行时，在spark-submit命令中添加以下内容

--conf spark.hadoop.io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec 
--packages org.apache.hadoop:hadoop-aws:3.2.0