s3 hadoop错误：NoSuchMethodError：org.apache.hadoop.metrics2.lib.MetricsRegistry.newCounter

Question

我创建了如下凭据：

    Configuration conf = new Configuration();
    conf.set("fs.s3a.impl", org.apache.hadoop.fs.s3a.S3AFileSystem.class.getName());
    conf.set("fs.s3a.access.key", "ACCESS_ID");
    conf.set("fs.s3a.secret.key", "SECRET VALUE");
    conf.set("fs.s3a.endpoint", "s3.us-east-1.amazonaws.com");
    conf.set("fs.s3a.aws.credentials.provider", SimpleAWSCredentialsProvider.NAME);
    conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());

和路径

 Path path = new Path("s3a://<backet>/<directory>/my.parquet");

[当尝试写入镶木地板数据时：

   try (ParquetWriter<GenericData.Record> writer = AvroParquetWriter.<GenericData.Record>builder(path)
                .withSchema(avroSchema)
                .withConf(conf)
                .withCompressionCodec(SNAPPY)
                .withWriteMode(OVERWRITE)
                .build()) 

并且发生以下错误：

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.metrics2.lib.MetricsRegistry.newCounter(Ljava/lang/String;Ljava/lang/String;J)Lorg/apache/hadoop/metrics2/lib/MutableCounterLong;
at org.apache.hadoop.fs.s3a.S3AInstrumentation.counter(S3AInstrumentation.java:183)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.counter(S3AInstrumentation.java:206)
at org.apache.hadoop.fs.s3a.S3AInstrumentation.<init>(S3AInstrumentation.java:160)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:174)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:209)
at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:266)
at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:489)
at Main.main(Main.java:64)

有我的依赖项：

org.apache.avro-1.8.2, org.apache.hadoop-hadoop-core-1.2.1, org.apache.parquet-parquet-hadoop-1.8.1, org.apache.parquet-parquet-avro -1.8.1, org.apache.hadoop-hadoop-hdfs-3.2.1, org.apache.hadoop-hadoop-aws -2.8.2

您知道如何解决吗？

Answer 1

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.metrics2.lib.MetricsRegistry.newCounter(Ljava/lang/String;Ljava/lang/String;J)Lorg/apache/hadoop/metrics2/lib/MutableCounterLong;

MetricsRegistry类是hadoop-common库的一部分。您从依赖项列表中丢失了它。确保此jar的版本与hadoop-common jar的版本相同。

此外，建议所有hadoop依赖罐都具有相同的版本。

更新：

hadoop-aws

[您正在尝试使用Hadoop库来写入S3。在这种情况下，Hadoop将尝试使用本地文件系统创建临时文件。这是Windows操作系统，您需要Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z 。

1. 根据您的Hadoop版本从winutils下载winutils二进制文件和其他所需文件。
2. 将环境变量winutils.exe设置为指向这些二进制文件的安装目录。

阅读更多： here