使用Kafka运行Apache Beam时出现依赖性错误

Question

[当我使用Direct Runner运行Apache Beam代码时，出现以下错误：

Caused by: java.lang.NoSuchMethodError: org.slf4j.helpers.MessageFormatter.arrayFormat(Ljava/lang/String;[Ljava/lang/Object;)Lorg/slf4j/helpers/FormattingTuple;
    at org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger.writeLog (LogContext.java:428)
    at org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger.info (LogContext.java:382)
    at org.apache.kafka.clients.consumer.KafkaConsumer.assign (KafkaConsumer.java:1123)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)

如下运行它：

mvn package

java -cp target/myjar.jar \
    com.CLASSNAME \
    --runner=DirectRunner \
    ...

按以下方式运行不起作用：

mvn compile exec:java \
    -Dexec.mainClass="com.CLASSNAME" \
    -Dexec.args="..."

似乎与slf4j依赖冲突有关。我已经看了几个小时了，似乎没有任何进展。

我尝试过的事情：

• 更改库的版本
• 将slf4j-api依赖项放在顶部
• 不包括依赖项
• 更改范围
• 使用mvn dependency:tree检查依赖项，但看起来都不错
• 更改JDK版本

我已经注意到identical版本之间存在以下冲突：

非常感谢任何其他想法或意见。

pom文件的清洁版本以确保完整性：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>...</groupId>
    <artifactId>...</artifactId>
    <version>...</version>

    <packaging>jar</packaging>

    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <beam.version>2.19.0</beam.version>
        <google.cloud.core.version>1.108.1</google.cloud.core.version>
        <maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
        <maven-surefire-plugin.version>2.21.0</maven-surefire-plugin.version>
        <maven-jar-plugin.version>3.0.2</maven-jar-plugin.version>
        <maven-shade-plugin.version>3.1.0</maven-shade-plugin.version>
        <maven-exec-plugin.version>1.6.0</maven-exec-plugin.version>
        <slf4j.version>1.7.30</slf4j.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
...
    <dependencies>

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-api</artifactId>
            <version>${slf4j.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-core</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-sdks-java-io-kafka</artifactId>
            <version>${beam.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.4.0</version>
        </dependency>

        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>1.9.2</version>
        </dependency>

        <dependency>
            <groupId>org.apache.beam</groupId>
            <artifactId>beam-runners-direct-java</artifactId>
            <version>${beam.version}</version>
        </dependency>
    </dependencies>

</project>

Answer 1

我设法解决了这个问题。显然，这不是由依赖项本身引起的，而是由插件引起的。我没问过pom.xml，因为我不希望这部分有特定的影响。

错误原因：

<pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <version>${maven-exec-plugin.version}</version>
                    <configuration>
                        <!-- the following is needed for logging to pick up the log4j.properties file -->
                        <includePluginDependencies>true</includePluginDependencies>
                        <cleanupDaemonThreads>false</cleanupDaemonThreads>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>

我们不得不删除includePluginDependencies子标签，现在它可以工作：

<pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.codehaus.mojo</groupId>
                    <artifactId>exec-maven-plugin</artifactId>
                    <version>${maven-exec-plugin.version}</version>
                    <configuration>
                        <cleanupDaemonThreads>false</cleanupDaemonThreads>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>

请注意，该行显然是为了选择log4j属性文件（请参见注释），我们现在可能必须解决该问题。我们仍然必须研究这是否会带来任何问题。