Google DataflowRunner“发生了JNI错误，请检查您的安装，然后重试”-Gradle Beam项目

Question

我有Gradle Apache Beam Java项目，该项目从pubsub中读取事件并写入Cloud Firestore。我可以使用DirectRunner运行此流作业。在DataflowRunner上执行时，辅助启动日志显示以下错误行：

Line1 : "A JNI error has occurred, please check your installation and try again"
Line2 : "java.lang.NoClassDefFoundError: org/apache/beam/vendor/guava/v26_0_jre/com/google/common/graph/Network
    at java.lang.Class.getDeclaredMethods0(Native Method) 
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) 
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048) 
    at java.lang.Class.getMethod0(Class.java:3018) 
    at java.lang.Class.getMethod(Class.java:1784) 
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) 
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) 
 Caused by: java.lang.ClassNotFoundException: org.apache.beam.vendor.guava.v26_0_jre.com.google.common.graph.Network 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)"

Line 3: "Error syncing pod 8db0cbe852963b21f84ca2ee52e431a4 ("dataflow-streamprocessorapplicatio-06011028-n4lo-harness-sz6k_default(8db0cbe852963b21f84ca2ee52e431a4)"), skipping: failed to "StartContainer" for "java-streaming" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=java-streaming pod=dataflow-streamprocessorapplicatio-06011028-n4lo-harness-sz6k_default(8db0cbe852963b21f84ca2ee52e431a4)"

我还尝试添加beam-vendors-guava和beam-vendors-grpc依赖项。但是在DataflowRunner上运行时并没有帮助。

这是我的build.gradle文件依赖项部分。我正在使用gradle shade插件构建uber jar并执行/创建数据流模板并运行它：

dependencies {
    testCompile('org.junit.jupiter:junit-jupiter-api:5.4.0')
    testCompile('org.junit.jupiter:junit-jupiter-params:5.4.0')
    testCompile('org.junit.jupiter:junit-jupiter-engine:5.4.0')
    testCompile(group: 'org.mockito', name: 'mockito-junit-jupiter', version: '3.2.4')
    testCompile(group: 'org.powermock', name: 'powermock-module-junit4', version: '2.0.4')
    testCompile(group: 'org.mockito', name: 'mockito-core', version: '2.28.2') { force true }
    testCompile(group: 'org.powermock', name: 'powermock-api-mockito2', version: '2.0.4')
    testCompile(group: 'org.hamcrest', name: 'hamcrest-all', version: '1.3')

//    compile(group: 'org.apache.beam', name: 'beam-runners-google-cloud-dataflow-java', version: '2.19.0')
//    runtimeClasspath(group: 'org.apache.beam', name: 'beam-runners-google-cloud-dataflow-java', version: '2.19.0')
    runtime(group: 'org.apache.beam', name: 'beam-runners-google-cloud-dataflow-java', version: '2.19.0')

    compile(group: 'org.apache.beam', name: 'beam-sdks-java-io-google-cloud-platform', version: '2.19.0') {
        exclude(group: 'io.grpc', module: 'grpc-all')
    }

    // https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-core
    compile group: 'org.apache.beam', name: 'beam-sdks-java-core', version: '2.19.0'

    compile group: 'org.apache.beam', name: 'beam-sdks-java-io-redis', version: '2.19.0'
    compile(group: 'org.apache.beam', name: 'beam-sdks-java-maven-archetypes-starter', version: '2.19.0')
//    runtime(group: 'org.apache.beam', name: 'beam-sdks-java-maven-archetypes-starter', version: '2.19.0')
    compileOnly group: 'org.apache.beam', name: 'beam-runners-direct-java', version: '2.19.0'
    compile(group: 'io.grpc', name: 'grpc-all', version: '1.25.0') {
        force true
    }

    compile group: 'org.projectlombok', name: 'lombok', version: '1.18.10'
    annotationProcessor 'org.projectlombok:lombok:1.18.8'

    compile group: 'org.slf4j', name: 'slf4j-api', version: '1.7.25'
    compile group: 'org.slf4j', name: 'slf4j-simple', version: '1.7.25'

    compile group: 'com.google.cloud', name: 'google-cloud-firestore', version: '1.32.2'
    compile group: 'org.apache.commons', name: 'commons-collections4', version: '4.0'
}

在这种情况下的任何帮助/建议都会有所帮助。我正在使用Java的Apache Beam SDK 2.19.0。

Answer 1

由于您的问题说有“任何帮助建议”，所以我还将写我的评论作为“我如何尝试调试此问题”的答案？

1. 检查org/apache/beam/vendor/guava/v26_0_jre/com/google/common/graph/Network是否在超级罐子中。如果不是，则为影子插件配置问题。
2. 检查是否正在上载超级jar。如果不是，则运行时类路径可能存在问题。您也可以在Java代码的--filesToStage的命令行上指定setFilesToStage。
3. 检查uber jar是否已在worker上正确下载。