[在运行oozie示例示例时,正在安排Oozie作业,并且状态显示为“正在运行”。一段时间后,工作被杀。在挖掘hadoop日志时,发现了这些异常。
我已经使用Hadoop 2.7.3设置了oozie 4.3.1。我还用正确的nameNode和jobTracker配置更新了job.properties。
请让我知道更改和解决问题所需的内容。
User: root
Name: oozie:launcher:T=java:W=java-main-wf:A=java-node:ID=0000000-191231120255907-oozie-root-W
Application Type: MAPREDUCE
Application Tags:
YarnApplicationState: FAILED
Queue: default
FinalStatus Reported by AM: FAILED
Started: Tue Dec 31 12:04:15 +0530 2019
Elapsed: 2mins, 0sec
Tracking URL: History
Diagnostics:
Application application_1576228338940_0013 failed 2 times due to AM Container for appattempt_1576228338940_0013_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://f091403isdpbato05:8088/cluster/app/application_1576228338940_0013Then, click on links to logs of each attempt.
Diagnostics: java.lang.NoClassDefFoundError: Could not initialize class java.net.NetworkInterface
Failing this attempt. Failing the application.
yarn-root-nodemanager-load-5.log
2019-12-31 12:04:45,264 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1576228338940_0013_000001 (auth:SIMPLE)
2019-12-31 12:04:45,267 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1576228338940_0013_01_000001 by user root
2019-12-31 12:04:45,267 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Creating a new application reference for app application_1576228338940_0013
2019-12-31 12:04:45,267 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=root IP=10.32.193.39 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1576228338940_0013 CONTAINERID=container_1576228338940_0013_01_000001
2019-12-31 12:04:45,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1576228338940_0013 transitioned from NEW to INITING
2019-12-31 12:04:45,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Adding container_1576228338940_0013_01_000001 to application application_1576228338940_0013
2019-12-31 12:04:45,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl: Application application_1576228338940_0013 transitioned from INITING to RUNNING
2019-12-31 12:04:45,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl: Container container_1576228338940_0013_01_000001 transitioned from NEW to LOCALIZING
2019-12-31 12:04:45,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1576228338940_0013
2019-12-31 12:04:45,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.32.193.39:9000/user/root/examples/apps/java-main/lib/oozie-examples-4.3.1.jar transitioned from INIT to DOWNLOADING
2019-12-31 12:04:45,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.32.193.39:9000/user/root/share/lib/lib_20191231120005/oozie/oozie-sharelib-oozie-4.3.1.jar transitioned from INIT to DOWNLOADING
2019-12-31 12:04:45,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.32.193.39:9000/user/root/share/lib/lib_20191231120005/oozie/oozie-hadoop-utils-hadoop-2-4.3.1.jar transitioned from INIT to DOWNLOADING
2019-12-31 12:04:45,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.32.193.39:9000/user/root/share/lib/lib_20191231120005/oozie/json-simple-1.1.jar transitioned from INIT to DOWNLOADING
2019-12-31 12:04:45,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.32.193.39:9000/tmp/hadoop-yarn/staging/root/.staging/job_1576228338940_0013/job.splitmetainfo transitioned from INIT to DOWNLOADING
2019-12-31 12:04:45,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.32.193.39:9000/tmp/hadoop-yarn/staging/root/.staging/job_1576228338940_0013/job.split transitioned from INIT to DOWNLOADING
2019-12-31 12:04:45,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.32.193.39:9000/tmp/hadoop-yarn/staging/root/.staging/job_1576228338940_0013/job.xml transitioned from INIT to DOWNLOADING
2019-12-31 12:04:45,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Downloading public rsrc:{ hdfs://10.32.193.39:9000/user/root/examples/apps/java-main/lib/oozie-examples-4.3.1.jar, 1577773923766, FILE, null }
2019-12-31 12:04:45,278 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Downloading public rsrc:{ hdfs://10.32.193.39:9000/user/root/share/lib/lib_20191231120005/oozie/oozie-sharelib-oozie-4.3.1.jar, 1577773807011, FILE, null }
2019-12-31 12:04:45,288 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Downloading public rsrc:{ hdfs://10.32.193.39:9000/user/root/share/lib/lib_20191231120005/oozie/oozie-hadoop-utils-hadoop-2-4.3.1.jar, 1577773807007, FILE, null }
2019-12-31 12:04:45,297 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Downloading public rsrc:{ hdfs://10.32.193.39:9000/user/root/share/lib/lib_20191231120005/oozie/json-simple-1.1.jar, 1577773807004, FILE, null }
2019-12-31 12:04:45,304 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Failed to download resource { { hdfs://10.32.193.39:9000/user/root/examples/apps/java-main/lib/oozie-examples-4.3.1.jar, 1577773923766, FILE, null },pending,[(container_1576228338940_0013_01_000001)],14920685961731101,DOWNLOADING}
java.lang.NoClassDefFoundError: Could not initialize class java.net.NetworkInterface
at org.apache.hadoop.net.NetUtils.isLocalAddress(NetUtils.java:691)
at org.apache.hadoop.hdfs.DFSClient.isLocalAddress(DFSClient.java:1079)
at org.apache.hadoop.hdfs.RemoteBlockReader2.<init>(RemoteBlockReader2.java:296)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:441)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:818)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:697)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:656)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:267)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-12-31 12:04:45,304 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource hdfs://10.32.193.39:9000/user/root/examples/apps/java-main/lib/oozie-examples-4.3.1.jar(->/tmp/hadoop-root/nm-local-dir/filecache/106/oozie-examples-4.3.1.jar) transitioned from DOWNLOADING to FAILED
2019-12-31 12:04:45,306 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Failed to download resource { { hdfs://10.32.193.39:9000/user/root/share/lib/lib_20191231120005/oozie/oozie-sharelib-oozie-4.3.1.jar, 1577773807011, FILE, null },pending,[(container_1576228338940_0013_01_000001)],14920685961866685,DOWNLOADING}
java.lang.NoClassDefFoundError: Could not initialize class java.net.NetworkInterface
at org.apache.hadoop.net.NetUtils.isLocalAddress(NetUtils.java:691)
at org.apache.hadoop.hdfs.DFSClient.isLocalAddress(DFSClient.java:1079)
at org.apache.hadoop.hdfs.RemoteBlockReader2.<init>(RemoteBlockReader2.java:296)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:441)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:818)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:697)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:656)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:267)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
yarn-root-resourcemanager-load-5.log
appattempt_1576228338940_0013_000002 with final state: FAILED, and exit status: -1000
2019-12-31 12:06:15,456 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1576228338940_0013_000002 State change from LAUNCHED to FINAL_SAVING
2019-12-31 12:06:15,456 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1576228338940_0013_000002
2019-12-31 12:06:15,456 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1576228338940_0013_000002
2019-12-31 12:06:15,456 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1576228338940_0013_000002 State change from FINAL_SAVING to FAILED
2019-12-31 12:06:15,456 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 2. The max attempts is 2
2019-12-31 12:06:15,456 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1576228338940_0013 with final state: FAILED
2019-12-31 12:06:15,457 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1576228338940_0013 State change from ACCEPTED to FINAL_SAVING
2019-12-31 12:06:15,457 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1576228338940_0013
2019-12-31 12:06:15,457 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Attempt appattempt_1576228338940_0013_000002 is done. finalState=FAILED
2019-12-31 12:06:15,457 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1576228338940_0013 failed 2 times due to AM Container for appattempt_1576228338940_0013_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://f091403isdpbato05:8088/cluster/app/application_1576228338940_0013Then, click on links to logs of each attempt.
Diagnostics: java.lang.NoClassDefFoundError: Could not initialize class java.net.NetworkInterface
Failing this attempt. Failing the application.
2019-12-31 12:06:15,457 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1576228338940_0013 State change from FINAL_SAVING to FAILED
2019-12-31 12:06:15,457 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1576228338940_0013 requests cleared
2019-12-31 12:06:15,457 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1576228338940_0013 failed 2 times due to AM Container for appattempt_1576228338940_0013_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://f091403isdpbato05:8088/cluster/app/application_1576228338940_0013Then, click on links to logs of each attempt.
Diagnostics: java.lang.NoClassDefFoundError: Could not initialize class java.net.NetworkInterface
Failing this attempt. Failing the application. APPID=application_1576228338940_0013
2019-12-31 12:06:15,457 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application removed - appId: application_1576228338940_0013 user: root queue: default #user-pending-applications: 0 #user-active-applications: 0 #queue-pending-applications: 0 #queue-active-applications: 0
2019-12-31 12:06:15,457 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application removed - appId: application_1576228338940_0013 user: root leaf-queue of parent: root #applications: 0
2019-12-31 12:06:15,457 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1576228338940_0013,name=oozie:launcher:T\=java:W\=java-main-wf:A\=java-node:ID\=0000000-191231120255907-oozie-root-W,user=root,queue=default,state=FAILED,trackingUrl=http://f091403isdpbato05:8088/cluster/app/application_1576228338940_0013,appMasterHost=N/A,startTime=1577774055215,finishTime=1577774175456,finalStatus=FAILED,memorySeconds=58978,vcoreSeconds=28,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=<memory:0\, vCores:0>,applicationType=MAPREDUCE
[如果可能,请包括xml和属性文件以及其他代码,以便通过屏蔽或更改数据敏感的实际值来更好地理解该问题。
检查所有罐子的oozie共享库路径。在hdfs中寻找类似的路径。
示例路径:/ user / oozie / share / lib /
Refer:sharelib-in-apache-oozie
将属性文件中的use system lib路径设置为true。
oozie.use.system.libpath = true
根据提供的数据,我假设您正在运行oozie java操作,该操作因未找到类异常而失败。如果是这种情况,请使用文件标签选项将包含引用的缺少类(java.net.NetworkInterface)的jar添加到xml,并将此添加jar也放置在指定的hdfs位置。
<file>[FILE-PATH]</file>
或者,将jar放在hdfs应用程序路径lib位置下。
((与map-reduce和pig操作一样,可以添加文件和归档文件以供Java应用程序使用。请参阅[#FilesAchives]节[为作业添加文件和归档文件]。]