Nifi出错:无法写入StandardFlowfileRecord

问题描述 投票:0回答:1

我最近创建了一个Nifi docker镜像,可以使用我的自定义配置和状态运行nifi docker。

一切都很好,但过了一段时间我收到以下错误:

2019-04-08 10:42:26,107 ERROR [Timer-Driven Process Thread-10] o.a.n.c.r.StandardProcessSession Failed to commit session java.lang.IllegalStateException: Cannot update journal file ./flowfile_repository/journals/449486.journal because no header has been written yet.. Will roll back.
2019-04-08 10:42:26,108 ERROR [Timer-Driven Process Thread-10] d.f.fit.processors.opcua.GetOPCData GetOPCData[id=e8affa98-0169-1000-1717-d9e8f54293ea] GetOPCData[id=e8affa98-0169-1000-1717-d9e8f54293ea] failed to process session due to java.lang.IllegalStateException: Cannot update journal file ./flowfile_repository/journals/449486.journal because no header has been written yet.; Processor Administratively Yielded for 1 sec: 
 java.lang.IllegalStateException: Cannot update journal file ./flowfile_repository/journals/449486.journal because no header has been written yet.
    at org.apache.nifi.wali.LengthDelimitedJournal.update(LengthDelimitedJournal.java:230)
    at org.apache.nifi.wali.SequentialAccessWriteAheadLog.update(SequentialAccessWriteAheadLog.java:123)
    at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:309)
    at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:259)
    at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:368)
    at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:343)
    at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
    at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
    at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:205)
    at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    Suppressed: java.lang.IllegalStateException: Cannot update journal file ./flowfile_repository/journals/449486.journal because no header has been written yet.
            at org.apache.nifi.wali.LengthDelimitedJournal.update(LengthDelimitedJournal.java:230)
            at org.apache.nifi.wali.SequentialAccessWriteAheadLog.update(SequentialAccessWriteAheadLog.java:123)
            at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:309)
            at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:259)
            at org.apache.nifi.controller.repository.StandardProcessSession.rollback(StandardProcessSession.java:1060)
            at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:501)
            ... 12 common frames omitted

它与java堆或内存有关吗?我该如何更正此错误?

docker dockerfile apache-nifi
1个回答
1
投票

如果您不关心流中的任何数据,可以关闭并删除所有“存储库”目录,默认情况下它们是在NiFi的主目录中创建的。

它没有解释是什么原因造成的,但它应该让你回到工作状态。


1
投票

在我的情况下,我用完了打开文件句柄,不幸的是错误没有给出任何指示。我怎么想的是:

除非查询诊断。来源:curl http://localhost:8080/nifi-api/system-diagnostics

检查所有“免费”字段(例如“provenanceRepositoryStorageUsage.freeSpace”)并确保您有足够的资源。

对我来说这返回的值如下:

flowFileRepositoryStorageUsage.totalSpaceBytes=-1

当NiFi无法确定这些值时,会报告-1。作为诊断的一部分,将记录正确的错误消息。您可能需要快速检查日志。在我的情况下,我设置了滚动日志,因此日志消息很快消失(许多处理器记录失败并填满日志)。错误看起来像:

2019-06-03 18:48:59,619 ERROR [pool-10-thread-1] org.apache.nifi.controller.repository.WriteAheadFlowFileRepository Unable to checkpoint FlowFile Repository due to java.io.FileNotFoundException: ./flowfile_repository/journals/8243.journal (Too many open files)
java.io.FileNotFoundException: ./flowfile_repository/journals/8243.journal (Too many open files)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.\u003cinit\u003e(FileOutputStream.java:213)
    at java.io.FileOutputStream.\u003cinit\u003e(FileOtutputStream.java:162)\n\tat org.apache.nifi.wali.LengthDelimitedJournal.getOutputStream(LengthDelimitedJournal.java:136)
    at org.apache.nifi.wali.LengthDelimitedJournal.writeHeader(LengthDelimitedJournal.java:151)
    at org.apache.nifi.wali.SequentialAccessWriteAheadLog.checkpoint(SequentialAccessWriteAheadLog.java:306)
    at org.apache.nifi.wali.SequentialAccessWriteAheadLog.checkpoint(SequentialAccessWriteAheadLog.java:251)
    at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.checkpoint(WriteAheadFlowFileRepository.java:735)
    at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository$1.run(WriteAheadFlowFileRepository.java:693)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

这里的重要部分是“打开文件过多”。

使用linux获取当前限制:ulimit -x -n

要使用linux获取打开文件的数量:sudo ls /proc/$NIFI_PID/fd | wc -l

我正在使用docker所以我使用docker run命令增加了限制:docker run --ulimit nofile=1048576:1048576 ...

其他解决方案是:增加主机的限制,确定哪些处理器占用了手柄并重新处理它们(不那么容易)。


0
投票

使用日志限制运行Nifi docker:

 docker run --name nifi -p 8080:8080 --log-driver json-file --log-opt max-size=5m --log-opt max-file=4  --restart always -d nifi
© www.soinside.com 2019 - 2024. All rights reserved.