[在我的应用程序中,正在使用apache骆驼处理txt文件,应用程序使用交换对象将整个文件读入字符串,并在每行上添加分隔符,并使用apache骆驼的交换对象将其作为整个字符串写入文件。我低于OutOfMemoryError。 ***我观察到此异常是在交换对象完成文件处理后发生的,并且应该将文件移动到另一个文件夹(端点)* **。
我的应用程序具有16GB的堆空间支持。处理前的文件大小为685MB,添加分隔符后可能会增加200 MB。有人能帮我一下吗?预先谢谢你。
from("file:<filepath>/processed/INPUT?filter=#fileFilter&delay=120000&exclusiveReadLockStrategy=#fileReadLockStrategy&delete=true&autoCreate=true")
.threads(10, 10)
.process(fileProcessor).log("Came out of file processor")
.recipientList(header(DESTIONATION_PATH));
这是一条简单的路线,它从INPUT文件夹中读取文件,而fileProcessor通过在其中添加delimter来处理文件内容。 DESTIONATION_PATH是将放置文件的路径。我可以看到它正在打印日志“文件处理器超出限制”,下面是当文件即将发送到目标文件夹时出现异常。
Error has occurred:
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
at org.apache.camel.util.ObjectHelper.wrapCamelExecutionException(ObjectHelper.java:1842) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.impl.DefaultExchange.setException(DefaultExchange.java:385) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:64) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.RecipientList.process(RecipientList.java:133) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87) [camel-core-2.24.1.jar!/:2.24.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_171]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Suppressed: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-x01btadmdts3a-vsi-uat-dbs-com-1586519346908-0-4]
... 26 common frames omitted
***Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit***
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
at java.lang.StringCoding.encode(StringCoding.java:344)
at java.lang.String.getBytes(String.java:918)
at org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:198)
at org.apache.camel.converter.IOConverterOptimised.convertTo(IOConverterOptimised.java:55)
at org.apache.camel.impl.converter.OptimisedTypeConverter.convertTo(OptimisedTypeConverter.java:63)
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:299)
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:188)
at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:115)
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:333)
at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305)
at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169)
at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80)
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715)
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638)
at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248)
at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173)
at org.apache.camel.processor.RecipientList.process(RecipientList.java:133)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43)
at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157)
at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87)
Caused by: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300) ~[na:1.8.0_171]
at java.lang.StringCoding.encode(StringCoding.java:344) ~[na:1.8.0_171]
at java.lang.String.getBytes(String.java:918) ~[na:1.8.0_171]
at org.apache.camel.converter.IOConverter.toInputStream(IOConverter.java:198) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.converter.IOConverterOptimised.convertTo(IOConverterOptimised.java:55) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.impl.converter.OptimisedTypeConverter.convertTo(OptimisedTypeConverter.java:63) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.doConvertTo(BaseTypeConverterRegistry.java:299) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.impl.converter.BaseTypeConverterRegistry.mandatoryConvertTo(BaseTypeConverterRegistry.java:188) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:115) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:333) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:305) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:169) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:80) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) ~[camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:638) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:248) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:173) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.RecipientList.process(RecipientList.java:133) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:76) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline.access$100(Pipeline.java:43) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:157) [camel-core-2.24.1.jar!/:2.24.1]
at org.apache.camel.processor.ThreadsProcessor$ProcessCall.run(ThreadsProcessor.java:87) [camel-core-2.24.1.jar!/:2.24.1]
很抱歉在这里迟到了。我能够解决上述问题。当我仔细查看日志时,它在标准Java类“ StringCoding.java”中失败。
我的代码中的罪魁祸首是以下两行:
String content = exchange.getIn().getBody(String.class);
//Add deleimters in each line content string & write back the whole string in file.
//derive new content string from original content
exchange.getOut().setBody(content);// Camel stores content in String & sends it via route using character array.
Analysis:] >>
Java将字符串作为字符数组存储在后台。根据逻辑,我们在每行中添加了一些距离。我观察到(原始文件中的字符数+添加的分隔符数)> Integer.MAX_VALUE。因此,这里的Java无法将整个字符串存储到字符数组中,并且骆驼无法将文件放入在目标文件夹路径中。发生这种情况是因为修改后的文件中的字符数超出了最大整数大小。
解决方案:
]File contentFile = exchange.getIn().getBody(File.class); //Process the contentFile as file line by line & write the records back in file. exchange.getOut().setBody(contentFile);// Camel stores content in File & sends it via route using InputStream.
我保留了上面的代码,并且它起作用了,我们没有看到OutofMemory问题,因为在通过路由发送文件时骆驼在哪里使用数组。
谢谢您的阅读。快乐编码... !!!