NIFI:内存不足错误，尽管增加了堆大小

Question

我陷入了一个问题，NIFI服务就是拒绝启动。我监控了nifi-app.log，发现是OutOfMemory错误。我在bootstap.conf文件中增加了内存。但是这个错误仍然存在。

附件是nifi-app.log

    2017-12-19 17:08:37,133 ERROR [ActiveMQ InactivityMonitor ReadCheckTimer] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[ActiveMQ InactivityMonitor ReadCheckTimer,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,134 ERROR [Provenance Maintenance Thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Provenance Maintenance Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,134 ERROR [logback-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[logback-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,133 ERROR [FileSystemRepository Workers Thread-2] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[FileSystemRepository Workers Thread-2,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,135 ERROR [pool-15-thread-1] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[pool-15-thread-1,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:37,133 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow Service Tasks Thread-1,5,main]: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [Provenance Maintenance Thread-2] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [logback-2] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,496 ERROR [FileSystemRepository Workers Thread-2] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [Flow Service Tasks Thread-1] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [Event-Driven Process Thread-1] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,483 ERROR [FileSystemRepository Workers Thread-4] org.apache.nifi.engine.FlowEngine A flow controller task execution stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:38,482 ERROR [pool-15-thread-1] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:39,497 ERROR [ActiveMQ InactivityMonitor ReadCheckTimer] org.apache.nifi.NiFi 
java.lang.OutOfMemoryError: Java heap space
2017-12-19 17:08:40,509 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5351d0ef checkpointed with 2358 Records and 0 Swap Files in 114969 milliseconds (Stop-the-world time = 2361 milliseconds, Clear Edit Logs time = 936 millis), max Transaction ID 7622
2017-12-19 17:10:41,433 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@5351d0ef checkpointed with 2358 Records and 0 Swap Files in 922 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID 7622

我还尝试删除了flowfile和内容存储库，担心它可能已经损坏。但是，仍然没有变化。

我所能理解的是，这是一个线程错误，但我无法找到它的解决方法。

Answer 1

尝试关机，删除repos，在nifi.properties中设置nifi.flowcontroller.autoResumeState=false，然后重新启动。

这将确保当NiFi启动时，整个流将被停止，这样您的流就不会带来更多的数据并导致内存不足错误，然后您可以启动部分流来查看问题所在。

OOM的典型原因是获取一个大的流文件，并将其拆分成数百万个小的流文件，比如有1000万行的CSV进入行数为1的SplitText。

另一个原因可能是使用将流文件的整个内容读入内存的处理器，并让多GB的流文件命中该处理器，或者使用ExtractText将大量内容提取到属性中。