我们在具有以下参数的4.6.0版上运行的Solr部署面临问题:
java -Xss256k -Xms512m -Xmx30g -jar start.jar
Solr在具有64GB RAM和12核的Windows Server 2008 R2上运行。 30 GB分配给Solr实例,数据导入后大约为索引大小。 1.2 GB。启动Solr实例后,Solr仪表板上的JVM内存使用情况将显示:
Used Memory: 8.61 GB
Peak Memory: 15.19 GB
Total Memory: 29.02 GB
[几天后,我们注意到内存的读像是
Used Memory: 22.32 GB
Peak Memory: 29.02 GB
Total Memory: 29.02 GB
每小时每小时刷新一次索引(凌晨2点至凌晨4点之间除外),其大小几乎从未改变。
但是,再等一会儿,Solr似乎变得毫无反应。 Solr仪表板无法加载,查询失败,并显示以下错误。我们进行了大量搜索,但似乎找不到任何解决方法。唯一的临时解决方法是,一旦Java进程超过30 GB的内存使用量,就重新启动Solr实例。此后,它可以正常运行约10-12天,但再次出现相同的问题。
并且以下给出的错误每天发生,通常发生在上午12点至早上6点之间。但是,这段时间内的流量非常少,并且在该时段没有数据更改的情况下,凌晨2点至凌晨4点之间也没有对Solr核心进行重新索引。
我们已经更改了jetty.xml中的以下几个配置,但这似乎并没有太大帮助。
<Set name="maxIdleTime">1200000</Set>
<Set name="lowResourceMaxIdleTime">10000</Set>
ERROR - 2015-12-04 02:10:56.821; org.apache.solr.common.SolrException; null:org.eclipse.jetty.io.EofException
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
at org.eclipse.jetty.http.AbstractGenerator.blockForOutput(AbstractGenerator.java:507)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:147)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
at sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
at sun.nio.cs.StreamEncoder.write(Unknown Source)
at java.io.OutputStreamWriter.write(Unknown Source)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:141)
at org.apache.solr.util.FastWriter.write(FastWriter.java:55)
at org.apache.solr.response.JSONWriter.writeStr(JSONResponseWriter.java:449)
at org.apache.solr.response.JSONWriter.writeKey(JSONResponseWriter.java:103)
at org.apache.solr.response.JSONWriter.writeSolrDocument(JSONResponseWriter.java:346)
at org.apache.solr.response.TextResponseWriter.writeDocuments(TextResponseWriter.java:275)
at org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:172)
at org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:183)
at org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:299)
at org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:95)
at org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:60)
at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:698)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:426)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at org.eclipse.jetty.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:375)
at org.eclipse.jetty.io.bio.StreamEndPoint.flush(StreamEndPoint.java:164)
at org.eclipse.jetty.io.bio.StreamEndPoint.flush(StreamEndPoint.java:182)
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:838)
根据我的经验,逐渐的无响应可能是JVM调整不良的症状。我建议使用带有VisualGC插件的VisualVM之类的工具监视JVM,或将GC登录到文件中,然后使用许多可用工具之一对其进行分析。
来自Wiki:
如果solr和java版本分别为> 5.5和> 1.8.0_5,我建议在ConcurrentMarkSweep上为solr配置G1GC。
感谢@Rajesh,这解决了我的问题。