-
Type:
Bug
-
Status: Review
-
Resolution: Unresolved
-
Affects Version/s: Search Services 2.0
-
Fix Version/s: Search Services 2.0.1
-
Labels:None
-
Bug Priority:
Category 2
-
Epic Link:
-
Sprint:Team Ninja-King - S&I 52, Team Ninja-King - S&I 53, Team Ninja-King - S&I 54
-
Story Points:13
The environment was set up for SEARCH-2322.
The reindex speed is ~150k docs/hour vs ~600k docs/hour in 1.4.3.
The problem are:
- CPU on solr is loaded on ~100% constantly. It is expected with mock indexing but distribution of load across solr treads are different in comparison with 1.4.
New threads appeared for 2.0 that there were not for 1.4.
It is ForkJoinPoolThreads. There are 64 threads in the pool:
18 Runnable,
15 Blocked,
30 Waiting.
In comparison with 1.4 there were no such thread among solr threads. Please pay attention that when ForkJoinThreads are running there are no SolrTrackingPool among threads what means that tracking of metadata and content is not being done during that period.
As we can see from Top every Runnable thread from this pool are using ~15-20% CPU on the average. Considering that 18 thread are Runnable then it can be calculated that ~70% of overall CPU is occupied by these threads.
18 * 15 = 270% from 400%
Due to that every index portion happens every 20-30 minutes only so indexing is so slow.
Because of solr is overloaded response time for tracking requests is long tool but it is consequence of the issue above.
Looks like Runnable threads are reading json and writing in the buffer:
Blocked threads are blocked by adding entry to UpdateLog or lookup of UpdateLog (7 threads blocked by adding, 6 threads are blocked by lookup) :
"ForkJoinPool-6-worker-29" #4462 daemon prio=5 os_prio=0 cpu=1311662.31ms elapsed=20675.76s tid=0x00007ffb20027000 nid=0x49dc waiting for monitor entry [0x00007ffa21f25000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.solr.update.UpdateLog.lookup(UpdateLog.java:948)
- waiting to lock <0x00000004408b15f8> (a org.apache.solr.update.UpdateLog)
at org.apache.solr.handler.component.RealTimeGetComponent.getInputDocumentFromTlog(RealTimeGetComponent.java:517)
at org.apache.solr.handler.component.RealTimeGetComponent.getInputDocument(RealTimeGetComponent.java:593)
at org.apache.solr.handler.component.RealTimeGetComponent.getInputDocument(RealTimeGetComponent.java:570)
at org.apache.solr.update.processor.DistributedUpdateProcessor.getUpdatedDocument(DistributedUpdateProcessor.java:1362)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1078)
- locked <0x00000004428c6680> (a org.apache.solr.update.VersionBucket)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
at org.alfresco.solr.SolrInformationServer.updateContent(SolrInformationServer.java:1910)
at org.alfresco.solr.tracker.ContentTracker$ContentIndexWorkerRunnable.doWork(ContentTracker.java:191)
at org.alfresco.solr.tracker.AbstractWorker.run(AbstractWorker.java:48)
at org.alfresco.solr.tracker.ContentTracker.lambda$doTrack$0(ContentTracker.java:130)
at org.alfresco.solr.tracker.ContentTracker$$Lambda$375/0x000000080089c440.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(java.base@11.0.1/ReferencePipeline.java:195)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(java.base@11.0.1/ArrayList.java:1654)
at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.1/AbstractPipeline.java:484)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.1/AbstractPipeline.java:474)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(java.base@11.0.1/ReduceOps.java:952)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(java.base@11.0.1/ReduceOps.java:926)
at java.util.stream.AbstractTask.compute(java.base@11.0.1/AbstractTask.java:327)
at java.util.concurrent.CountedCompleter.exec(java.base@11.0.1/CountedCompleter.java:746)
at java.util.concurrent.ForkJoinTask.doExec(java.base@11.0.1/ForkJoinTask.java:290)
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@11.0.1/ForkJoinPool.java:1020)
at java.util.concurrent.ForkJoinPool.scan(java.base@11.0.1/ForkJoinPool.java:1656)
at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.1/ForkJoinPool.java:1594)
at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.1/ForkJoinWorkerThread.java:177)
- In regards to solr errors. There are no any errors on solr. Only new warnings that I haven’t seen for 1.4. Looks like they are related to Update log too. This warning occurs just on start of solr.
- is related to by
-
SEARCH-2402 Configure IE in order to use best compression method for stored fields
-
- Open
-
-
SEARCH-2403 Reflect transaction log configuration in the documentation for 2.0
-
- Open
-
- relates to
-
-
- Done
-