Uploaded image for project: 'Search and Discovery'
  1. Search and Discovery
  2. SEARCH-2415

Slow reindex speed in 2.0 comparing to 1.4 in "Sizing Guide" environment

    Details

    • Type: Bug
    • Status: Review
    • Resolution: Unresolved
    • Affects Version/s: Search Services 2.0
    • Fix Version/s: Search Services 2.0.1
    • Labels:
      None

      Description

      The environment was set up for SEARCH-2322.
      The reindex speed is ~150k docs/hour vs ~600k docs/hour in 1.4.3.
      The problem are:

      • CPU on solr is loaded on ~100% constantly. It is expected with mock indexing but distribution of load across solr treads are different in comparison with 1.4.
        New threads appeared for 2.0 that there were not for 1.4.
        It is ForkJoinPoolThreads. There are 64 threads in the pool:
        18 Runnable,
        15 Blocked,
        30 Waiting.

      In comparison with 1.4 there were no such thread among solr threads. Please pay attention that when ForkJoinThreads are running there are no SolrTrackingPool among threads what means that tracking of metadata and content is not being done during that period.


      As we can see from Top every Runnable thread from this pool are using ~15-20% CPU on the average. Considering that 18 thread are Runnable then it can be calculated that ~70% of overall CPU is occupied by these threads.

      18 * 15 = 270% from 400%

      Due to that every index portion happens every 20-30 minutes only so indexing is so slow.
      Because of solr is overloaded response time for tracking requests is long tool but it is consequence of the issue above.

      Looks like Runnable threads are reading json and writing in the buffer:


      Blocked threads are blocked by adding entry to UpdateLog or lookup of UpdateLog (7 threads blocked by adding, 6 threads are blocked by lookup) :



      "ForkJoinPool-6-worker-29" #4462 daemon prio=5 os_prio=0 cpu=1311662.31ms elapsed=20675.76s tid=0x00007ffb20027000 nid=0x49dc waiting for monitor entry  [0x00007ffa21f25000]
      
         java.lang.Thread.State: BLOCKED (on object monitor)
      
                     at org.apache.solr.update.UpdateLog.lookup(UpdateLog.java:948)
      
                     - waiting to lock <0x00000004408b15f8> (a org.apache.solr.update.UpdateLog)
      
                     at org.apache.solr.handler.component.RealTimeGetComponent.getInputDocumentFromTlog(RealTimeGetComponent.java:517)
      
                     at org.apache.solr.handler.component.RealTimeGetComponent.getInputDocument(RealTimeGetComponent.java:593)
      
                     at org.apache.solr.handler.component.RealTimeGetComponent.getInputDocument(RealTimeGetComponent.java:570)
      
                     at org.apache.solr.update.processor.DistributedUpdateProcessor.getUpdatedDocument(DistributedUpdateProcessor.java:1362)
      
                     at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1078)
      
                     - locked <0x00000004428c6680> (a org.apache.solr.update.VersionBucket)
      
                     at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
      
                     at org.alfresco.solr.SolrInformationServer.updateContent(SolrInformationServer.java:1910)
      
                     at org.alfresco.solr.tracker.ContentTracker$ContentIndexWorkerRunnable.doWork(ContentTracker.java:191)
      
                     at org.alfresco.solr.tracker.AbstractWorker.run(AbstractWorker.java:48)
      
                     at org.alfresco.solr.tracker.ContentTracker.lambda$doTrack$0(ContentTracker.java:130)
      
                     at org.alfresco.solr.tracker.ContentTracker$$Lambda$375/0x000000080089c440.apply(Unknown Source)
      
                     at java.util.stream.ReferencePipeline$3$1.accept(java.base@11.0.1/ReferencePipeline.java:195)
      
                     at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(java.base@11.0.1/ArrayList.java:1654)
      
                     at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.1/AbstractPipeline.java:484)
      
                     at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.1/AbstractPipeline.java:474)
      
                     at java.util.stream.ReduceOps$ReduceTask.doLeaf(java.base@11.0.1/ReduceOps.java:952)
      
                     at java.util.stream.ReduceOps$ReduceTask.doLeaf(java.base@11.0.1/ReduceOps.java:926)
      
                     at java.util.stream.AbstractTask.compute(java.base@11.0.1/AbstractTask.java:327)
      
                     at java.util.concurrent.CountedCompleter.exec(java.base@11.0.1/CountedCompleter.java:746)
      
                     at java.util.concurrent.ForkJoinTask.doExec(java.base@11.0.1/ForkJoinTask.java:290)
      
                     at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(java.base@11.0.1/ForkJoinPool.java:1020)
      
                     at java.util.concurrent.ForkJoinPool.scan(java.base@11.0.1/ForkJoinPool.java:1656)
      
                     at java.util.concurrent.ForkJoinPool.runWorker(java.base@11.0.1/ForkJoinPool.java:1594)
      
                     at java.util.concurrent.ForkJoinWorkerThread.run(java.base@11.0.1/ForkJoinWorkerThread.java:177)
      
      • In regards to solr errors. There are no any errors on solr. Only new warnings that I haven’t seen for 1.4. Looks like they are related to Update log too. This warning occurs just on start of solr.

        Attachments

        1. image011.png
          image011.png
          49 kB
        2. image025.png
          image025.png
          98 kB
        3. image026.jpg
          image026.jpg
          29 kB
        4. image027.jpg
          image027.jpg
          215 kB
        5. image028.jpg
          image028.jpg
          100 kB
        6. image029.jpg
          image029.jpg
          215 kB
        7. image030.jpg
          image030.jpg
          242 kB
        8. image031.jpg
          image031.jpg
          277 kB
        9. image038.png
          image038.png
          91 kB
        10. image039.jpg
          image039.jpg
          190 kB
        11. image040.jpg
          image040.jpg
          200 kB
        12. image041.jpg
          image041.jpg
          169 kB
        13. image045.png
          image045.png
          715 kB
        14. thread_dump_solr.log
          228 kB
        15. tracker-performance.png
          tracker-performance.png
          27 kB

          Issue Links

            Structure

              Activity

                People

                • Assignee:
                  eporciani Elia Porciani
                  Reporter:
                  amukha Alex Mukha
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:

                    Structure Helper Panel