[SEARCH-2330] Implement disable/enable indexing commands for tracking process Created: 30-Jun-20  Updated: 16-Sep-20  Resolved: 31-Jul-20

Status: Done
Project: Search and Discovery
Component/s: None
Affects Version/s: None
Fix Version/s: Search Services 2.0

Type: Story
Reporter: Alex Mukha Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File logs after disabling indexing .png     PNG File trackers_after_search_2330.png     PNG File trackers_before_search_2330.png    
Issue Links:
Related
is related to by SEARCH-2405 Reimplement SEARCH-2330 on SS 1.3.x Done
Epic Link: Indexing STOP Command
Sprint: Team Ninja-King - S&I 37, Team Ninja-King - S&I 38, Team Ninja-King - S&I 39, Team Ninja-King - S&I 40
Release Train: Southall
Delivery Team: Search
Story Points: 8

 Description   

Acceptance criteria:

  • Add STOP* and START* commands to asynchronous actions API: https://docs.alfresco.com/sie/concepts/solr-admin-asynchronous-actions.html
  • STOP* should stop the tracking process. The tracking can be started again by issuing START* command or restarting the application.
  • The org.alfresco.solr.TrackerState should reflect the new state of the trackers
  • If the tracking has not started yet and a STOP* command received, the trackers should not begin the work. The check can be added to org.alfresco.solr.tracker.AbstractTracker#track
  • If the tracking has started and STOP* command was received, then the rollback of all trackers should be performed. The existing workers would have to finish the work, but new ones will not start.
  • The STOP command should not affect the propagation of the ShardState to content repository (org.alfresco.solr.tracker.NodeStatePublisher)
  • The execution of STOP* command clear the list of work scheduled by FIX command so that after restart this work is not executed again.
  • The logs should contain a message once the STOP* command was issued.

*See the comment below: the actions have been renamed in ENABLED-INDEXING / DISABLE-INDEXING



 Comments   
Comment by Martin Stanford [ 09-Jul-20 ]

Andrea Gazzarini - Agree that use of START/STOP is potentially confusing. Good spot.

Agree that ENABLE/DISABLE is better than START/STOP.

What about ENABLE-INDEXING/DISABLE-INDEXING? This then describes the result, rather than the means by which that result will be acheived. Does that work?

Comment by Martin Stanford [ 20-Jul-20 ]

IMO the overall aim of the STOP command should be to get the system to a point where is a) not attempting to index anything, and b) stable (i.e. all transactions either committed or rolled back, as quickly as possible.

Therefore I would tentatively suggest option b - empty the lists that are not yet been reindexed. 

Open to discussion though. Would like to know Alex's view and your (Andrea Gazzarini) recommendation.

Comment by Andrea Gazzarini [ 22-Jul-20 ]

Purpose and description of the enable/disable indexing admin commands

Enabling or disabling the indexing on a master or standalone core means the following trackers will be put in standby

  • MetadataTracker
  • CascadeTracker (if enabled on the target core)
  • ContentTracker
  • AclTracker

as consequence of that, if the indexing is disabled on core A, that core won’t be tracking and it won't be pulling changes from repository. The commands don’t affect the following components:

  • CommitTracker: because we still need to be able to execute commit/rollback/maintenance ops
  • ShardStatePublisher: because the node will still need to publish its state towards Alfresco (i.e. towards the ShardRegistry)

How to enable/disable indexing

The two commands are available as additional actions in the Alfresco Solr Admin endpoint (/admin); they can be triggered using the following syntax (which is basically the same of the other admin endpoints):

Example A: disable indexing on all (master or standalone) cores

http://localhost:8983/solr/admin/cores?action=disable-indexing 

Example B: disable indexing on a specific (master or standalone) core

http://localhost:8983/solr/admin/cores?action=disable-indexing&core=alfresco 

Example C: disable indexing on all (master or standalone) cores

http://localhost:8983/solr/admin/cores?action=enable-indexing 

Example D: disable indexing on a specific (master or standalone) core

http://localhost:8983/solr/admin/cores?action=enable-indexing&core=alfresco

Note the commands can target only a master or a standalone core.

That means: 

  • if the target core is a slave then the command has no effect: an informational message is returned in response
  • if a Solr instance hosts mixed cores (slaves and masters) the enable/disable commands issued without a target core parameter will affect only master or standalone cores
Tests:

 

org.alfresco.solr.AlfrescoCoreAdminHandlerTest#disableIndexingActionParameter_shouldTriggerTheDisableIndexingAction
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#disableIndexingOnSpecificSlaveCore_shouldReturnAnErrorResponse
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#disableIndexingWithoutIndicatingSpecificCore_shouldAffectOnlyMasterOrStandaloneCores
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#disableIndexingWithoutIndicatingSpecificCore_shouldHaveNoEffectIfAllCoresAreSlave
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#enableIndexingActionParameter_shouldTriggerTheIndexingEnabling
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#enableIndexingOnSpecificSlaveCore_shouldReturnAnErrorResponse
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#enableIndexingWithoutIndicatingSpecificCore_shouldAffectOnlyMasterOrStandaloneCores
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#enableIndexingWithoutIndicatingSpecificCore_shouldHaveNoEffectIfAllCoresAreSlave
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#unknownCoreNameInDisableIndexingCommand_shouldReturnAnErrorResponse
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#unknownCoreNameInEnableIndexingCommand_shouldReturnAnErrorResponse 

Commands are Idempotent

The Enable and disable indexing commands are idempotent; that means

  • the very first time they are executed, a transition happens in the indexing state (from enabled to disabled or viceversa)
  • subsequent calls using the same command (i.e. two subsequent enable-indexing commands) leave the system untouched there's no any side-effect   
Tests: 
org.alfresco.solr.tracker.ActivatableTrackerTest#assertActivatableTrackersList
org.alfresco.solr.tracker.ActivatableTrackerTest#disableIsIdempotent
org.alfresco.solr.tracker.ActivatableTrackerTest#assertAlwaysActivatedTrackersList
org.alfresco.solr.tracker.ActivatableTrackerTest#enabledShouldBeTheDefaultState
org.alfresco.solr.tracker.ActivatableTrackerTest#trackersCanBeExplicitlyDisabled
org.alfresco.solr.tracker.ActivatableTrackerTest#enableIsIdempotent

Persistence

The indexing state survives across core reloads but not across node restarts.

That means if core A is tracking, the administrator disables the indexing and then reload the core A, the reloaded core instance will still have the indexing disabled: in order to restart the tracking, the administrator needs to explicitly issue the “enable-tracking” command.

However, if the hosting Solr instance (or even the whole machine) that hosts the core A is restarted then the indexing state is lost and at startup all (master or standalone) cores will have the indexing enabled.     

Test: 
org.alfresco.solr.AlfrescoIndexingStatePersistenceAcrossReloadsIT#testIndexingStateAcrossReloads

What happens when a tracker is in the middle of its tracking cycle and the administrator disables the indexing?

In this case:

  • the trackers will complete its tracking cycle
  • the trackers will be set in rollback mode
  • in the next commit tracker cycle a rollback will be performed
Tests: 
org.alfresco.solr.tracker.ActivatableTrackerTest#disablingATracker_shouldClearTheScheduledMaintenanceWork
org.alfresco.solr.tracker.ActivatableTrackerTest#disableIndexingOnRunningTracker_shouldDisableTheTrackerAnSetItInRollbackMode

What happens to the scheduled maintenance work?

There are a set of admin actions (PURGE, FIX, INDEX, REINDEX, RETRY) that are asynchronous: when the administrator invokes one of them, some work is scheduled for being executed the next time the commit tracker will run (this is what we call "maintenance").

The disable-indexing command clears all the scheduled maintenance work: that includes any transaction, node id, ACL id, ACL changeset  asynchronously scheduled as a consequence of a previous invocation of the FIX, PURGE, INDEX, RETY and REINDEX actions: when the disable-indexing command is executed it “tries” to clear the scheduled sets of identifiers.

However, it’s possible that when the administrator disables the indexing, the commit tracker is working, and it already started the maintenance cycle; in that case, at the end of its cycle, the CommitTracker will issue a rollback instead of a commit.   

Tests: 
org.alfresco.solr.tracker.ActivatableTrackerTest#disablingATracker_shouldClearTheScheduledMaintenanceWork
org.alfresco.solr.tracker.ActivatableTrackerTest#disableIndexingOnRunningTracker_shouldDisableTheTrackerAnSetItInRollbackMode

Impact on the existing admin actions

The administration endpoint offers several actions for managing the Solr instance on a given node. Some of them needs to take in account the indexing state before executing their logic.

This is a list of the existing actions where the behaviour slightly changed as consequence of this ticket.      

REPORT
  • The subreport generated by the ACL tracker has an additional information indicating if the tracker is enabled (enabled) or not (disabled): 

 

<str name=“ACL Tracker>enabled</str> 
  • the subreport generated by the Metadata tracker has an additional information indicating if the tracker is enabled (enabled) or not (disabled):
<str name=“Metadata Tracker>enabled</str>
SUMMARY

Each tracker provides, in the SUMMARY, an additional information about its state: 

<bool name=“ACLTracker Enabled”>true</str>
<bool name=“MetadataTracker Enabled”>true</str>
<bool name=“ContentTracker Enabled”>true</str>
<bool name=“CascadeTracker Enabled”>true</str>
INDEX, RETRY, REINDEX, PURGE 

When the indexing is enabled, the old behaviour applies: each of those actions collects some object (e.g transactions, nodes, acls) and it schedules them for the next maintenance cycle.

When the indexing is disabled, those actions have no effect: the status element in the response will have a “NotScheduled” value, and a proper informational message is set in the response indicating the command didn’t have any effect because the indexing state.    

<str name=“status>enabled</str>
<str name=“additionalInfo>
     Trackers have been disabled: the PURGE request cannot be executed; 
     please enable indexing and then resubmit this command.
</str>
Tests: 
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#indexActionOnSlaveNode_shouldReturnWarningMessage
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#indexActionWhenIndexingIsDisabled_shouldReturnAnInfoMessage
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#indexActionWhenIndexingIsEnabled_shouldCollectThingsToReindex
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#purgeActionOnSlaveNode_shouldReturnWarningMessage
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#purgeActionWhenIndexingIsDisabled_shouldReturnAnInfoMessage
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#purgeActionWhenIndexingIsEnabled_shouldCollectTransactionsToPurge
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#reindexActionOnSlaveNode_shouldReturnWarningMessage
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#reindexActionWhenIndexingIsDisabled_shouldReturnAnInfoMessage
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#reindexActionWhenIndexingIsEnabled_shouldCollectThingsToReindex
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#retryActionOnSlaveNode_shouldReturnWarningMessage
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#retryActionWhenIndexingIsDisabled_shouldReturnAnInfoMessage
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#retryActionWhenIndexingIsEnabled_shouldCollectThingsToReindex 
FIX

When the indexing is enabled, the old behaviour applies: the action collects some object (e.g transactions, nodes, acls) and schedules them for the next maintenance cycle.

When the indexing is disabled, the FIX tool is be forced to execute in a dryRun mode: that means nothing will be scheduled, but the response will include the amount of work that would be scheduled if the indexing would be enabled.    

Tests: 
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#fixOnSlaveNodeHasNoEffect
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#manageTransactionsToBeFixed_shouldRespectTheInputGlobalLimit
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#masterOrStandaloneNode_explicitDryRunParameterIsEchoed
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#masterOrStandaloneNode_explicitFromCommitTimeParameterIsEchoed
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#masterOrStandaloneNode_explicitMaxTransactionsToScheduleParameterIsEchoed
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#masterOrStandaloneNode_explicitToCommitTimeParameterIsEchoed
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#masterOrStandaloneNode_implicitDryRunParameterIsEchoed
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#masterOrStandaloneNodeWithTrackersDisabled_DryRunParameterShouldBeForcedToTrue
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#maxAclTransactionsGlobalLimitShouldBeAppliedInCascade
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#maxTransactionScheduledIsNull_shouldBeGatheredFromCoreProperties
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#maxTransactionScheduledParameterAndConfigurationIsNull_shouldGetTheHardCodedDefault
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#maxTransactionScheduledParameterIsNotNull
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#maxTransactionsGlobalLimitShouldBeAppliedInCascade
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#noTransactionToReindex_shouldReturnAnEmptyResponse
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#noTargetCoreToFixInParams
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#unknownTargetCoreToFixInParams
org.alfresco.solr.AlfrescoCoreAdminHandlerTest#subsequentInvocationsToManageTransactionsToBeFixed_shouldRespectTheInputGlobalLimit

 

 

Generated at Sun Mar 07 18:31:25 GMT 2021 using Jira 7.13.15#713015-sha1:7c5ddd2c3e1709974ae9c48c17df8edd3919fe2c.