I am investigating methods for setting up Disaster Recovery for a client (TIAA) and have been thinking about a couple of options.
- Periodic back up and restores (from the primary to the DR site)
- Echoing the Elastic Search Events to the DR site.
In both cases, the biggest challenge will be synchronization. When recovering from a Business Continuity Incident, there is a high probability that the Elastic Search will not be in sync with the database. The challenge will be to:
- Detect the level of misalignment between the database and the Elastic Search
- Performing the appropriate clean up to force alignment between the two
In considering this challenge, a couple of questions come to mind.
- Is there a way to tell, by looking at the elastic search information only, the last event that was processed.
- If the elastic search has processed events that are not yet in the database (a possibility in a DR scenario) is there any way to roll back the processed events.
Here are my assessments of each of the methods –
please advise on any pros and cons of the methods and on what is possible. thanks
- Periodically back up the elastic search index
- Send the back up across to the DR site
- Restore the index on the DR site
- This uses common infrastructure utilities requires no special extensions to the Activiti App.
- Very low potential of the index being ahead of the database
- Backup will grow over time causing the size of the periodic data transfer to grow over time
- Echo the elastic search events to an additional host or set of hosts (on the DR site) that are not queried as part of any search/reporting requests
- This is an incremental method that will have more predictable bandwitdh requirements
- Requires extension code to be added to the activiti app
- May require an intermediate message queue to account for the possibility that the DR site is down.
- Increased possibility that the elastic search index could be ahead of the Database (which would mean that we would need to back out some of the events from the index)