Backport (reimplement) FIX tool enhancements in 1.3.x (SEARCH-2368)

[SEARCH-2404] Reimplement SEARCH-2248 and SEARCH-2233 on SS 1.3.x Created: 03-Sep-20  Updated: 15-Oct-20  Resolved: 15-Sep-20

Status: Done
Project: Search and Discovery
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Sub-task Priority: Blocker
Reporter: Tom Page Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
relates to SEARCH-2233 Expose additional parameters for FIX ... Done
relates to SEARCH-2248 Add a limit to a number of txns per r... Done
Sprint: Team Ninja-King - S&I 45, Team Ninja-King - S&I 46, Team Ninja-King - S&I 47
Release Train: Hayes & Harlington
Delivery Team: Search


Reimplement the fixes from SEARCH-2248 and SEARCH-2233 on SS 1.3.x:

  • Add ability to restrict transactions to fix by start and end times.
  • Add ability to limit number of transactions to repair (together with a system-wide default)
  • Add a dry-run parameter that will output the number of transactions and nodes that would be affected (split by "Tx in index and not in DB", "Tx in index multiple times" and "Tx in DB and not in index")

Comment by Tom Page [ 03-Sep-20 ]

On master (2.1.x) then the core name is a mandatory parameter when calling the FIX action:

On 1.3.x then if the core name is not supplied the FIX action will be run against all cores:

                    if (cname != null) {
                        rsp.add(ACTION_LABEL, actionFIX(cname));
                    } else {
                        for (String coreName : getTrackerRegistry().getCoreNames()) {
                            rsp.add(ACTION_LABEL, actionFIX(coreName));

Master also contains some code to fix all cores:, but it comes after the validation that a core name has been supplied:

For the time being I'll proceed with the assumption that the core name is not mandatory on 1.3.x, but we may need to revisit this decision later.

Comment by Tom Page [ 07-Sep-20 ]

While checking the new parameters were being accepted by the server without causing errors I discovered that the fromTxCommitTime being in the past causes the process to take longer, even if the toTxCommitTime is very close to it. For example the following FIX requests takes about 12 seconds locally:
As does this:
Omitting the toTxCommitTime takes about 6s:
Omitting the fromTxCommitTime returns instantly:
As does sending timestamps in the very recent past:

Comment by Tom Page [ 07-Sep-20 ]


Comment by Tom Page [ 07-Sep-20 ]

Merged an extra test on master:

Comment by Tom Page [ 09-Sep-20 ]

Merged PR for dry run option:

Generated at Mon Jun 21 20:18:31 BST 2021 using Jira 7.13.15#713015-sha1:7c5ddd2c3e1709974ae9c48c17df8edd3919fe2c.