This article helps to share details about information to be retrieved from a PAA setup in time of an event failure (Like Monitor results or service failure) before restoring the services for analysis and RCA.

We understand that service continuity is customer's priority and that, in the event of a failure, rollback may need to be performed immediately. To support subsequent troubleshooting and RCA, we recommend collecting the following

information before proceeding with the rollback, wherever feasible.

1. Information recommended for collection when a failure occurs

• Exact date and time of failure, including timezone.
• Routing Active Testing test/monitor name or ID and details of the affected test. *(Config and settings)
• Source/Test Agent and destination/target information.
• Screenshot/export of the failed test result, including the displayed error message or error code.
• Test Agent hostname/ID, software version, and current status.
• Relevant Test Agent/application logs covering the failure period.
  • From control Center capture the complete /var/log directory if not sure of exact logs.
• Any relevant alarms/events observed around the same time.
• Details of any recent configuration, software, network, or infrastructure changes.

2. Information required when contacting JTAC Support

When opening a JTAC case, please provide the following, where available:

• Clear description of the issue and the observed vs. expected behavior.
• Exact failure timestamp and timezone.
• Affected Routing Active Testing test/monitor and Test Agent details.
• Relevant screenshots/error messages/test results. *(if issue was seen, may be a small video capture of screen showing the issue and error noticed)
• Product/component software versions.
• Relevant logs covering the failure window.
• Applicable device/routing operational outputs.
• Details of any recent changes or maintenance performed before the issue.
• A brief incident timeline, including when the issue was detected, when rollback was performed, and the result after rollback.

3. Information that should preferably be collected before rollback

Since rollback may remove the original failure condition and make the issue difficult to reproduce, we recommend capturing the following minimum information before initiating the rollback, whenever the operational situation permits:

1. Failure timestamp + timezone
2. Test/monitor name or ID
3. Source/Test Agent and destination
4. Exact failure/error message
5. Screenshot or export of the failed test result
6. Test Agent status and software version
7. Relevant Test Agent/application logs around the failure
8. Relevant network/routing operational state at the time of failure
9. Any alarms/events observed
10. Details of recent changes or configuration activity

We understand that in a production environment, maintaining service availability takes precedence over information collection. Therefore, the above collection should not delay an urgent rollback.

Note: Troubleshooting an issue while it is actively occurring generally provides the best opportunity to identify the exact failure condition, perform live troubleshooting, and establish a definitive RCA. However, we fully understand that service continuity and minimizing production impact are the primary priorities.

In scenarios where rollback is required before detailed troubleshooting can be performed, the minimum information outlined above will provide JTAC with the best available evidence to analyze the issue, determine the possible cause, and provide an RCA. Where additional data or deeper product-level investigation is required, JTAC will also engage the relevant Engineering teams for further analysis and assistance.