Description

This article explains what all logs to collect in case the LSP stops getting updated on the UI and the current path starts showing as 0.0.0.0 for all the LSPs.

Symptoms

LSP path stops getting updated on the UI.

Current path starts showing as 0.0.0.0.

All data from before this issue is still seen on the UI.

Solution

In this case, there will be error messages seen in the atom-db and the toposerver logs which indicate that the issue happens when the communication between the etcd or the api-serverl ogs similar to what is mentioned below:


Error messages seen in the toposerver logs are:


{"@timestamp":"2025-08-13T06:41:46.822279+00:00", "host":"ns-toposerver-6d347berd6-fgdh6", "level":"err", "facility":"local6", "syslog-tag":"TopoServer:", "source":"TopoServer", "msg":" ExecuteBatch - Error executing transaction: insert_pcs_lsp_event_into_db2 - ERROR: cannot execute INSERT in a read-only transaction"}


Similar error messages can be seen in the atom-db logs as well. You can collect these as mentioned below:


kubectl exec -it -n common atom-db-0 -- bash

cd pgdata/pgroot/pg_log/

grep -i "link_event" * | more

grep -i "pcs_lsp_event" * | more

 

kubectl exec -it -n common atom-db-1 -- bash

cd pgdata/pgroot/pg_log/

grep -i "link_event" * | more

grep -i "pcs_lsp_event" * | more

 

kubectl exec -it -n common atom-db-2 -- bash

cd pgdata/pgroot/pg_log/

grep -i "link_event" * | more

grep -i "pcs_lsp_event" * | more


Error messages seen here would be as follows:


oot@atom-db-1:/home/postgres/pgdata/pgroot/pg_log# grep -i "link_event" * | more

postgresql-1.csv:2025-08-11 00:02:16.571 UTC,"northstar","ns_pcs",95673,"10.x.x.x:43404",6895398d.17591,33780,"INSERT",2025-08-07 23:41:01 UTC,7/33790,0,ERROR,25006,"cannot execute INSERT in a read-only transaction",,,,,,"INSERT INTO link_event (timeid, topo_id, dat

e_str, ts, link_id, ns_link_id, data) VALUES ($1, $2, $3, $4, $5, $6, $7)",,,""


This needs to be looked into as to why the communication issue has happened.


For that, please collect the following information as well as what is mentioned above. Please open a ticket with JTAC for further analysis:

kubectl get pods -A | grep etcd - take the name of th e etcd master pod

kubectl logs -n kube-system <name of the etcd master pod> >/tmp/etcd.log

kubectl logs -n common atom-db-0 >/tmp/atom-db-0.log

kubectl logs -n common atom-db-1 > /tmp/atom-db-1.log

kubectl logs -n common atom-db-2 > /tmp/atom-db-2.log



Modification History

2025-09-19 : Article Created