In Paragon Automation 23.1, we may encounter that the audit logs page won't be displaying any data.

When the logs are checked we may notice that the audit log connectivity with the DB would be lost.

time="2024-10-28T07:26:10Z" level=error msg="error while handling request" duration=2h1m25.491547143s duration_ms=7285491 error="rpc error: code = Internal desc = failed to store audit data: operation:\"PATCH /traffic-engineering/api/topology/v2/1/te-lsps/146928/\" description:\"modify lsp TEST-LSP-BASELINE\" status:SUCCESS object_id:\"146928\" object_name:\"TEST-LSP-BASELINE\" object_type:\"lsp\" logged_time:1730093084661 details:\"[{\\\"op\\\":\\\"add\\\",\\\"path\\\":\\\"/plannedProperties/design/adminGroups/attributeIncludeAll\\\",\\\"value\\\":2048},{\\\"op\\\":\\\"add\\\",\\\"path\\\":\\\"/plannedProperties/design/adminGroups/attributeExclude\\\",\\\"value\\\":2150629632},{\\\"op\\\":\\\"add\\\",\\\"path\\\":\\\"/plannedProperties/color\\\",\\\"value\\\":11}]\": failed to retrieve DB connection: context canceled" method_name=/auditlog.AuditlogRPCService/CreateLogEntry mt_sp_id=3be2165c-64c8-462f-96ca-cd356991a591 request-body="{\"log\":{\"operation\":\"PATCH /traffic-engineering/api/topology/v2/1/te-lsps/146928/\",\"op_desc\":\"modify lsp TEST-LSP-BASELINE\",\"status\":\"SUCCESS\",\"object_id\":\"146928\",\"object_name\":\"TEST-LSP-BASELINE\",\"object_type\":\"lsp\",\"logged_time\":\"1730093084661\",\"details\":\"[{\\\"op\\\":\\\"add\\\",\\\"path\\\":\\\"/plannedProperties/design/adminGroups/attributeIncludeAll\\\",\\\"value\\\":2048},{\\\"op\\\":\\\"add\\\",\\\"path\\\":\\\"/plannedProperties/design/adminGroups/attributeExclude\\\",\\\"value\\\":2150629632},{\\\"op\\\":\\\"add\\\",\\\"path\\\":\\\"/plannedProperties/color\\\",\\\"value\\\":11}]\"}}" response-body="<nil>" traceid=req-c73cc14b-1a7d-495d-a8ff-2934eb7e0a34 type=APPLOG user_id=63be7546-a3c8-4040-be50-a570a973020c username=admin

time="2024-10-28T08:33:39Z" level=error msg="error while handling request" duration=4m59.975008261s duration_ms=299975 error="failed to retrieve DB connection: context canceled" method_name=/auditlog.AuditlogService/ListLog mt_sp_id=3be2165c-64c8-462f-96ca-cd356991a591 request-body="{\"spec\":{\"size\":\"50\",\"sortby\":\"(logged_time(descending))\"}}" response-body="<nil>" traceid=5792f06f-c0ec-4652-b16b-86e331bd17c2 type=APPLOG user_id=167c7dc2-426e-4400-bd92-b0f2d0d845b4 username=admin

To identify the exact root cause, we may need to collect below info at the time of problematic state.

• Capture auditlog log

#kubectl logs -n auditlog auditlog-7fcb5d4f5f-5dprg

• Capture atom-db logs for all the three pods.

#kubectl logs -n common atom-db-0

#kubectl logs -n common atom-db-1

#kubectl logs -n common atom-db-2

• Validate the connectivity of DB connectivity

Follow below steps:

• Describe  the pod audit logs

#kubectl describe pod auditlog-7fcb5d4f5f-5dprg -n auditlog

• Run the below command to get environment's auditlog dataname 

#kubectl get secret auditlog-database -n auditlog -o jsonpath="{.data.name}" | base64 --decode

Eg: 

root@davinci-master:~# kubectl get secret auditlog-database -n auditlog -o jsonpath="{.data.name}" | base64 --decode

auditlog

root@davinci-master:~# 

• get the ip of auditlog pod & ip for atom-db 

#kubectl get pod -n auditlog -owide

#kubectl get svc -A -owide | grep atom-db

 Eg: 

auditlogroot@davinci-master:~# kubectl get pod -n auditlog -owide

NAME                                READY  STATUS     RESTARTS  AGE    IP              NODE          NOMINATED NODE  READINESS GATES

auditlog-7fcb5d4f5f-5dprg           1/1    Running    0         8d      10.244.176.219  172.16.18.23  <none>          <none>

auditlog-purge-cron-28867680-mkb5r  0/1    Completed  0         7h45m  10.244.86.239   172.16.18.20  <none>          <none>

root@davinci-master:~#

 root@davinci-master:~# kubectl get svc -A -owide | grep atom-db

common                 atom-db                          ClusterIP      10.101.26.142   <none>        5432/TCP                                                  8d     <none>

• now get the username and password of the postgres database

#kubectl get secret postgres.atom-db.credentials -n common -o jsonpath="{.data.username}" | base64 --decode

#kubectl get secret postgres.atom-db.credentials -n common -o jsonpath="{.data.password}" | base64 --decode

Eg: 

root@davinci-master:~# kubectl get secret postgres.atom-db.credentials -n common -o jsonpath="{.data.username}" | base64 --decode

Postgres

root@davinci-master:~#  

root@davinci-master:~# kubectl get secret postgres.atom-db.credentials -n common -o jsonpath="{.data.password}" | base64 --decode

yIbL9lRCtrwDGiiGuSnnQuMvzpQpe6mq55Ruf5m5MdKazZYT3QkGZlieBUukv8xp

root@davinci-master:~# 

• connect with postegres database using psql and after connection just check whether postgres has active connection with auditlog pod

#psql -h <atom-db ip> -p 5432 -U <username> -d postgres

#SELECT client_addr, usename, datname, state, query FROM pg_stat_activity WHERE client_addr = ‘<auditlog-pod ip>' AND usename = 'auditlog';

 Eg:

root@davinci-master:~# psql -h 10.101.26.142 -p 5432 -U postgres -d postgres

Password for user postgres: 

psql (12.20 (Ubuntu 12.20-0ubuntu0.20.04.1), server 12.6 (Ubuntu 12.6-1.pgdg18.04+1))

SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)

Type "help" for help.

postgres=# SELECT client_addr, usename, datname, state, query FROM pg_stat_activity WHERE client_addr = '10.244.176.219' AND usename = 'auditlog';

 client_addr  | usename | datname | state |     query      

----------------+----------+----------+-------+-----------------

10.244.176.219 | auditlog | auditlog | idle | COMMIT

10.244.176.219 | auditlog | auditlog | idle | LISTEN "events"

(2 rows)