In Paragon Automation 24.1, when the image upload is tried, every time it fails as shown in the below image.



For troubleshooting, we need to start checking dpm logs

-       Dpm logs

o   kubectl logs -n ems dpm-<UUID>

• In this case we noticed “client disconnected” message. We can always cross check in the network if any firewall’s or other elements dropping the packets. Capture a TCPDUMP on the client end or from paragon end to verify why the connection is failing.

time="2025-03-07T11:31:59Z" level=error msg="Unable to upload \"junos-evo-install-acx-f-x86-64-23.2R2.21-EVO.iso\" to \"default\", MultipartUpload: upload multipart failed\n\tupload id: 2~TokL_Bpeu0kuX4Ggn3IG5wmcx4sZK14\ncaused by: ReadRequestBody: read multipart upload data failed\ncaused by: client disconnected" mt_sp_id=b978fac6-75b7-46b3-a2e2-a56c3556e469 traceid=6afdf546-3ee6-47dd-b9d5-5460e177c3b0 type=APPLOG user_id=b5e0502a-23ac-412c-b2e9-9a253959f88d username=admin

• In this case, we don’t see any firewall blocking the traffic. We could see there is TCP reset[rst] from Paragon end.
• In general all the web request will be first hitting ambassador and then from there it will forwarded to respective service based on the request. 
• So when checked ambassador details, we could see the master pod was restarting whenever we try to upload the image.

• From the logs we could see pod getting shutdown and When checked the pod describe, we notice terminated reason OOMkilled[OutOfMemory].

ACCESS [2025-04-03T07:04:03.772Z] "GET /api/v2/license/status HTTP/1.1" 200 - 0 1680 27 24 "10.253.68.120" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36""f70f02cf-a147-4eef-b9b0-e62e27daffb2" "10.252.224.184" "10.106.66.54:9000"

ACCESS [2025-04-03T07:04:03.772Z] "GET /api/v2/license/keys/contents HTTP/1.1" 200 - 0 9240 31 29 "10.253.68.120" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" "c36bfa9c-790c-4f30-ad1c-03aabc9e6e26" "10.252.224.184" "10.106.66.54:9000"

time="2025-04-03 07:04:08.3406" level=info msg="finished with error: signal: killed" func="github.com/datawire/dlib/dexec.(*Cmd).Wait" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:257" CMD=entrypoint PID=1 THREAD=/envoy dexec.pid=51

time="2025-04-03 07:04:08.3407" level=error msg="goroutine \"/envoy\" exited with error: signal: killed" func="github.com/datawire/dlib/dgroup.(*Group).goWorkerCtx.func1.1"file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:380" CMD=entrypoint PID=1 THREAD=/envoy

time="2025-04-03 07:04:08.3408" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238"CMD=entrypoint PID=1 THREAD="/watcher:shutdown_logger"

time="2025-04-03 07:04:08.3410" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238"CMD=entrypoint PID=1 THREAD=":shutdown_logger"

time="2025-04-03 07:04:08.3411" level=info msg="shutting down (gracefully)..." func="github.com/datawire/dlib/dgroup.(*Group).launchSupervisors.func1" file="/go/vendor/github.com/datawire/dlib/dgroup/group.go:238"CMD=entrypoint PID=1 THREAD="/ambex:shutdown_logger"

time="2025-04-03 07:04:08.3411" level=info msg="sending SIGINT" func="github.com/datawire/dlib/dexec.(*Cmd).Start.func1" file="/go/vendor/github.com/datawire/dlib/dexec/cmd.go:211" CMD=entrypoint PID=1 THREAD=/diagd

[2025-04-03 07:04:08 +0000] [36] [INFO] Handling signal: int

time="2025-04-03 07:04:08.3448" level=info msg="Memory Usage 0.12Gi (29%)\n   PID 1, 0.11Gi: busyambassador entrypoint \n   PID 36, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 \n   PID 40, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --port 8004 --kick kill -HUP 1 \n   PID 51, 0.11Gi (exited): envoy -c /ambassador/bootstrap-ads.json --base-id 0 --drain-time-s 600 -l error "func="github.com/emissary-ingress/emissary/v3/pkg/memory.(*MemoryUsage).Watch" file="/go/pkg/memory/memory.go:43" CMD=entrypoint PID=1 THREAD=/memory

[2025-04-03 07:04:08 +0000] [40] [INFO] Worker exiting (pid: 40)

[2025-04-03 07:04:08 +0000] [36] [INFO] Shutting down: Master

Pod describe snippet:

Containers:

 ambassador:

   Container ID:  containerd://a6e79c4e0bcdda9f1eb8a7f0232d618dde43d75b79146e7cb9d3a93f8022d1fe

   Image:         paragon-registry.local/s-artifactory.juniper.net/northstar-docker-local/emissaryingress/emissary:3.8.2

   Image ID:       paragon-registry.local/s-artifactory.juniper.net/northstar-docker-local/emissaryingress/emissary@sha256:f548db91b4b11e6c35327c7f3bf8410a5a9ef893b4d3f6a6f76882cd52afee01

   Ports:         8080/TCP, 8443/TCP, 8877/TCP

   Host Ports:    0/TCP, 0/TCP, 0/TCP

   State:         Running

     Started:     Thu, 03 Apr 2025 18:24:55 +1100

   Last State:    Terminated

      Reason:      OOMKilled

     Exit Code:   1

     Started:     Thu, 03 Apr 2025 18:04:38 +1100

     Finished:    Thu, 03 Apr 2025 18:24:54 +1100

   Ready:         True

   Restart Count: 99

• Based the messages related to OOM, we may need to increase the resource for “emissary-ingress” pods.
• Steps for increasing the resources:
  • Edit the deployment
  • kubectl edit deployments.apps -n ambassador emissary-ingress
  • Change the memory value

Prior:

   Limits:

    cpu:    1

    memory:  400Mi

   Requests:

    cpu:    200m

    memory:   100Mi

Post:

   Limits:

    cpu:    1

    memory:  800Mi

   Requests:

    cpu:    200m

    memory:   200Mi

• • Save the changes. Post the respective “emissary-ingress” pods will be restarted and will load with new resource values.

• Post this we can see image is getting uploaded successfully.