After upgrading lab PTX10K to the fixed version (FFT JSU fixes) it has stopped responding to the GRPC and Openssl connections and is throwing connection refused. ssaroha@Shreyanshs-MacBook-Pro-2 ~ % openssl s_client -showcerts -connect usgrx107-bbisp-gw3.gns-devlab.apple.com:57400 connect: Connection refused connect:errno=61 JSD socket (port 57400) is dead. gnsadmin@usgrx107-bbisp-gw3-re0> show system connections | grep 57400 tcp 0 0 128.0.0.4:57400 128.0.0.4:9000 ESTABLISHED 15476/l2cpd tcp 0 0 128.0.0.4:9000 128.0.0.4:57400 ESTABLISHED 14933/dfwd-junos-re Live JSD traces do not generate anything when queries are sent which is another indication that application is in bad state: {master} gnsadmin@usgrx107-bbisp-gw3-re0> show trace application jsd live ^C[abort] {master} gnsadmin@usgrx107-bbisp-gw3-re0> No issues after performing the same EVO upgrade on pilot PTX10K. Initial debugging was done (attached). Generally, in majority of such cases the culprit is always either a missing certificate, wrong certificate location or wrong certificate name being used. Seems like that is not the case here. Next steps: 1) deactivate GRPC configuration and then re-activate back (if useful syslog will be generated) 2) Collect /var/log (EXCLUDING TRACES!!!) 3) Analyze logs 4) If necessary get JSD live core for analysis
All GRPC and OpenSSL queries fail after a reboot / reset event:
gnsadmin@usgrx107-bbisp-gw3-re0> show system connections | grep 57400
tcp 0 0 128.0.0.4:57400 128.0.0.4:9000 ESTABLISHED 15476/l2cpd
tcp 0 0 128.0.0.4:9000 128.0.0.4:57400 ESTABLISHED 14933/dfwd-junos-re
The following logs were seen poin4ng to a missing certificate causing JSD to fail to come up:
Dec 12 23:57:10.830 usgrx107-bbisp-gw3-re0 jsd[14521]: %DAEMON-1-
LIBJSD_GRPC_SERVER_RESTART_ON_CONFIG_CHANGE: gRPC Server will restart for config changes to take effect
Dec 12 23:57:10.838 usgrx107-bbisp-gw3-re0 jsd: %USER-3-JSD_SERVER_START_FAILURE:
/var/db/certs/common/cer4fica4on-authority/saroha-919_1-ca1.cert cer4ficate authority file not found
Feb 5 23:24:27.362 usgrx107-bbisp-gw3-re0 jsd[17200]: %DAEMON-1-
Feb 5 23:24:27.368 usgrx107-bbisp-gw3-re0 jsd: %USER-3-JSD_SERVER_START_FAILURE:
Steps which led to this state:
a) 'saroha-919' chain was loaded and referenced in grpc configuration:
Sep 19 21:46:01.475 usgrx107-bbisp-gw3-re0 mgd[13717]: UI_CMDLINE_READ_LINE: User 'ggnsdcext', command 'request security pki ca-certificate caprofile-group load ca-group-name saroha-919 filename /var/tmp/saroha-919.chain.pem '
set system services extension-service request-response grpc ssl mutual-authentication certificate-authority saroha-919
b) Later CA certificate were cleared and new CA chain loaded:
Nov 6 23:18:38.348 usgrx107-bbisp-gw3-re0 mgd[31093]: UI_CMDLINE_READ_LINE: User 'gnsadmin', command 'clear security pki ca-certificate all '
Nov 6 23:19:16.592 usgrx107-bbisp-gw3-re0 mgd[31093]: UI_CMDLINE_READ_LINE: User 'gnsadmin', command 'request security pki ca-certificate caprofile-group load ca-group-name Apple-CA-2023-11-06 filename /var/tmp/certificate.chain-2.pem '
Underlying problem is that 'clear security pki ca-cer4ficate all ' command only deletes CA certificate on the disk but leaves whole /security/pki configura4on stanza untouched when used to clear for CA chain group. As a result, CA certificate doesn’t exist on the disk, but CA chain group name is s4ll configured and referenced by GRPC.
Although a new CA chain ‘certificate’ was loaded it was not configured and referenced under GRPC
(saroha-919 was still configured under GRPC). Any trigger (JSD restart from CLI, box reboot or upgrade, most likely RE switchover) causing JSD restart will make it fail with ‘JSD_SERVER_START_FAILURE’ syslog complaining of a missing certificate.
a) Instead of ‘clear security pki ca-cer4ficate all’ we need to use ‘clear security pki ca-cer0ficate ca-profile-group
(name)’. It will clear /security/pki and delete CA certificate on the disk for the specific chain name provided in
the command.
b) Try to avoid manual (delete) clean-up of /security/pki configura4on stanza
c) Make sure that CA group name referenced from GRPC is the right one: it does exist as well as certificates on the
disk (/var/db/certs/common/cer4fica4on-authority)