Description

The pccd-service subsystem is not responding to management requests.

Symptoms

  • The pccd-service subsystem is not responding to management requests
    lab@dut> show path-computation-client status
    error: the pccd-service subsystem is not responding to management requests
    
    lab@dut> show version node all
    re0:
    --------------------------------------------------------------------------
    Hostname: dut
    Model: ptx10001-36mr
    Junos: 20.4X16.1-EVO
    Yocto: 2.2.1
    Linux Kernel: 4.8.28-WR2.2.1_standard-gc37d253
    JUNOS-EVO OS 64-bit [junos-evo-install-ptx-fixed-x86-64-20.4X16.1-EVO]
    
    
    

            pcep {
                pce NORTHSTAR-LID {
                    local-address 192.168.1.1;
                    destination-ipv4-address 192.168.1.3;
                    destination-port 4189;
                    pce-type active stateful;
                    lsp-provisioning;
                }
                pce PARAGON {
                    local-address 192.168.1.1;
                    destination-ipv4-address 192.168.1.2;
                    destination-port 4189;
                    pce-type active stateful;
                    lsp-provisioning;
                }
            }

        There will be a chance when the TCP session with Northstar is down but with Paragon it is up.
        Northstar Version is 6.2.0(Jan 5 2021) because of too many requests.

    tcp        0      0 x.x.x.x:42099     192.168.1.3:4189     ESTABLISHED 17884/pccd
    tcp        0      0 128.0.0.4:49746   128.0.0.4:705   ESTABLISHED 17884/pccd
    tcp       33      0 192.168.1.1:53203     192.168.1.2:4189     CLOSE_WAIT  17884/pccd

        The keepalives are getting missed between the Northstart and PTX.


    Oct 20 15:38:45.203588000 [17884] pccd_core_pce_transition_state(): Changing state of pce 'NORTHSTAR-LID' from PCE_STATE_UP to PCE_STATE_NEW <….> 
    Oct 20 15:38:45.237676000 [17884] pccd_core_send_message(): Could not send out message PCReport to PCE 'NORTHSTAR-LID' as it is in PCE_STATE_RECONNECT 
    Oct 20 15:38:49.206581000 [17884] pccd_core_pce_transition_state(): Changing state of pce 'NORTHSTAR-LID' from PCE_STATE_RECONNECT to PCE_STATE_CONNECTING 
    Oct 20 15:38:50.209594000 [17884] pccd_pcep_onmessage_cb(): onmessage objp: 0x1092680 objc: 1 msg_type:1 pce:'NORTHSTAR-LID' 
    Oct 20 15:38:50.212014000 [17884] pccd_pcep_onmessage_cb(): onmessage objp: 0xc7cad0 objc: 0 msg_type:2 pce:'NORTHSTAR-LID' 
    Oct 20 15:38:50.212055000 [17884] pccd_core_pce_transition_state(): Changing state of pce 'NORTHSTAR-LID' from PCE_STATE_CONNECTING to PCE_STATE_CONNECTED 
    Oct 20 15:38:50.212074000 [17884] pccd_core_pce_transition_state(): Changing state of pce 'NORTHSTAR-LID' from PCE_STATE_CONNECTED to PCE_STATE_SYNC
     Oct 20 15:38:50.218741000 [17884] pccd_core_report_lsp(): Sending message(s) objs 0x109a310 count: 97 to pce 'NORTHSTAR-LID' 
     Oct 20 15:38:50.223020000 [17884] pccd_core_report_lsp(): Sending message(s) objs 0x109a310 count: 96 to pce 'NORTHSTAR-LID' 
     Oct 20 15:38:50.224992000 [17884] pccd_core_report_lsp(): Sending message(s) objs 0x109a310 count: 10 to pce 'NORTHSTAR-LID' 
     Oct 20 15:38:50.225138000 [17884] pccd_core_report_empty_lsp(): Sending message objs 0x7ffd0291a4d0 count: 2 to pce 'NORTHSTAR-LID' 
     Oct 20 15:38:50.225173000 [17884] pccd_core_pce_transition_state(): Changing state of pce 'NORTHSTAR-LID' from PCE_STATE_SYNC to PCE_STATE_UP 
     Oct 20 15:39:00.206544000 [17884] pccd_core_set_new_main_pce_timer_cb(): Main PCE: 'NORTHSTAR-LID' already connected

    Workaround:

    lab@dut# deactivate protocols pcep 
    [edit]
    lab@dut# commit 
    commit complete
    [edit]
    lab@dut# rollback 1 
    load complete

    lab@dut:~$ netstat -an | grep 4189
    tcp        0      0 192.168.1.1:39323     192.168.1.2:4189     ESTABLISHED

    
    

Solution

  • Remove the unwanted/not reachable controllers to avoid unnecessary keepalives interaction with the device.
  • If the above doesnt help please deactivate/activate pcep and if that also doesnt help then please capture the below logs for further triaging.
           set protocols pcep pce ns1 traceoptions file pcep.log
           set protocols pcep pce ns1 traceoptions file size 100m
           set protocols pcep pce ns1 traceoptions flag all
           set protocols pcep traceoptions file pcep.log
           set protocols pcep traceoptions file size 100m
           set protocols pcep traceoptions flag all
  • Need pcap from both the server and router to confirm if the hellos are being received or not.

      tcpdump -i <Interface used for MX communication> -s 0 -w /var/tmp/northstar_dddmmyy.pcap -C 100 -Z root -W 100 port 4189

Modification History

2024-01-09 : Article Created