Description

This article provides information on how to troubleshoot generic T3 interface related issues. It also provides certain verification procedures and information about some common T3 alarms.

Symptoms

How to troubleshoot T3 interfaces

Solution

 

Troubleshooting procedure :
  1. For a brief status overview of the T3 interfaces on the router, use the show interface terse t3* command: 
    user@router> show interfaces terse t3*
Interface  Admin  Link  Proto Local Remote
t3-1/0/0   down  down < administratively disabled
t3-1/0/0.0 up    down   inet 1.1.1.1/30
t3-1/0/1   up    down
t3-1/0/1.0 up    down   inet 2.2.2.2/30 < link layer down
t3-1/0/2   up    up
t3-1/0/2.0 up    up     inet 3.3.3.3/30 < link layer up
t3-1/0/3   up    down
    If the T3 interface is administratively disabled, the physical interface has a status of admin down and link down and the logical interface has a status of admin up and link down .

    If the T3 interface is not administratively disabled and the link layer between the local router and the remote router is down and not functioning, both the physical and logical interface have a status of admin up and link down .

    If the link layer between the local router and the remote router is up and running, both the physical and logical interface have a status of admin up and link up .

  • To display the statistics for a physical interface, use the monitor interface interface-name command. This commands also checks and displays common interface failures, whether the loopback is detected, and increases in framing errors. Information from this command can narrow down possible causes of an interface issue.

    Note : if you are accessing the router from the console connection, make sure to set the CLI terminal type by using the ‘set cli terminal’ command. 
    user@host> monitor interface t3-1/0/0
router                       Seconds: 78                  Time: 21:44:15
Interface: t3-1/0/0, Enabled, Link is Down
Encapsulation: Cisco-HDLC, Keepalives, Speed: T3
Traffic statistics:                              Current Delta
  Input bytes:              0 (0 bps)                      [0]
  Output bytes:           207 (184 bps)                  [184]
  Input packets:            0 (0 pps)                      [0]
  Output packets:           9 (1 pps)                      [8]
Encapsulation statistics:
  Input keepalives:         0                              [0]
  Output keepalives:        9                              [8]
Error statistics:
  Input errors:             0                              [0]
  Input drops:              0                              [0]
   Input framing errors :     9                              [8]
  CCV                       0                              [0]
Interface warnings:
  o Received keepalive count is zero
  o Framing errors, check FCS, scrambling and subrate configuration
Next='n', Quit='q' or ESC, Freeze='f', Thaw='t', Clear='c', Interface='i'
    If the framing errors increase, check the FCS, scrambling, and subrate configurations. If the configuration are correct, check the cabling to the router and have the carrier verify the integrity of the line.

    If the input errors increase, check the cabling to the router and have the carrier verify the integrity of the line.

  • To display the status of a specific T3 interface, use the show interface t3-x/x/x command: 
    user@host> show interfaces t3-1/0/0
Physical interface: t3-1/0/0, Enabled,  Physical link is Down
  Interface index: 9, SNMP ifIndex: 10
  Link-level type: Cisco-HDLC, MTU: 4474, Clocking: Internal
  Speed: T3, Loopback: None, CRC: 16, Mode: C/Bit parity
  Device flags   : Present Running Down
  Interface flags: Hardware-Down Link-Layer-Down Point-To-Point SNMP-Traps
  Link flags     : Keepalives
  Keepalive Input: 116 (00:02:32 ago), Output: 185 (00:00:02 ago)
  Input rate     : 0 bps (0 pps), Output rate: 0 bps (0 pps)
  Active alarms  : LOF, LOS
  Active defects : LOF, LOS
  Logical interface t3-1/0/0.0 (Index 12) (SNMP ifIndex 32)
    Flags: Device-down Point-To-Point SNMP-Traps, Encapsulation: Cisco-HDLC
    Protocol inet, MTU: 4470
      Addresses, Flags: Dest-route-down Is-Preferred Is-Primary
        Destination: 1.1.1.0/30, Local: 1.1.1.1
    If the first line indicates that the Physical link is up, it means that the physical link is healthy and can pass packets.

    If the first line indicates that the Physical link is down, it means that the physical link is unhealthy and cannot pass packets. To display more extensive information about the T3 interface, if the physical link is down, use the show interface t3-x/x/x extensive command.

    Look at the active alarms and active defects for the T3 interface and troubleshoot the T3 media accordingly. Refer to the Additional Notes section at the end of this article for a brief explanation of the DS3 alarms. 
    user@router> show interfaces t3-1/0/0 extensive
Physical interface: t3-1/0/0, Enabled, Physical link is Down
  Interface index: 9, SNMP ifIndex: 10
  Link-level type: Cisco-HDLC, MTU: 4474, Clocking: Internal
  Speed: T3, Loopback: None, CRC: 16, Mode: C/Bit parity
  Device flags   : Present Running Down
  Interface flags: Hardware-Down Link-Layer-Down Point-To-Point SNMP-Traps
  Link flags     : Keepalives
  Keepalive statistics:
    Input : 116 (last seen 00:02:59 ago)
    Output: 187 (last seen 00:00:09 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :                 2552                    0 bps
   Output bytes  :                 3703                    0 bps
   Input  packets:                  116                    0 pps
   Output packets:                  161                    0 pps
  Input errors: - Input errors
    Errors: 0, Drops: 0, Framing errors: 229, Policed discards: 1
    L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0
    SRAM errors: 0, HS link CRC errors: 0
  Output errors: - Output errors
    Carrier transitions: 4, Errors: 0, Drops: 0, Aged packets: 0
  Active alarms  : LOF, LOS - DS3 active alarms and defects
  Active defects : LOF, LOS
  DS3 Media:            Seconds        Count  State  - T3 media-specific errors
    PLL Lock                  0            0  OK
    Reframing               273            2  Defect Active
    AIS                       0            0  OK
    LOF                     273            2  Defect Active
    LOS                     273            2  Defect Active
    IDLE                      0            0  OK
    YELLOW                    0            0  OK
    BPV                       0            0
    EXZ                       0            0
    LCV                     275     18022125
    PCV                       0            0
    CCV                       0            0
    LES                     275
    PES                     273
    PSES                    273
    CES                     273
    CSES                    273
    SEFS                    273
    UAS                     277
  HDLC configuration:
    Policing bucket: Disabled
    Shaping bucket : Disabled
    Giant threshold: 4484, Runt threshold: 3
  DSU configuration:
    Compatibility mode: None, Scrambling: Disabled, Subrate: Disabled
    FEAC loopback: Inactive, Response: Disabled, Count: 0
    BERT time period: 10 seconds, Elapsed: 0 seconds
    Algorithm: 2^3 - 1, Pseudorandom (1), Error rate: 10e-0
  PFE configuration:
    Destination slot: 1, Stream number: 0, PLP byte: 1 (0x00)
    COS transmit queue bandwidth:
      Queue0: 95, Queue1: 0, Queue2: 0, Queue3: 5
    COS weighted round robin:
      Queue0: 95, Queue1: 0, Queue2: 0, Queue3: 5
  Logical interface t3-1/0/0.0 (Index 12) (SNMP ifIndex 32)
    Flags: Device-down Point-To-Point SNMP-Traps, Encapsulation: Cisco-HDLC
    Protocol inet, MTU: 4470, Flags: None
      Addresses, Flags: Dest-route-down Is-Preferred Is-Primary
        Destination: 1.1.1.0/30, Local: 1.1.1.1, Broadcast: Unspecified
  • If you suspect a hardware issue with the T3 interface, create a physical loopback at the port by connecting the transmit port to the receive port. To display the status of the interface, use the show interface t3-x/x/x command.

    Example output: 
    user@host> show interfaces t3-1/0/0
Physical interface: t3-1/0/0, Enabled,  Physical link is UP
  Interface index: 9, SNMP ifIndex: 10
  Link-level type: Cisco-HDLC, MTU: 4474, Clocking: Internal
  Speed: T3, Loopback: None, CRC: 16, Mode: C/Bit parity
  Device flags   : Present Running Loop-Detected
  Interface flags: Hardware-Down Link-Layer-Down Point-To-Point SNMP-Traps
  Link flags     : Keepalives
  Keepalive Input: 116 (00:02:32 ago), Output: 185 (00:00:02 ago)
  Input rate     : 0 bps (0 pps), Output rate: 0 bps (0 pps)
  Active alarms  : LOF, LOS
  Active defects : LOF, LOS
  Logical interface t3-1/0/0.0 (Index 12) (SNMP ifIndex 32)
    Flags: Device-down Point-To-Point SNMP-Traps, Encapsulation: Cisco-HDLC
    Protocol inet, MTU: 4470
      Addresses, Flags: Dest-route-down Is-Preferred Is-Primary
        Destination: 1.1.1.0/30, Local: 1.1.1.1
    If the T3 interface is functioning properly, you should see a loop detected flag in the device flags section of the show interface command. Also, the monitor interface command shows that the input packet count matches the output packet count.

    For Cisco HDLC encapsulation, the input keepalive packet count also matches the output keepalive packet count. If you do not see the loop detected flag, move the T3 link to another T3 interface on the router and verify. To move the configuration of the existing T3 interface to the new interface, use the rename command under [edit interfaces] in the configuration mode: 
    [edit interfaces]
user@router# rename t3-1/0/0 to t3-1/0/1
    To test the operation of the DS3 PIC, without physically connecting the transmit port to the receive port, configure the local loopback T3 option in the configuration mode. Additionally, make sure that the clocking is set to internal: 
    [edit interfaces t3-1/0/1 t3-options]
user@router# set loopback local
    If T3 PIC is functioning properly, you should see a loop detected flag in the device flags section of the show interface command. Also, the monitor interface command shows that the input packet count matches the output packet count.

    For Cisco HDLC encapsulation, the input keepalive packet count will also match the output keepalive packet count. This test does not verify the T3 physical interface.
  • If you suspect an issue with the T3 link and want to isolate the location of the issue, insert loops at various points of the network and see whether the loopback on the router can be detected.

    Use the show interface command to check if the loop detected flag is displayed. Also, the monitor interface command shows that the input packet count matches the output packet count. For Cisco HDLC encapsulation, the input keepalive packet count also matches the output keepalive packet count.

    Note : You cannot use a normal ICMP ping in a loopback test for Cisco HDLC and PPP connections on a Juniper Networks router. When the router detects a loopback on the DS3 interface, it flags the local address as 'reject' in the routing table. As a result, a ping to the local address results in no route to host messages.

    Use the byepass ping to verify the connectivity: 
    user@host> show interfaces t3-1/0/0
Physical interface: t3-1/0/0, Enabled,  Physical link is UP
  Interface index: 9, SNMP ifIndex: 10
  Link-level type: Cisco-HDLC, MTU: 4474, Clocking: Internal
  Speed: T3, Loopback: None, CRC: 16, Mode: C/Bit parity
  Device flags   : Present Running Loop-Detected
  Interface flags: Hardware-Down Link-Layer-Down Point-To-Point SNMP-Traps
  Link flags     : Keepalives
  Keepalive Input: 116 (00:02:32 ago), Output: 185 (00:00:02 ago)
  Input rate     : 0 bps (0 pps), Output rate: 0 bps (0 pps)
  Active alarms  : LOF, LOS
  Active defects : LOF, LOS
  Logical interface t3-1/0/0.0 (Index 12) (SNMP ifIndex 32)
    Flags: Device-down Point-To-Point SNMP-Traps, Encapsulation: Cisco-HDLC
    Protocol inet, MTU: 4470
      Addresses, Flags: Dest-route-down Is-Preferred Is-Primary
        Destination: 1.1.1.0/30, Local: 1.1.1.1

user@router> ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss


user@router> show route 1.1.1.1
inet.0: 68700 destinations, 68700 routes (68699 active, 0
holddown, 1 hidden)
+ = Active Route, - = Last Active, * = Both
1.1.1.1/32      *[Local/0] 2d 23:12:11
                 Reject
    To create a loopback at the T3 interface, towards the remote router, configure the remote loopback T3 option in the configuration mode: 
    [edit interfaces t3-1/0/0 t3-options]
user@router# set loopback remote
  • If the router configuration is correct and the T3 link is clean, but the link layer is still down between the local router and the remote router, contact Juniper TAC for further assistance. If the issue is hardware related, contact Juniper TAC for hardware replacement.

Verification :

When the T3 interface or link issue is resolved, check the T3 connection by using the ping command.
  1. Type ping interface <local logical interface name> <far end IP address> bypass-routing size 4470 . By specifying the local logical interface name and the bypass-routing option, the ping uses the IP address of the local logical interface as the source address and leaves the router at the local interface under test: 
    user@router> ping interface t3-1/0/0.0 1.1.1.2 bypass-routing
size 44...
PING 1.1.1.2 (1.1.1.2): 4470 data bytes
4478 bytes from 1.1.1.2: icmp_seq=0 ttl=255 time=4.645 ms
4478 bytes from 1.1.1.2: icmp_seq=1 ttl=255 time=4.541 ms
4478 bytes from 1.1.1.2: icmp_seq=2 ttl=255 time=4.531 ms
4478 bytes from 1.1.1.2: icmp_seq=3 ttl=255 time=4.526 ms
4478 bytes from 1.1.1.2: icmp_seq=4 ttl=255 time=4.533 ms
^C
--- 1.1.1.2 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 4.526/4.555/4.645/0.045 ms
  2. to generate more ping traffic, type ping interface <local logical interface name> <far end IP address> bypass-routing size 4470 rapid count 9999 : 
    user@router> ping interface t3-1/0/0.0 1.1.1.2 bypass-routing size 44
PING 1.1.1.2 (1.1.1.2): 4470 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!^C
--- 1.1.1.2 ping statistics ---
686 packets transmitted, 685 packets received, 0% packet loss
round-trip min/avg/max/stddev = 4.486/4.839/175.933/6.577 ms
  3. To check for unusual input or output errors, use the show interface <interface name> extensive command. To clear the statistics counters, use the clear interfaces statistics <interface name> command.
Alarms :

Some of the DS3 alarms are as follows:

  • AIS : An incoming alarm indication signal (AIS) indicates an issue with the line upstream of the network equipment that is connected to the DS3 interface. Make the carrier check the equipment that is connected to the DS3 interface and trace the issue.

  • LOF : A loss of frame (LOF) signal indicates a loss of DS3 framing bits. Verify the framing format configuration (C-bit parity mode) and check the DS3 line. Bursts of line coding violations (LCVs) could indicate a timing issue.

  • LOS : A loss of signal (LOS) means that a signal could not be detected at the DS3 interface. Check the cables that are connected to the DS3 interface and have the carrier verify the integrity of the line.

  • IDLE : Idle indicates that the line has not been provisioned for service. Make the the carrier ensure that the line is provisioned for service.

  • Yellow : An incoming yellow alarm indicates that the equipment, which is connected to the DS3 interface, has an issue with the signal that it receives from the DS3 interface. Check the cable between this DS3 interface and the directly connected network equipment.

For more information about the DS3 error events (BPV, EXZ, LCV, PCV, CCV) and performance parameters (LES, PES, PSES, CES, CSES, SEFS, UAS), refer to RFC 2496 .