ECMP issue on MX - all protocol next-hop not installed in the multi-path

Description

This article explains the ECMP issue where all only 5 PNH are being installed in multipath out of 8 PNH

Symptoms

xyz@abc-rpi-wan1-pe> show route table prod.inet .0 10.174.0.0/20 | no-more

prod.inet.0: 5031 destinations, 53095 routes (5031 active, 0 holddown, 0 hidden)
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

10.174.0.0/20 @[BGP/170] 3w0d 11:51:23, localpref 100, from 10.20.48.5
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.59
[BGP/170] 3w0d 11:51:23, localpref 100, from 10.20.48.6
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.59
[BGP/170] 3w0d 11:51:23, localpref 100, from 10.21.192.21
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.59
[BGP/170] 3w0d 11:51:23, localpref 100, from 10.21.192.22
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.59
[BGP/170] 3w0d 11:51:35, localpref 100, from 10.20.48.5
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.37 via ae0.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.38.97.37->10.20.3.95
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.61
[BGP/170] 3w0d 11:51:35, localpref 100, from 10.20.48.6
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.37 via ae0.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.38.97.37->10.20.3.95
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.61
[BGP/170] 3w0d 11:51:35, localpref 100, from 10.21.192.21
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.37 via ae0.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.38.97.37->10.20.3.95
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.61
[BGP/170] 3w0d 11:51:35, localpref 100, from 10.21.192.22
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.37 via ae0.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.38.97.37->10.20.3.95
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.61
[BGP/170] 3w0d 11:51:27, localpref 100, from 10.20.48.5
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.163
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.20.3.91->10.20.3.105
[BGP/170] 3w0d 11:51:27, localpref 100, from 10.20.48.6
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.163
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.20.3.91->10.20.3.105
[BGP/170] 3w0d 11:51:27, localpref 100, from 10.21.192.21
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.163
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.20.3.91->10.20.3.105
[BGP/170] 3w0d 11:51:27, localpref 100, from 10.21.192.22
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.163
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.20.3.91->10.20.3.105
[BGP/170] 3w0d 11:51:27, localpref 100, from 10.20.48.5
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91->10.20.3.103
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.165
[BGP/170] 3w0d 11:51:27, localpref 100, from 10.20.48.6
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91->10.20.3.103
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.165
[BGP/170] 3w0d 11:51:27, localpref 100, from 10.21.192.21
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91->10.20.3.103
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.165
[BGP/170] 3w0d 11:51:27, localpref 100, from 10.21.192.22
AS path: 4281335818 4281466980 4281476980 I, validation-state: unverified
> to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91->10.20.3.103
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.165
[BGP/170] 1w4d 22:25:07, localpref 100, from 10.20.48.5
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.114
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.128
[BGP/170] 1w4d 22:17:18, localpref 100, from 10.20.48.6
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.114
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.128
[BGP/170] 1w5d 22:53:58, localpref 100, from 10.21.192.21
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.114
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.128
[BGP/170] 1w5d 22:53:58, localpref 100, from 10.21.192.22
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.114
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.128
[BGP/170] 22:27:35, localpref 100, from 10.20.48.5
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.128
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 1w5d 22:31:34, localpref 100, from 10.20.48.6
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.128
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 1w5d 22:31:34, localpref 100, from 10.21.192.21
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.128
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 1w5d 22:31:34, localpref 100, from 10.21.192.22
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.128
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 2w4d 10:17:27, localpref 100, from 10.20.48.5
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.116
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 2w4d 10:17:27, localpref 100, from 10.20.48.6
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.116
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 2w4d 10:17:27, localpref 100, from 10.21.192.21
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.116
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 2w4d 10:17:27, localpref 100, from 10.21.192.22
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.116
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 2w4d 10:17:27, localpref 100, from 10.20.48.5
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.116
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 2w4d 10:17:27, localpref 100, from 10.20.48.6
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.116
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 2w4d 10:17:27, localpref 100, from 10.21.192.21
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.116
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
[BGP/170] 2w4d 10:17:27, localpref 100, from 10.21.192.22
AS path: 4281466890 4281466980 4281476980 I, validation-state: unverified
> to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf11-rpi-wan2-pe
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.116
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.126
#[Multipath/255] 1w4d 22:25:07, metric2 2000
to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.59
> to 10.38.97.37 via ae0.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan2-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.38.97.37->10.20.3.95
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.61
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan3-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.163
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.20.3.91->10.20.3.105
to 10.20.3.91 via et-1/1/6.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf02-rpi-wan4-pe
to 10.38.97.58 via ae1.0, label-switched-path Bypass->10.20.3.91->10.20.3.103
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.58->10.20.3.165
to 10.38.97.63 via ae2.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.38.97.58 via ae1.0, label-switched-path routerG16-rpi-wan1-pe-to-smf13-rpi-wan1-pe
to 10.20.3.91 via et-1/1/6.0, label-switched-path Bypass->10.38.97.63->10.20.3.114
to 10.38.97.63 via ae2.0, label-switched-path Bypass->10.38.97.58->10.20.3.128

abc@abc-rpi-wan1-pe> show route forwarding-table destination 10.174.0.0/20
Routing table: default.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
default user 2 0:1c:73:0:0:77 ucst 348 5 fxp0.0
default perm 0 rjct 51 1

Routing table: __juniper_services__.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 515 2

Routing table: __master.anon__.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 rjct 534 1

Routing table: prod.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
10.174.0.0/20 user 0 ulst 1049108 7
indr 1049261 6
ulst 1050077 2
10.20.3.91 Push 16 4353 2 et-1/1/6.0
10.38.97.63 Push 16, Push 303446(top) 732 2 ae2.0 -
10.38.97.58 Push 16, Push 299702(top) 4712 2 ae1.0 -
10.20.3.91 Push 16 1772 2 et-1/1/6.0 -
indr 1048748 6
ulst 1050018 2
10.38.97.37 Push 16, Push 343000(top) 2791 2 ae0.0
10.38.97.58 Push 16, Push 333173(top) 2588 2 ae1.0
10.38.97.58 Push 16, Push 305801(top) 3530 2 ae1.0 -
10.20.3.91 Push 16, Push 461559(top) 1362 2 et-1/1/6.0 -
indr 1048922 6
ulst 1049301 2
10.20.3.91 Push 16, Push 460508(top) 2965 2 et-1/1/6.0 -
10.38.97.58 Push 16, Push 333584(top) 2966 2 ae1.0
10.38.97.63 Push 16, Push 303418(top) 3692 2 ae2.0
10.20.3.91 Push 16, Push 125035, Push 977692(top) 1501 2 et-1/1/6.0 -
indr 1048785 6
ulst 1049822 2
10.20.3.91 Push 16, Push 460419(top) 656 2 et-1/1/6.0 -
---(more)---

10.38.97.58 Push 16, Push 333013(top) 915 2 ae1.0 -
10.38.97.58 Push 16, Push 24051(top) 3188 2 ae1.0
10.20.3.91 Push 16, Push 977692(top) 3730 2 et-1/1/6.0
indr 1048589 6
ulst 1050020 2
10.38.97.63 Push 16, Push 336994(top) 2045 2 ae2.0 -
10.38.97.58 Push 16, Push 32468(top) 2975 2 ae1.0 -
10.20.3.91 Push 16, Push 995104(top) 3531 2 et-1/1/6.0 -
10.38.97.63 Push 16, Push 663957, Push 324124(top) 4362 2 ae2.0

SMPC0(abc-rpi-wan1-pe vty)# show nh id 1049108 recursive
1049108(Unilist, IPv4, ifl:0:-, pfe-id:5)
1049261(Indirect, IPv4, ifl:113:et-1/1/6.0, pfe-id:5, i-ifl:0:-)
1049492(Unilist, IPv4, ifl:0:-, pfe-id:5)
4290(Unicast, IPv4->MPLS, ifl:113:et-1/1/6.0, pfe-id:5)
732(Unicast, IPv4->MPLS, ifl:78:ae2.0, pfe-id:0)
2650(Unicast, IPv4->MPLS, ifl:77:ae1.0, pfe-id:0)
1772(Unicast, IPv4->MPLS, ifl:113:et-1/1/6.0, pfe-id:5)
1048748(Indirect, IPv4, ifl:113:et-1/1/6.0, pfe-id:5, i-ifl:0:-)
1050078(Unilist, IPv4, ifl:0:-, pfe-id:5)
3725(Unicast, IPv4->MPLS, ifl:113:et-1/1/6.0, pfe-id:5)
2588(Unicast, IPv4->MPLS, ifl:77:ae1.0, pfe-id:0)
4394(Unicast, IPv4->MPLS, ifl:76:ae0.0, pfe-id:0)
1362(Unicast, IPv4->MPLS, ifl:113:et-1/1/6.0, pfe-id:5)
1048922(Indirect, IPv4, ifl:113:et-1/1/6.0, pfe-id:5, i-ifl:0:-)
1049974(Unilist, IPv4, ifl:0:-, pfe-id:5)
1246(Unicast, IPv4->MPLS, ifl:113:et-1/1/6.0, pfe-id:5)
3494(Unicast, IPv4->MPLS, ifl:78:ae2.0, pfe-id:0)
3999(Unicast, IPv4->MPLS, ifl:78:ae2.0, pfe-id:0)
4068(Unicast, IPv4->MPLS, ifl:113:et-1/1/6.0, pfe-id:5)
1048785(Indirect, IPv4, ifl:113:et-1/1/6.0, pfe-id:5, i-ifl:0:-)
1049859(Unilist, IPv4, ifl:0:-, pfe-id:5)
3519(Unicast, IPv4->MPLS, ifl:113:et-1/1/6.0, pfe-id:5)
915(Unicast, IPv4->MPLS, ifl:77:ae1.0, pfe-id:0)
4046(Unicast, IPv4->MPLS, ifl:77:ae1.0, pfe-id:0)
3730(Unicast, IPv4->MPLS, ifl:113:et-1/1/6.0, pfe-id:5)
1048589(Indirect, IPv4, ifl:78:ae2.0, pfe-id:0, i-ifl:0:-)
1049579(Unilist, IPv4, ifl:0:-, pfe-id:0)
4086(Unicast, IPv4->MPLS, ifl:78:ae2.0, pfe-id:0)
2975(Unicast, IPv4->MPLS, ifl:77:ae1.0, pfe-id:0)
2876(Unicast, IPv4->MPLS, ifl:113:et-1/1/6.0, pfe-id:5)
4362(Unicast, IPv4->MPLS, ifl:78:ae2.0, pfe-id:0)

Solution

https://gnats.juniper.net/web/default/1668481#summary_tab

Root-Cause
When creating rt_multipath, nexthops of BGP routes that are eligible for rt_multipath are merged into a single nexthop. After the nexthops are merged additional flags are set for the merged nexthop in rtg_flags field. During merge, first merged nexthop is checked to see if it already contains member that needs to be merged. If not found, it is added to merged nexthop and flags are updated. When same nexthop is searched again, it will not find the existing nexthop as flags are different. This causes duplicate nexthops to be added in the merged nexthop. This causes multipath to prematurely hit maximum-ecmp limit.
Fix is to update the flags and then compare the member in the merged nexth

Detectability-Info
'show route' can give an idea of this problem. Under 'Multipath' desired number of NHs would not be seen.
Service-Restoration
not possible

Modification History

2024-04-19 : Article Created