Description

We are unable to view the application data for the last 24 hours for WAN Edges.

The application data is visible for the last 7 days for WAN Edges. 

Symptoms

Application Data is not visible for 24 Hours for WAN Edges, however it is visible for a week.

Solution

The Mist WAN Assurance portal's 24-hour graph shows missing data as a result of four concurrent telemetry degradation conditions:


Primary — CIA Monitoring-Once Service Persistent Failure (since ≈ 2026-04-23):

The April 23 Mist config push introduced a policy-hit-count input stanza into the CIA periodic monitoring config (mist-wan-assurance-once-periodic.conf). This measurement path (stats/application-policy-hit-count/*) does not exist in SSR 6.2.8 LTS. As a result, every 3-minute telegraf --once run exits non-zero, and policy-hit-count metrics are permanently absent from the telemetry stream. Mist portal widgets or graph series that rely on this metric category show a continuous flat line or empty series from April 23 onward. Bandwidth metrics for the unused physical interface ge-0-7 (HA fabric, forwarding: false, Admin: n/a) are also dropped each cycle.


Secondary — WAN Assurance Telegraf Persistent Flush Backpressure (since 2026-04-30 10:48 UTC):

The mist-wan-assurance-continuous telegraf process (PID 27945), which reads WAN interface bandwidth and session statistics and forwards them to the CIA daemon's local HTTP endpoint, has been unable to complete its outputs.http flush within the configured 1-minute interval since April 30 10:48 UTC. All 940 journal entries for this service during the bundle window are W! [agent] ["outputs.http"] did not complete within its flush interval — approximately 14–20% of 1-minute collection intervals each day. No explicit buffer-overflow or metric-drop errors are present, but sustained backpressure of this magnitude indicates the CIA's HTTP endpoint is consistently slower to acknowledge than the telegraf flush timeout.


Secondary — MARS Metric Bus Broken Pipe Errors (since 2026-04-30 11:21 UTC):

A separate internal telegraf agent (PID 32421) that writes SSR metrics to MARS (Metrics Aggregation and Reporting Service) via <http://localhost/metrics/bank/publish> (unix socket /var/run/128technology/mars.sock) experienced 38 broken pipe and connection-reset errors between April 30 and bundle collection. Events escalated on May 4 morning: 7 events between 02:40 and 06:59 UTC. Each broken pipe drops the in-flight metrics batch silently. MARS feeds the internal SSR REST API and operational dashboards that supply some of the data consumed by the CIA stats pipeline.


Secondary — Highway App Stats Store Saturation (onset 2026-05-03 17:56 UTC):

Repeated SVR peer-path flapping storms on May 3 and May 4 caused the highway AppStatsStore to reach capacity during high-frequency session teardown and re-establishment cycles. Each saturation event (three total in the 24-hour window) prevented newly created sessions from receiving per-application identification stats (StatsAppIdApplicationsAcknowledgementRoundTripForwardMax and related). These sessions' per-application RTT, throughput-by-application-name, and reachability data are absent from the app-summary stream. This accounts for the progressive loss of application-level graph data specifically within the 24-hour window.

All four conditions appeared or intensified during the 24-hour window captured in this bundle, while the earlier part of the week that populates the weekly graph was largely unaffected.


Why Weekly Graph Works but 24h Does Not

Aspect

24-Hour View

Weekly View

Data source

Real-time CIA stats stream (service_summary, app_summary, once-periodic metrics)

Pre-computed daily aggregates stored server-side in Mist cloud

Policy-hit-count

Absent since ~Apr 23 (monitoring-once failure)

Historical daily rollups from before Apr 23 are intact

App-level metrics

Degraded from May 3 17:56 onward (AppStatsStore saturation)

Aggregates from the week's earlier days (Mon–Fri) were computed before saturation

Service summary

Flowing every 60 seconds — not the issue for this metric class

N/A


The weekly graph shows correctly because it aggregates data from days earlier in the week when the telemetry stream was intact. The 24-hour graph relies on fresh, high-frequency data within the current rolling window, which is where both failures manifest.


The metrics related services settled on their own and the 24 Hour graphs were automatically populated.

Modification History

2026-05-13 : Article Created