When deploying the Routing Observability use case, several observability services may remain in the Pending state or experience repeated pod recreations due to insufficient Kubernetes cluster resources.

Analysis of Kubernetes events indicates that sustained memory pressure, and in some cases CPU shortages, prevent the Kubernetes scheduler from placing critical observability workloads on the available nodes.

Customers may observe one or more of the following symptoms:

• Routing Observability services remain in Pending state.
• Pods repeatedly fail scheduling with FailedScheduling events.
• Frequent pod deletion and recreation.
• Delayed startup of observability services after installation or reboot.
• Missing or incomplete observability dashboards due to unavailable backend services. 

Event Analysis

Reviewing the Kubernetes Events (kubectl get events) shows multiple scheduling failures across several observability components.

VictoriaMetrics (vmstorage / vmselect)

• Repeated FailedScheduling events due to insufficient memory.
• vmselect-export pods experienced the highest impact.
• More than 50 scheduling failures were observed before successful scheduling.
• Continuous retry cycles resulted in repeated pod recreation.

MEMS

• Multiple FailedScheduling events caused by insufficient memory.
• Pods eventually recovered after resources became available.
• Indicates ongoing memory contention within the cluster.

Agatha

• Scheduling delayed because of both:
  • Insufficient Memory
  • Insufficient CPU
• Pod remained pending for approximately 16 hours before successful scheduling.

Routing Bot

• Experienced scheduling failures caused by cluster memory exhaustion.
• Successfully scheduled only after resources became available.

Observation

The event analysis indicates sustained resource exhaustion across the four-node Kubernetes cluster.

The primary constraint is memory pressure, with occasional CPU shortages contributing to delayed scheduling.

The most affected workloads were:

These repeated scheduling failures are expected behavior when the Kubernetes scheduler cannot satisfy the resource requests of the workloads.

Root Cause

The Kubernetes cluster does not have sufficient allocatable CPU and memory resources to schedule all Routing Observability workloads.

As available resources become exhausted, Kubernetes continuously retries scheduling pending pods until enough resources are released.

This results in:

• Repeated FailedScheduling events
• Pod churn
• Extended startup times
• Delayed availability of observability services

Recommended Resolution

Refer System Requirement for JRD.

Ensure that the Kubernetes cluster meets the minimum hardware requirements for the Routing Observability deployment.

Minimum Cluster Sizing

For a four-node cluster, each node should provide at least:

Failure to meet these minimum requirements can result in scheduling failures and degraded observability functionality.

Verification

After increasing cluster resources, verify that:

• All observability pods transition to the Running state.
• No new FailedScheduling events are generated.
• kubectl get events no longer reports memory or CPU scheduling failures.
• VictoriaMetrics, MEMS, Agatha, and Routing Bot services are healthy.
• Routing Observability dashboards populate successfully.

Conclusion

Repeated FailedScheduling events for VictoriaMetrics, MEMS, Agatha, and Routing Bot are indicative of insufficient cluster resources rather than application failures. Ensuring that the Kubernetes cluster meets the recommended 48 vCPU / 96 GB RAM / 2000 GB SSD per node sizing for a four-node deployment resolves the scheduling contention and allows Routing Observability services to operate normally.

For any issues contact JTAC support