Juniper Apstra software product version 6.1.0 is available to licensed, registered Juniper customers from the Juniper Apstra software download site.Documentation for Juniper Apstra 6.1.0 is available from the Juniper Apstra documentation site.
NA
You can now use the Device Profiles Juniper_ptx10004_4x36dd_mdp and Juniper_ptx10008_8x36dd_mdp as a Tech Preview. These profiles have been updated to correctly support the 4x and 8x Juniper_ptx10k-lc1301-36dd-lc line cards.
Apstra now supports Juniper QFX5241-64OD and QFX5241-64QD models, which are also available as qualified platforms in the AI template Designer.
New Device Profile for Juniper QFX5241-32OD has been added. This model is also available in the AI template calculator as Juniper supported.
New Device Profiles are now available for the Arista 7280CR3A-48D6, 7280DR3A-54, and the 7804R3 chassis with the 7800R3-36D line card.
You can now deploy the Juniper PTX10002-36QDD switch in Apstra
Device Profiles now feature validations which describe and ensure compliance with inter-port configuration limitations associated with various hardware platforms. For example, a platform might require running pairs of ports at a common speed. The validations will warn you if the staged configuration requires setting members of a pair to different speeds. You can view the validations when reviewing the Device Profile details, and can configure the validations when creating or editing a device profile.
New Device Profile for Accton AS9817-64D, a Tomahawk5 based switch with 64x800G interfaces.
You now can use SONiC version 4.4.2.
Qualification of Cisco NX-OS 10.4(4)
With this new feature you will be logged out of your Apstra instance within 30 mins (default time) of the instance being idle.You can also change the default timeout value (5 min minimum).You will see a timer of 60 seconds before the instance is logged out.
This new feature allows you to leverage the AI Assistant within your Apstra instance to learn more about the status of your network and understand and analyze what anomalies are currently in your network. You can bring your own LLM and integrate it to your Apstra instance. You can access the chatbot within the Apstra UI.
This new UI enhancement for Anomalies provides a comprehensive and intuitive platform to identify, analyze, and resolve anomalies within your system. With the Current Anomalies view, you can quickly detect potential issues, leverage filtering and sorting capabilities, and gain insights through meaningful visualizations. On the Historical Anomalies page, you can view all historical data related to an anomaly in an interactive time-series view.The anomaly details page for each anomaly offers all relevant information, including type, affected elements and devices.
With this new feature, you can now perform in-place updates for security patches
You can now configure static MAC addresses for Integrated Routing and Bridging (IRB) interfaces in Apstra. This feature works similarly to configuring other network parameters such as Maximum Transmission Unit (MTU) settings. The solution includes a global default virtual MAC address setting that applies automatically to all IRB interfaces across your fabric. When you need specific MAC addresses for particular virtual networks, you can override the global setting on a per-virtual network basis. Your configured MAC addresses remain preserved during system updates and maintenance operations, ensuring network stability.
You can now leverage the AOS SDKs on PyPI through a more accessible and user-friendly experience. This provides version information and facilitates dependency management, including dependencies for the Apstra Ansible module.
Users can now upgrade Apstra ZTP either in-place or via VM-to-VM migration, allowing users to maintain ZTP configurations across new versions.
You can now reboot managed devices from the UI/API in the Devices - Managed Devices menu.
A new menu item has been added at the bottom of the left navigation bar to launch DC Assurance (dc.ai.juniper.net) in a new browser tab.
Apstra now supports managing devices operating in FIPS mode, ensuring enhanced security compliance for regulated environments. This update enables seamless integration of FIPS-compliant devices into your Apstra-managed network fabric.
You now can control in a granular the IP addressing for your underlay and choose between IPv4 you IPv6, allowing you to deploy EVPN fabrics over IPv6 underlays.
You can specify these settings on Day1 and update them on Day2 by editing the Routing Zone. Keep in mind that some Day2 changes may be disruptive and lead to traffic interruptions on the switches.
Audit logs in the Event Log under Platform now show events for device config deviation accepted (when the user accepts changes) and device config change (when the user applies the full config) with a diff on the configuration changes for each case.
You now can organize systems in "Upgrade Groups" enabling you to perform batch NOS upgrades across multiple systems simultaneously. By eliminating the tedious process of upgrading individual devices one by one, and executing bulk upgrades instead, you can significantly streamline your NOS upgrade process even for infrastructure at scale.Additionally, "Upgrade Groups" include an impact assessment feature that provides valuable insights into potential risks before the upgrade is executed.
You can now see a prominent UI indication after a Commit-Check operation that shows, for each device in the table, whether it will receive an incremental configuration as part of the next blueprint commit.
When enabling drain mode, users will now see a warning if all devices in a given fabric tier have been selected.This includes cases such as :All spines in a 3-stage fabricAll super spines or all spines in a pod (5-stage fabric)Both leaf switches in a leaf-pairThe warning allows users to review and deselect individual switches before committing, helping to prevent outages at that tier.
You can now directly upload the Network Operating System (NOS) image to the target device using the Copy Image action. This streamlines your NOS upgrade workflow by uploading the NOS images ahead of time, prior to the scheduled maintenance window. This capability is available for both individual devices and for groups of devices through the Upgrade Groups construct.
A new tab in the ZTP UI, called "Services," has been added to view the status of various ZTP components.
You can now easily add generic systems to a blueprint that have been discovered based upon automatically discovered LLDP data
With this new feature, Apstra actively monitors critical switch resources required for a reliable EVPN data center network and alerts you when thresholds are approaching. This capability is supported for both Juniper devices and third-party vendors.
This new probe allows you to gauge Control Plane Policing statistics.
This probe validates the VXLAN flood list entries on every leaf in the network. It collects appropriate telemetry data, compares it to the set of flood list forwarding entries expected to be present and alerts if expected entries are missing on any device.
This probe validates Control Plane Policing (COPP) output on all managed switches in the fabric. This is supported for Cisco NXOS, SONiC, Arista EOS, and Juniper devices. The anomaly threshold is based on any non-zero values seen and it collects the system response of DDOS outputs.
You can now install Apstra's compute agent on AMD-based GPU servers equipped with either Broadcom Thor2 NIC or Pensando's Pollara NIC providing access to comprehensive telemetry and analytics capabilities.The AMD deployments offer identical monitoring scope to their NVIDIA counterparts in version 6.0.0, including the following telemetry services: "hostname", "LLDP", "resource utilization", "interface", and "Interface counters", "GPU hardware Counters" used in ROCvE analytics.
You can now use definitions of custom telemetry collectors shipped as "Built-in" collectors which are predefined and immutable, shipped by default with the product. These collectors are ready to use and cannot be modified or deleted. However, you may clone a Built-in collector and customize the copy to meet your specific requirements.
You now can obtain an analytics report that provides comprehensive insights on MAC addresses and EVPN Route-Type 2. This report includes:
Active MAC address distribution per RZ and VN at start and end time.
Virtual Network growth over time for top 25 Virtual Networks.
Virtual networks size comparison by Routing Zone at start and end time.
Active MAC address distribution across devices (Top 3 VN's).
Missing MAC count observation.
Active and Missing MAC Address Counts.
You can now see the details of the GPUs vendor details for the GPU hosts in Managed device details.
This release supports upgrade paths from previous Apstra 5.1.X and 6.0.X releases.
Users must use VM-to-VM upgrades from Apstra 5.1.X and 6.0.X releases.Refer to the Apstra Installation and Upgrade Guide for more information on upgrade procedures.
A new page under Platform → Product Usage will show various statistics on features consumed/enabled and the associated license level based on usage.
You now can see the failed tasks with clear notifications that remain visible until you acknowledge them. This improvement ensures you stay informed about all task failures, regardless of when they happened or who initiated the task.
The API endpoints for anomalies now include the 'anomalous_node_id' field to provide a mapping between IBA anomalies and graph node IDs. This is applicable for the following API Endpoints: GET /api/anomalies to list all anomalies and GET /api/blueprints/{blueprint_id}/anomalies to list all anomalies for a given blueprint.All anomaly raising processors have been augmented with the new processor property named "Anomaly Anomalous Node ID" to be used in conjunction with Static Context keys to select a specific graph node.
You can now use IPv6 only to manage devices from the Apstra server. You can configure the network devices to have only IPv6 addresses, as well as the Apstra server, and Apstra will manage the devices directly using IPv6.
You now can use the Show Tech feature to collect comprehensive information from Compute agents running on NVIDIA or AMD GPU servers, including:
- GPU Information nvidia-smi -L - Lists all GPUs in the system along with their names, UUIDs, and GPU indices nvidia-smi topo -m - Lists NIC to GPU topology mappings, showing the connectivity between network interfaces and GPUs amd-smi static -B --json - Gather AMD GPU/accelerator diagnostics - Mellanox Hardware Information sudo mst status -v - Uses Mellanox Software Tools (MST) to display detailed information about Mellanox devices detected in the system sudo mlxconfig -d <device> query - Queries configuration parameters for detected Mellanox devices (dynamically discovers devices from MST output) - System Hardware Information sudo dmidecode -t system - Displays basic system hardware information including manufacturer, product name, version, and serial number as part of the system facts gathering process rdma link show --json - For RDMA interface mapping ethtool -i {interface} - Collect driver and version info for all interfaces
All the new diagnostic files are generated under the system_info folder
You can now deploy and manage the Apstra VM on RedHat OpenShift Virtualization
Apstra now compares your product usage to your applied licenses and provides warnings when a mismatch is detected, helping you stay compliant and optimize your investment. You can navigate to the Product Usage dashboard under Platform for detailed usage and entitlement reports, as well as guidance on resolving any discrepancies.
You can now configure the Apstra server with an IPv6 address and access the Apstra UI and Apstra Console via SSH through an IPv6 address.
You can now deploy and manage the Apstra VM on Nutanix 7.3 as the hypervisor
You can now create and execute lists of show commands across multiple devices using a new Apstra CLI command. This feature streamlines device troubleshooting and data collection by allowing you to define command sets once and run them against specific devices or device roles such as leaf, spine, or access switches within your data centre. The command outputs results both to your screen and saves them automatically to a compressed tar.gz file for later analysis.
This enhancement reduces the time needed for routine device queries and provides a consistent method for collecting diagnostic information across your network infrastructure. The compressed output format makes it simple to share diagnostic data with support teams or archive operational snapshots, whilst the role-based targeting eliminates the need to specify individual device identifiers for common troubleshooting scenarios.
The following updates have been made for switch operating systems qualified for the Apstra 6.1.0 release.
Juniper Networks:Junos Evolved for AIDC:23.4x100-D2023.4x100-D40
Junos (All roles):22.2R322.4R323.4R2-S5
Junos Evolved for IP-Forwarder role (Spines in EVPN or any role in an IP-Fabric):22.2R3-EVO22.4R3-EVO23.4R2-S5-EVO
Junos Evolved for EVPN leaf roles:22.2R3-EVO22.4R3-EVO23.4R2-S5-EVO
Junos Interconnect Gateway Leaf:22.4R3 (minimum)23.4R2-S5
Junos Evolved Interconnect Gateway Leaf:22.4R3-EVO (minimum)23.4R2-S5-EVO
Cisco Systems:10.2(6)10.3(4a)10.4(4)
Arista Networks:4.28.7.1M4.30.3M4.33.4M
Dell EMC & Edgecore:Enterprise SONiC 4.2.3Enterprise SONiC Edge Standard 4.2.3Enterprise SONiC 4.4.2Enterprise SONiC Edge Standard 4.4.2
The clone function in Rack Type Designer was improved. Now a single node gets copied below its parent, while multiple nodes get copied one step to the right from the selected group. All copied nodes are automatically selected after being created so they can be moved to the correct location.
You can now use Time-Voyager to store more revisions. Instead of limiting by revision count, it limits by time and disk space. The system uses diff-encoding like Git, which means revisions share disk space by removing duplicate parts. This allows storing hundreds or thousands of revision changes while using less disk space.
Within a Blueprint, Configlet Scopes written as `name in ["spine1"]` are now displayed as `id in ["id-of-spine1"]`. This change is cosmetic only: Under the covers, name-based scopes were always expressed by ID.
You can now write truly name-based Configlet Scopes using this notation: `label in ["spine1", "spine2"]`, or by using the constructor in the web UI.
You now can control with greater flexibility the application of configlets to your Cisco, Arista and SONiC blueprints through new capabilities including Differential Rendering which provides access to both previous and incoming device model for more granular/conditional rendering.The new feature also provides lifecycle awareness, enabling better identification of when a configlet appears for the first time and allowing more precise control over when the negation part is applied.
The EVO platforms now support sFlow on the management interface via the mgmt_junos VRF, with a minimum NOS version of 23.4X100-D31.6-EVO and 23.4R2-S3.11-EVO. You can now use the latest built-in configlet to leverage sending sFlow via the management interface on the mgmt_junos VRF.
You now can use a new modular architecture with Custom Telemetry which has evolved from a fixed functionality to being fully customizable. This new architecture lets you define advanced data processing in the form a data processing piepeline which is a collection of nodes, each performing a single function on a set of telemetry data, linked together to form a data flow. This follows the Function composition paradigm.
New capabilities have been introduced such as the "Join" and "Group" functionalities for improved data manipulation. This allows for example to define telemetry collectors leveraging more than one command.
You now can create and test pipeline queries without upfront choosing or creating the target service. i.e no need to define the Schema upfront like in previous releases. This allows you to focus on creating the desired query first and then create the service that will match the query output.
You now can use the Interface Queue Stats probe to collect and analyze critical interface metrics on the Spine-to-Leaf portion, representing an expansion compared to the previous monitoring scope, which was limited to the Generic-facing interfaces only. This includes both Uplink interfaces on the on Leaf devices and downlink interfaces on the Spines devices.
You now can benefit from more granular reporting in the predefined Queue Stats Monitoring dashboard, now structured to display for every metric, first the values on the Leaf-To-Spine interfaces portion, following by the ones in the Leaf-To-Generic interfaces portion.
The probe remains supported on QFX platforms only, which share the same internal architecture for traffic management, enabling consistency in the data collection and reporting. As such, PTX platforms, even if supported as qualified models in AI Cluster Reference-Design are excluded from the scope of this probe.
You now can set differentiated thresholds for anomaly raising in the Device System Health probe on a per System type basis Switch, Server.You now can Enable/Disable the anomaly raising for the Device System Health probe on a per System type basis Switch, Server.
You now can use the predefined and auto-enabled Device Traffic probe that has been enhanced to support multiple use-cases that were previously handled by six separate IBA probes which had to be manually instantiated. The list of these six probes for which the logic has been added to the Device Traffic probe are the following (Note that these IBA probes remain in the product as of 6.1.0 release. Their deprecation (removal) will be planned in the following release: Minecraft):
Hot/Cold Interface Counters (Fabric Interfaces)
Hot/Cold Interface Counters (Spine to Superspine Interfaces)
Hot/Cold Interface Counters (Specific Interfaces)
ECMP Imbalance (Fabric Interfaces)
ECMP Imbalance (Spine to Superspine Interfaces)
Packet Discard percentage
The incorporation of the analytics from those 7 probes into the new "Device Traffic" probe is designed with a focus on the core essentials, providing you with valuable insights. The result is a simplification by eliminating excessive options so you don't suffer from "choice paralysis" syndrome.
You now have the ability to stream any type of IBA data including dynamic stages. This means that all forms of IBA data can be accessed and transmitted in real-time through streaming capabilities.
Users can now install the compute Device agent for GPU servers using Out-Of-Band installation methods eliminating the requirement to maintain GPU server credentials within Apstra.
Junos and Junos EVO switches now use load-update instead of load-override for configuration management. Load-update modifies only the specific settings that need to change, rather than replacing your entire configuration. The system compares your current configuration with new data and updates only the differences between them. This approach provides better support for advanced features like MACsec and AAA, which require precise configuration updates rather than complete replacements.
The Product Usage reports now track the number of GPUs deployed and managed in an AIDC cluster.
You now have a DEPRECATED status for API endpoints that comes before REMOVED. Deprecated endpoints include a Deprecation HTTP Header Field and appear strikethrough in the API Explorer before being physically removed from the code. The changlog displays the changes in a stacked way ordering different release changes by importance, with the most recent information on top: REMOVED, DEPRECATED, CHANGED, NEW.
You now can access the API changelog in the documentation so you can review it for any changes before you install the new version.
Displaying for every release the total count of change by type acting as hyperlinks to update the page with that filter. With this users can select a specific release and request to view only NEW changes or any other single change type.
Upon installing a new system agent, Apstra now verifies that the device's Netconf server is operational. This same validation process is also executed when initiating a Check Job request.
When attempting to delete a Virtual Network that are used in Connectivity Templates configured with Multiple VLAN primitives, the system now implements a two-stage verification workflow. Because removing this network could cascade and affect additional Virtual Networks, potentially triggering their removal as well, you must first review the deletion's consequences through a "View the impact" option before proceeding with the final confirmation. This ensures you are fully informed of the impact and can update or adapt the Connectivity Template design as needed.
Tech Previews give you the ability to test functionality and provide feedback during the development process of innovations that are not final production features. The goal of a Tech Preview is for the feature to gain wider exposure and potential full support in a future release. Customers are encouraged to provide feedback and functionality suggestions for a Technology Preview feature before it becomes fully supported.
Tech Previews may not be functionally complete, may have functional alterations in future releases, or may get dropped under changing markets or unexpected conditions, at Juniper’s sole discretion. Juniper recommends that you use Tech Preview features in non-production environments only.
Juniper considers feedback to add and improve future iterations of the general availability of the innovations. Your feedback does not assert any intellectual property claim, and Juniper may implement your feedback without violating your or any other party's rights.
These features are "as is" and voluntary use. Support Services will attempt to resolve any issues that customers experience when using these features and create bug reports on behalf of support cases. However, Juniper may not provide comprehensive support services to Tech Preview features. Certain features may have reduced or modified security, accessibility, availability, and reliability standards relative to General Availability software. Tech Preview is not supported under existing service agreements, SLAs, or support service.
For additional details, please contact Juniper Support or your local account team.
You can now create an AI based rail-aligned blueprint with EVPN overlay control plane. This is a Tech-Preview feature to start prototyping the GPUaaS use-cases.
You now can use the offbox unit mode when creating offbox agents, enabling you to associate multiple managed devices with a single Docker container instead of the traditional 1:1 Device to Offbox Agent mapping. In this new mode, delivered as Tech-Preview in 6.1.0, one offbox unit docker container will host the agents for multiple devices (by default 16). This offers significant resources optimization (such as the memory footprint), hence significantly enhancing the Offbox Worker Node Scaling.Â
Attempting to pull the configlet preview for specific blueprint device ("On Device Configlet Preview"), by clicking on the device label under the general configlet preview page might fail with a slightly misleading error, if the device is unassigned. Certainly trying to get a preview for a device which is not assigned is bound to cause an error, as a real preview for a device that doesn't exist isn't possible. However, the error emitted is slightly confusing.
The error message will be improved to convey the actual problem accurately in Apstra 5.1.0.
In the AI Cluster Template workflow, selecting "Manually Selected" under Device Models incorrectly shows "4 selected" even when no models are chosen. The count should reflect actual selections or indicate default behavior clearly.
The fix to resolve the counting error in the device profile tables selection setting is targeted for version 6.1.0.
AOS server details stored in the local aos.conf file currently take precedence over any AOS server information entered through the ZTP UI.
When using the ZTP UI to post or update AOS server details, the updates will be stored in the database - this is going to be the primary source of truth for AOS server information. If the database is unavailable or the details are not yet posted, the AOS configuration file located at /containers_data/status/app/aos.conf is used as a fallback to retrieve the AOS server details. These AOS server details are essential for ZTP to forward devices ZTP status and ZTP services status to AOS, and to trigger system-agent jobs if directed by the user via the ztp configuration file(ztp.json).
When instantiating predefined probes such as "VMs Without Fabric Configured VLANs" Apstra UI may fail to display the probe with a type error, making the probe unusable.
The issue is resolved in Apstra 6.1.0, where predefined probes may not be usable from a typeerror in the UI
In Apstra 6.0, the Main Topology View (Blueprint > Staged > Physical > Topology) displays ESI/MLAG pairs with leaf2 positioned on the left and leaf1 on the right, which is the reverse of the layout seen in prior versions. In previous versions, the UI consistently rendered ESI/MLAG pairs with leaf1 on the left and leaf2 on the right, aligning with user expectations. The change in Apstra 6.0 is purely cosmetic and does not affect system behavior or configuration.
The issue is resolved in Apstra 6.1.0, where the ESI/MLAG leaf switches are displayed in the correct left-to-right order, consistent with previous versions.
The timestamp field of AosMessage in the streaming has used uint64 format for both millisecond and microsecond timestamp information. The current timestamp uin64 field will be changed in release 6.1.0 to the google.protobuf.Timestamp format, which is incompatible with the previous version, in order to provide consistent, accurate timestamp information. The streaming receiver side must be modified to accommodate this incompatible modification.
The timestamp field of AosMessage is changed from uint64 to google.protobuf.Timestamp format
An event involving BGP flaps from BGP peers configured on IRB/Loopback interfaces for Juniper EVO device has been reported during the commit with incremental changes (deleting an IRB/Loopback interface in the same VRF). The BGP flaps happen when the device's configuration is committed in override mode, which was Apstra's default setting prior to 6.1.0. However, using load update mode did not reveal the issue.
6.1.0 uses default mode as a load update for commit operations for both on-box and off-box agents.
Starting with Apstra-6.1.0 release, blueprints with disabled IPv6 application support and IPv6 addressing on fabric links are no longer supported.
Starting with Apstra-6.1.0 release, subinterfaces and static routes with IPv6 addressing over IPv4 underlay are no longer supported for default Routing Zone. Additionally, IPv6 link-local subinterfaces in a blueprint without IPv6 application support are no longer allowed for any Routing Zone. These configurations are not valid logically considering the underlay capabilities for each case.
In the case of IPv6 subinterfaces and static routes with IPv4 underlay (spine_leaf_links == ipv4), the only solution is to delete the problematic entities. For the IPv6 link-local subinterfaces there are two options, either enable IPv6 support for the blueprint or delete the problematic subinterfaces. Usually subinterfaces and static routes are created by Connectivity Templates, so to delete them the corresponding Connectivity Template should be unassigned.
User feedback was added in Apstra 6.0.0 to enable users to share their Apstra experiences. After the release of Apstra 6.0.0, the link to the feedback form was changed. It results in an error message about a missing form.
The link to the feedback form is updated with permanent link information, which allows feedback form to be referenced correctly.
If the rendered device configuration for the VLAN description contains newline characters populated from the virtual network's description field, the commit operation fails because JUNOS and JUNOS-EVO devices do not support multi-line string for the VLAN description.
Starting with Apstra-6.1.0 release, connectivity templates (CTs) with the same primitive policy having multiple batch or pipeline parent policies are no longer supported. This configuration causes ambiguity in CT application and can lead to unexpected behavior.
Delete the problematic connectivity templates from the UI and recreate them. The UI will automatically create the correct structure with proper parent-child relationships.If the /obj-policy-import API is used directly, ensure that primitive policy IDs are not reused across different "batch" or "pipeline" subpolicies. Each policy (primitive policy, pipeline, batch) should have a unique parent policy.
While viewing/editing CT (Connectivity Template)s across blueprints, it's possible that the CT may incorrectly display empty field values.
In version 4.2.0, Apstra introduces the capability for users to forcibly delete a Virtual Network, even if it has active endpoints. Apstra will initially display the interfaces to which the Virtual Network (VN) is currently allocated and prompt the user to confirm the deletion. It's important to note a limitation in the current design: if a user deletes a VN assigned in a CT where Multiple VLANs are present, all active endpoints will be unassigned.
RFE-3417 (addressed in 6.1.0) provides more contextual information to prevent unexpected consequences before action is taken.
RFE-3417
DeviceTelemetryAgent.{pid}.log files in /var/log/aos/ in the offbox agents become large and can fill up the disk
ECN marked packet information is not currently visible in the dashboard widget, although it is correctly displayed in the staged view in the probe.
Execute CLI commands in the Juniper device supported only show and request chassis beacon commands in the Apstra < 6.1.0 environment. Additional commands (ping and traceroute) are introduced in Execute CLI commands in the Apstra >= 6.1.0 environment for easier troubleshooting environments.
Because the JUNOS device might not send the full snapshot of interface-related data per reporting interval (default interval = 120 seconds) after initial synchronization, Device Telemetry Health detects continuous anomalies for gRPC Periodic Response Timeout in the interface telemetry service in a high-scale environment. The problem still exists even if the reporting interval is extended.
In 6.1.0, the interface telemetry, which uses gRPC periodic collection, will be enhanced to accommodate the gRPC updates from the device without triggering hold timeout, as long as the device sends the data per reporting interval. gRPC can be enabled back safely by setting grpc_enabled = 1 in the apstra configuration file (/user/root/etc/aos/aos.conf) and restarting AOS services (sudo service aos restart).
[telemetry_global_config] # Python multithreading enable/disable knob for telemetry collection multithreading_config = 1 # Execution timeout for extensible telemetry collectors command_timeout = 120 # Knob to enable/disable gRPC based service collectors grpc_enabled = 1
Because the MAC telemetry service uses the gRPCOnChange mode, the device only sends updates after initial synchronization. When the JUNOS device subscribes to the PATH (/network-instances/network-instance/mac-table/entries/entry) for MAC telemetry service, Apstra's gRPC client (Apstra) receives the first full data from two processes (l2ald, l2aldTM). During the initial synchronization, these processes use their own sequence number range (duplicate range), which makes the gRPC client think that the gRPC packets may be dropped internally. Granular sequence number handling will be introduced in 6.1.0 to address the existing sequence overrun issue.
The more granular sequence handling is implemented in gRPC sequence handling logic. gRPC can be enabled back safely by setting grpc_enabled = 1 in the apstra configuration file (/user/root/etc/aos/aos.conf) and restarting AOS services (sudo service aos restart).
The default collection period for IBA interface flapping is only 60 seconds and the flapping threshold is 5 times. But, the default collection period for interface service is 2 minutes, the interface will receive updates every 2 minutes only.
So, the default anomaly window is too small to capture 5 interface flaps. For 5 flaps it should be at least 10 minutes.
The threshold was changed from 5 to 2 and increasing duration from 1 minute to 6 minutes. With this change the probe is capable of tracking at least 3 interface statuses meaning 2 flaps/transitions, i.e. state -> state -> state where the value of the state doesn't matter. If so, an anomaly is raised.
Using the system parameters (go to the staging tab of the freeform blueprint, choose the target switch, enter the edit mode for the S/N field, then click the Reset value button) will not allow the Apstra UI to unassign a serial number for a system. The failure of "deploy_mode" will result in the message "Value is required." because the Apstra backend expects the deploy_mode value to be one of the values ("deploy", "ready", "drain", or "undeploy"), but the Apstra UI sends the deploy_mode value as null when the serial number is unassigned.
Apstra UI validation changed to allow un-assign the S/N when the system is part of a freeform blueprint
Customers attempting an in-place upgrade to AOS 6.1.0 may experience a failure during the upgrade pre-check stage if the system is running an unsupported or incompatible version of the docker-ce package.
When performing a ZTP upgrade to version 6.1.0, ensure that the required docker-ce package (version 5:27.5.1-1~ubuntu.22.04~jammy) is installed to maintain compatibility.
Installation failure of the out-of-band system agent for GPU servers () is observed under the below activities1. Install the "telemetry only" system agent on device1 using the Managed Devices in the UI. Select "Install" option for "Job run after creation" field2. Uninstall the agent via the Managed Devices in the UI. Also delete the device once the agent uninstallation is complete.3. Install the OOB agent on the device using device agent images (aos_device_agent-XXX.run)4. Using the UI create an onbox agent and select "telemetry only" and "None" option for "Job after creation".
A fix was merged to easier allow oob device agent install (following the below workaround) until this issue can be fully resolvedDelete system Agent and then install the OOB agent by running the command : bash aos_device_agent.run – --metadb tbt://{controller_ip}:29731 --interface eth0 --install-type outOfBand
interface 25g speed configuration was not properly rendered over ports 4-27 when 25g transformation is used in the Juniper ACX7024 device profile
interface speed 25g now rendered as part of 25g transformation for Juniper ACX7024 devices
If a user adds an invalid character like a "space" to elements (e.g. link labels) in an Apstra Blueprint, it will cause errors when the user attempts to add a pod to a Blueprint.
Junos RFC5549 BGP peer sessions were rendering the same route-map twice on import/export statements. This only applies to 'ipv6-only' bgp peers.
Configurations were rendering:
neighbor a05:fab:192:168:50::254 { description "facing_leaf1-generic"; local-address a05:fab:192:168:50::1; peer-as 65510; family inet { unicast { extended-nexthop; } } family inet6 { unicast; } import ( RoutesFromExt-default-Default_immutable && RoutesFromExt-default-Default_immutable ); export ( RoutesToExt-default-Default_immutable && RoutesToExt-default-Default_immutable ); }
This has been addressed in 6.1.0, where the import and export statements will contain that route-map entry only once.
This may result in a service disruption on upgrade for those BGP peers as Junos generically may reset the peer when it detects any import/export policy reference change.
This will also apply to ipv6-only, non-EVPN blueprints for route-maps such as "LEAF_TO_SPINE_FABRIC_OUT" between all superspine/spine/leaves.
The configuration will automatically be corrected upon upgrade to Apstra 6.1.0.
The Router MAC (also known as Master Bridge MAC or bridge MAC) on a SONiC device is a unique MAC address assigned to the switch and used as the source MAC address for packets originating from the switch itself, such as those generated by the virtual router or bridge interfaces. The current Mac Monitor probe does not report Router MAC on the originating switch (for example, if leaf1 has MasterBridgeMac as 52:54:00:4e:c6:60, this MAC will show up as a missing MAC on leaf1 for all VNs).This is a bug related to analytics only; it does not have any network operational impact.
The NOS upgrade procedure must parse the pristine configuration in order to determine which ports must be disabled when the Skip Shutting Down Interface During Upgrade option is not checked in the Advanced Settings of Managed Device. The NOS upgrade would fail if the configuration line in the pristine configuration extended into multiple lines because parsing the command line misses the end-of-command-line character (.
For deployment, the system agent attempts to map the agent's ID (UUID) to the system serial number using two step resolutions: system agent ID to management IP address mapping configured when the agent is created, and management IP address to system serial number mapping during device fact collection. When a system agent is configured as a FQDN rather than an IP address for the management IP address, the resolution from the agent's ID to the system serial is incorrect, causing deployment to stall until it is resolved.
When the Optical Transceivers Probe is turned on, the optical_xcvr telemetry service is enabled for all systems with on-box agents (including GPU servers) or off-box agents. Because the optical_xcvr telemetry service is not designed to run on GPU servers, it fails to collect optical transceiver information and returns an error message.
The graph query of the Optical Xcvr Stats process in the Optical Transceivers Probe is refined to cover only switch nodes.
Every leaf (device that is a part of the VN network) device has its EVPN Type2 expectations validated by Mac Monitor Probe. By using EVPN type 2 advertisement ingestion, it indicates as missing the MAC addresses that should be on the devices but aren't.
Mac Monitor Probe
It is observed in a SONiC environment that a device does not ingest the type 2 advertisements to the corresponding VLAN forwarding (MAC) table if it is not connected to a host for the given VN. In the event that the device is a part of the MLAG pair, the type 2 advertisements are ingested into the Mac Table even if it is not host-attached.
As of right now, the MacMonitor probe ignores any device from the previously mentioned type 2 validation that has no host attached. As a result, SONiC devices that are part of the MLAG but lack VN endpoints (hosts attached) are excluded from the type 2 missing MAC check. As a result, on such devices, the probe might not find any missing MACs.
However, in MLAG scenarios, no traffic is expected for the specified VLAN on the rack if a leaf device has no connected hosts, i.e., VN endpoints, so this issue has no functional impact.
An exception will be added for SONiC MLAG racks when no VN endpoints are attached in order to fix the problem. When the exception is enabled, this modification will guarantee precise verification of missing MACs against the total number of MACs discovered for the EVPN-VN. 5.1.0 is the intended release date.
When a port group in the vCenter is configured with a private vlan mode, the VLAN specification contains the pvlanid property instead of the vlanid property. However, the vlanid property is always expected from the port group's vlan specification by the Device Telemetry Agent's collector if it is not trunk mode. Anomalies could be reported if the collector's execution fails due to a reference to the vlanid property, which is nonexistent.
Sutatined Optical Threshold anomalies are frequently observed over disconnected (not connected) interfaces in the JUNOS device. The main reason for the problem is that the JUNOS device sometimes reports the received average power as a very low value or - Inf value when the interface is disconnected, and Apstra does not correctly parse the - Inf value. When a very low power value falls below the warning level, Apstra creates an anomaly for the low receive power port. But when the same interface reports a - Inf value that is later incorrectly parsed, Apstra eliminates the interface from the list of interfaces with an optical transceiver, thereby resolving the raised anomaly falsely. This is the reason anomalies are frequently raised and cleared over the same interface. No workaround is available for this issue
Since the UI misses polling of the node detail information to the Apstra backend, the Virtual Network Endpoints view of Generic System Node (Staged > Physical > Topology > Virtual Networks Endpoints) shows empty information.
Apstra creates a relationship between the PNIC and the Link Discovery Policy (which determines which discovery protocol is used) configured in the VDS when a PNIC is assigned to a VDS (Virtual Distributed Switch). One PNIC may inadvertently become linked to two relationships without clearing out the previous relationship when a user moves a PNIC directly from one VDS to another VDS. An error message below appears when the PNIC becomes unassigned from VDS because there is more than one relationship between the PNIC and Link Discovery Policy that is invalid.
virtual_infra failed to collect data, plugin raised exception: {'item_iter': <aos.sdk.graph.graph.RelationshipIterator object at 0x7f60c03f9570>, 'items': [df57a2f0-969c-4dee-9831-a53526bd7d5a-[:policy]->4c8c4c31-df9a-4188-933f-6b5d67703a1f, df57a2f0-969c-4dee-9831-a53526bd7d5a-[:policy]->1787d662-4a41-4b8b-9b77-acac5508e771]}
When multiple vNICs from a single VM are assigned to the same vNET (port group or VDS) in the Virtual Infra, the VMs Without Fabric Configured VLANs probe raises anomalies in the Analytics->Anomalies because the graph query in the VMs backed by Fabric VLANs processor treats those vNICs as identical.
When a user attempts to open the racks or pods tab in the Staged->Physical section, rack schema validation logic will be executed for every rack in the blueprint. In order to avoid incompatibilities with the underlying database, non-ASCII characters cannot be included in the link group label, which might be explicitly supplied when adding a generic system to leaf or access nodes.
Anomalies are raised due to mismatch in the operational status of interfaces due to interface status showing "unknown" on Juniper EX4400-48T devices running Junos 22.4R3.
Issue not observed in higher versions of Junos. Recommend NOS upgrade to Apstra-qualified higher JUNOS version (>= 23.4R2-S4).
When Apstra Edge container is configured with Apstra Flow together in the Apstra Controller VM, Apstra Edge shows 50% CPU utilization, which may affect the performance of the Apstra Controller and other containers. The issue is observed when the Apstra Edge container 0.5.0 image is used. In the latest version, 0.16.5, the issue is not shown anymore.
Upgrade Apstra Cloud Services Edge image from 0.5.0 to the latest image, 0.16.5 in the Apstra 5.1.0 release.
When the Apstra commit was executed, the JUNOS device with the off-box agent reported BFD underlay flaps between the committed device and the other devices. Apstra currently uses the load override option as the default action for device commits, which may result in high CPU utilization, preventing time-sensitive daemons from acquiring CPU time slices and triggering unexpected events such as BFD timeouts or writing EEPROM errors. Starting with 6.1.0, Apstra intends to use load update as the default commit action for JUNOS and EVO devices.
In 6.1.0, Apstra uses load update as default for commit action instead of load override for JUNOS/EVO on-box and off-box agents
Apstra 6.1.0 enables users to change VTEP addressing on Junos/Junos EVO devices between IPv4 and IPv6.This typically results in a PFE restart with new forwarding option configurations on some platforms as well as restarting all of the EVPN sessions in the blueprint as they transition from ipv4 & ipv6.
Users must take caution and plan maintenance windows when moving VTEP addressing to and from IPv6 as a day2 operation.
In Junos Evolved, the customer is able to remove the device FIPS mode by simply changing the configuration and, importantly, without having to zeroize the device. This is not a bug.
Customers should be aware of this and should not modify their pristine configuration in a way that removes the FIPS configuration.
Due to an outstanding bug in all available versions of Junos EVO, when two virtual networks are hosted on the same set of ESI leafs, one with DHCP enabled and one with DHCP disabled, the virtual network with DHCP disabled will also have DHCP requests forwarded.
NX-OS when configured for First Hop redundancy uses the Virtual Mac address assigned for BGP sessions. This causes BGP session tcp sessions to not come up as they may end up on the incorrect switch.
For power supplies, Apstra primarily uses the output of the show chassis environment pem or show chassis environment psm command. On the QFX10002-36Q, both PEMs are reported, however, only one includes the XML tag that designates the component class as Power. The power supply information is further augmented using the show chassis environment command. Since the tag is missing for one PEM, Apstra does not recognize it as a valid power supply component. This is a known issue in Junos. Below is an example of the XML output from the show chassis environment without the class tag:
<environment-item> <name>FPC 0 Power Supply 1</name> <status>Present</status> </environment-item>
Due to inconsistent Junos behavior, the Power Supply State Check processor of Apstra does not evaluate the affected PEM, and no anomaly is raised. No workaround is available for this issue. This issue is expected to be addressed in newer Junos versions from 24.4R2, and the corrected behavior is expected to be present in supported versions for Apstra 6.1.0 and later.
Recommend using Apstra Qualified Junos NOS version >= 24.4R2
When copying NOS images to SONiC devices, the images are placed in a temporary directory. If the device reboots before the image is installed, the copied image will be deleted.
The state_check processor outputs a state column for each series. However, users may not be able to filter series in the output stage using the per series state column when querying stage data. Users attempting to use a filter such as properties.state = 'false, true' may see no results, even if matching data is present. This is a known limitation in how filtering works on series-level data in the state_check processors output. The filtering behavior may not align with user expectations when attempting to use conditions on the state column. Engineering evaluating potential improvements to allow filtering on state column in state processor for a future release.
MetricQueryManagerAgent handles large historical data to serve the '/blueprints//anomalies-history' endpoint. Depending on the amount of data, the agent's memory footprint may increase significantly. The benchmark environment recorded a memory footprint of up to 2.2Gb for the agent. The memory footprint settles after the initial bump.If the system administrator is concerned about the MetricQueryManagerAgent footprint's impact on the system's available memory, the following workaround is recommended.
Restart MetricQueryManagerAgent and avoid using the 'Time Series' query.
Decreasing the value of the "Time Window" property of the "Time in State" processor, either directly or via predefined probe parameters, may not have an immediate effect but rather after the expiration of a previously configured time window or an update of the input stage.
Disable and enable back the affected probe
Showtech collection fails with a message that the command partially completed.
The log indicated in the error will contain a message that looks like "Unknown type ID: MigrationCheckpoint"
This is caused by a error processing metricdb AosUpgrade logs
Please delete the .tel files causing the issue from the /var/lib/aos/metricdb/audit/AosUpgrade directory.
This is typically an old upgrade as an example 4.1.2 > 4.2.2 which may fail to parse
service attach aoscd /var/lib/aos/metricdb/audit/AosUpgradefor file in *.tel ; do taccEventLogExport $file ; done examine the .evl files that were created and fine the likely .tel files both the meta and samples that need to be removedrm meta-.telrm samples-.tel
and test show tech again
Apstra's BuilderAgent may crash if the generic system's interface name (if_name), identified by LLDP from the generic system, contains trailing space after the user creates generic systems based on the gathered LLDP data. This is because the automatically generated interface's description for the generic system contains if_name information with trailing space, which causes a validation error in the BuilderAgent.
The below signature can be seen in the BuildAgent.err log file.File "/usr/lib/python3.10/dist-packages/aos/grappa/table/cpp_graph.py", line 909, in set_noderaise ValidationError(errors)lollipop.errors.ValidationError: Invalid data: {'description': 'Invalid description. Only ASCII characters with codes 32 - 126, except "?", "<", ">" ,\'"\' and are allowed. The description should not start or end with a space.'}
The if_name with trailig space (coming from LLDP) needs to be correct by REST API Post call. Please contact Juniper Apstra Support for further assistance.
If a show-tech job encountered a partial failure, the UI would hide the download link and instruct users to retrieve the archive manually via the CLI. However, in many cases, a usable archive is still generated despite partial errors. The current UI logic needs to be enhanced to ensure the download link is visible and accessible whenever an archive is available, regardless of the final job state, simplifying the process for gathering debug data for support.
When multiple show-tech jobs are triggered simultaneously, the /var/log partition can reach high utilization, causing the controller to enter read-only mode. This may result in incomplete job status updates, leaving several jobs stuck in in-progress or pending states. In a corner case, this condition can also lead to repeated crashes of SystemAgentManager, preventing automatic recovery even after disk space is reclaimed. Avoid triggering bulk show-tech collection on a large number of devices. Ensure sufficient disk space is available before running show-tech.
If the issue occurs, reclaim space in /var/log and restart AOS services. In most cases, SystemAgentManager recovers automatically, clearing stuck jobs and completing or failing pending ones. If SystemAgentManager crash persist and jobs remain stuck, manual cleanup via Acons is required. It is recommended to contact Apstra Support for assistance.
When modifying the maximum disk usage setting in a blueprint, the following behavior may be observed:
1. If the maximum disk usage is set to a value lower than the current disk usage, the blueprint commit fails. 2. If the maximum disk usage is increased to a value greater than the current disk usage and committed, the commit may still fail.
In the above scenario, the blueprint commit version does not advance, and the updated maximum disk usage setting is not honored.
After increasing the maximum disk usage to a valid value, make an additional, unrelated change in the blueprint and commit again. This subsequent commit succeeds and correctly applies the updated maximum disk usage setting.
In rare situations where several dozens of BGP peerings to external generic devices have been setup on an NXOS edge leaf switch, the upgrade configuration may fail to apply within the allotted 120 seconds with a timeout. The upgrade configuration is modifying all the external BGP neighbors to always use the next-hop-self command, to properly support RFC-5549.
After the upgrade completes, perform a full configuration apply on the affected device. This ensures that the correct configuration is successfully applied. Note that this workaround is not necessary unless there is a setup that routes IPv4 packets using IPv6 next hops, per RFC-5549.
While configlet is being applied to the SONiC device, if the device agent restarts after being disconnected from the controller, the agent executes any remaining changes and collects the running configuration as golden configuration to monitor for configuration anomalies. Because the process of applying configlet changes is still running independently of the agent, it introduces changes into the running configuration even when the golden configuration is collected by the agent. The following changes from the process cause configuration anomalies in the SONiC device.
After reviewing the running configuration on the SONiC device, if all the changes from the configlet are correctly applied, the customer can safely accept changes to avoid further configuration anomalies.
Adding multiple AAA servers in the blueprint through Staged > Catalog > AAA Servers leads to a configuration load error in the JUNOS and EVO device during commit check or commit.
Recommend using configlet instead of using UI (Stage > Catalog > AAA Server) when multiple AAA servers needs to configured.
When attempting to convert leaf switches from ESI to MLAG within the same blueprint, the operation fails because Apstra does not allow mixing ESI and MLAG redundancy models at the rack level. This restriction is enforced starting in Apstra 4.2.0 and is expected behavior. During the operation, users may see the following error in the UI or REST API Explorer:
"Combining MLAG and ESI leaf pairs not supported"
Currently, converting ESI racks to MLAG racks or vice versa requires replacing all racks within a single FE operation using the REST API. However, due to limitation, this conversion can cause BuilderAgent to fail when links are present between external generic system and leaf switches. In this occurs, please revert the changes and follow the steps outlined in workaround section.
To successfully convert the rack type using modify-racks API, the following workaround can be used:
1. Identify the leaf switches connected to the external generic system 2. Identify the Connectivity Templates (CTs) associated with the external generic interfaces and unassign the corresponding application points 3. Delete the links between the leaf switches and the external generic system 4. Undeploy and unassign the leaf devices from the blueprint 5. Unassign interface maps from the blueprint 6. Use the REST API /api/blueprints/{blueprint_id}/modify-racks to convert the racks from ESI to MLAG 7. Import MLAG-compatible interface maps and assign them to the switches 8. Recreate the links to the external generic system 9. Reassign the endpoints to the appropriate Connectivity Templates
For additional guidance, please contact Apstra Technical Support.
Task Error will show: "virtual_network": "IPv4 in VRF \"default\" is not supported with Routing Zone disable_ipv4=\ "True\"" with rail-id's in the payload.
Provisioning of VLANS is currently not possible when the default RZ is configured with disable_ipv4=True (IPv6 only). When changes to Apstra allowed the creation of RZ without IPv4 (IPv6 only), the API endpoint, which creates VLANs, incorrectly assumed IPv4 is always present.
The current Radius client in the Apstra Controller adheres to standard RFC 2865 functionality, inherently not FIPS-compliant because it relies on weak, non-compliant algorithms (MD5/MD4) for password hashing and authentication.
The 6.1.0 published VMWARE ESXi OVA image is set up with two hard disks, disk 1 (230G) and disk 2 (80G). But disk 1's size is incorrectly configured to 230G instead of 150G (default Apstra image size in 6.1.0); therefore, the extra 80G disk resource is assigned in disk 1 but not used. The workaround can be used to deploy the OVA image with fewer disk resource requirements.
Please follow the below steps to create a correct VMWARE ESXi OVA image, which has correct size information for disk 1 based on the 6.1.0 released OVA image, and deploy the newly created OVA image for the 6.1.0 controller and worker VMs. If you need further assistance, please contact the Juniper Apstra Support team.
1. Untar aos_server_6.1.0-64.ova image $ tar xvf aos_server_6.1.0-64.ova aos_server_6.1.0-64.ovf aos_server_6.1.0-64-disk1.vmdk aos_server_6.1.0-64-disk2.vmdk 2. Modify capacity information from 230G into 150G in the ovf file $ sed -i 's/capacity="230"/capacity="150"/g' aos_server_6.1.0-64.ovf 3. Create a new OVA file by creating tar archive from all three files (the below example creates a new aos_server_6.1.0-64-size-150g.ova file) $ tar --owner=someone --group=someone -cvf ./aos_server_6.1.0-64-size-150g.ova ./aos_server_6.1.0-64.ovf ./aos_server_6.1.0-64-disk1.vmdk ./aos_server_6.1.0-64-disk2.vmdk
After upgrading Apstra from version 5.1.0 to 6.1.1, frequent updates to the LLDP graph result in repeated updates to the operational graph. This, in turn, triggers the DashboardManager to evaluate predefined dashboard graph query conditions for all blueprints, leading to high CPU utilization of the ScotchAgent. As a result, the customer is unable to perform any actions on the blueprint.
Users must disable the Auto-Enable toggle (set it to Off) by navigating to BP > Analytics > Dashboard > Configure Auto-Enabled Dashboards for all blueprints in their infrastructure.
When a configlet action (import/delete) occurs in a blueprint with a large number of configlets, it can fail with a timeout, and the BlueprintDiffProducerAgent process can crash due to a heartbeat timeout. The problem occurs when the blueprint has a configlet with an incorrect Jinja expression via configlet import/delete actions. Failure of rendering configuration with incorrect Jina expressions can cause all configlets to be re-evaluated for all eligible devices, potentially resulting in much longer configlet processing.
The below steps can be applied as a workaround. If further assistance is needed, please contact the Apstra support team.
1. In order to recover from the crash of BlueprintDiffProducerAgent, please increase the heartbeat_period to 1200 secs in agent_management section of aos.conf file (<=6.1.0: /etc/aos/aos.conf, >=6.1.1: /user/root/etc/aos/aos.conf) and restart AOS service (service aos restart).
[agent_management] # Override the default heartbeat timeout for agents spawned dynamically by # AgentManager. The value must be a non-negative number. The unit is seconds. # The value 0 is used to turn off heartbeat-based agent timeouts and restarts. # The minimum non-0 value allowed is 60. If not provided, then the default # timeout value (600 seconds) is used. heartbeat_period = 1200
2. Please check each configlet in the blueprint and delete the configlet with incorrect Jinja expression from the blueprint.
In Apstra 6.1.0 and later, Junos configlet containing multiline login banner text using newline escape sequences fails during commit check with a ConfigLoadError. The same configlet works as expected in Apstra 6.0 and earlier releases. In Apstra 6.1.0, a regex change was introduced in the config processing logic. As a result, the login message is incorrectly split into multiple separate lines, leaving orphaned tokens. These orphaned tokens are interpreted by Junos as unknown commands, causing the commit check to fail.
The currently viable workaround is to remove the newline escape sequences entirely from the configlet. While this avoids the parsing issue and allows commit check to pass, it results in the login banner being rendered as a single line message. Multiline formatting is therefore lost. Although not ideal, this approach prevents commit check failures.
When rack types are exported, modified, and then re-imported into a blueprint, generic systems in the rack may be unexpectedly renamed, resulting in some servers losing their original user-defined labels even though only rack parameters were changed.
Customers should avoid editing the rack type in the Global Catalog UI when they need to preserve generic system names.(1) Export the rack type from the blueprint to the Global Catalog.(2) use the API PUT /api/design/rack-types/{rack_type_id} to update only the link_per_spine_speed (e.g., from 25 to 100) directly in the rack type JSON without changing the generic system group_label or count structure.(3) re-import the updated rack type into the blueprint.
The Last Modified field of the interface telemetry service, which uses gRPC periodic mode, is inadvertently updated according to the interface telemetry service's default interval even if no data has been collected from the device. This problem is noticed when gRPC periodic mode is used for Interface telemetry service. The Last Modified field should be updated if a status change is observed, and the Last Fetched field should be updated if the device reports telemetry data.
The Platform -> Product Usage page does not take into account the quantity field in each license. It simply counts the number of licenses. Product Usage erroneously displays insufficient license warnings even though a license with multiple quantities is applied correctly since it is seen as a single license. Apstra operation would remain unaffected even if there were product usage warnings.
After configuring the GCM ciphers [email protected] and [email protected] on the Junos device, the check job began failing because these ciphers are not supported in Apstra 6.0.0. As a result, the cipher exchange between the client and server couldn't be handshaken between the device and Apstra, leading to an SSH negotiation failure leading to connection failure.
Apstra 6.1.1 supports GCM ciphers. The user needs to upgrade to Apstra version 6.1.1 to use GCM ciphers between JUNOS devices and Apstra.
Starting in Apstra 6.1.0, which introduces EVPN VXLAN over IPv6 underlay (RFE-1364), Apstra automatically applies next-hop self in the RoutesToExt routing policy for all BGP sessions toward external generic routers. This is expected behavior.
RFE-1364
In deployments using IPv6 underlay with RFC 5549, IPv4 routes may carry IPv6 next-hops. External routers using IPv4 BGP sessions cannot use these next-hops. Without next-hop rewriting, affected IPv4 prefixes may not be advertised to external routers, resulting in loss of reachability and potential service impact (for example, DHCPv4 relay failures).
Customers upgrading to 6.1.x who have external generic router peerings configured will observe next-hop self added to their RoutesToExt policy as part of the upgrade. This change ensures consistent route advertisement and requires no manual action.
No workaround is required.
On ACX7100-32C devices, configuring a port with 10GE speed (non-channelized) could result in the port failing to link up, accompanied by "Invalid Port Speed Configuration" and "Optics does not support configured speed" alarms. This occurred because the built-in ACX7100-32C device profile did not automatically generate the required "unused" configuration for the adjacent port within the same port group(for example 0 and 1 are in the same group for 10GE)
Clone the existing built-in ACX7100-32C Device Profile and update Transformation #7 (10GE non-channelized) to include an unused_port_list configuration for the adjacent interface in the same port group. If you need further assistance, please reach out HPE Apstra Support Team.
Steps:1. Clone the ACX7100-32C Device Profile to create a custom copy.2. In the cloned profile, edit port setting for Transformation #7 (10GE non-channelized port) to add unused_interface_list entries for the other port in the same port group (e.g., port 0 and port 1 share a group, port 2 and port 3 share a group, etc.).
Example) Port 0 setting
Old setting:
{"global": {"breakout": false, "fpc": 0, "pic": 0, "port": 0, "speed": ""}, "interface": {"speed": "10g"}, "validations": [{"constraint": "1x25or1x10", "port_group": "P0_P1"}, {"constraint": "no_constraint", "port_group": "P0_P1_P2_P3"}]}
New setting: add "unused_interfaces_list" key with value ["et-0/0/1"] for adjacent port.
{"global": {"breakout": false, "fpc": 0, "pic": 0, "port": 0, "speed": ""}, "interface": {"speed": "10g","unused_interfaces_list": ["et-0/0/1"]}, "validations": [{"constraint": "1x25or1x10", "port_group": "P0_P1"}, {"constraint": "no_constraint", "port_group": "P0_P1_P2_P3"}]}
3. Create new Interface Map with the updated Device Profile.
4. Assign the new cloned device profile into the managed devices and import the new IM into blueprint5. Assign the new IM into the deployed devices
Apstra anticipates that the NOS upgrade will result in a version change after the NOS device image installation and device reboot with the new image. However, JSU (Junos Selective Update) installs only selected packages without changing the version, followed by a restart process rather than a device reboot. Therefore, NOS upgrade by Apstra using JSU image will fail with the post-validation check (pre-install version != post-install version and image filename must include version information).
Please use the CLI to upgrade JSU instead of utilizing Apstra's NOS upgrade, or run Apstra NOS upgrade (make sure the JSU image filename includes version information) and then execute a full push configuration when the NOS upgrade fails due to post-validation check.
Starting in Apstra 6.1.0, the root partition is read-only, and the NTP service override configuration (override.conf) is stored in a non-standard path (/var/root/usr/local/lib/systemd/system/ntp.service.d/) instead of under /etc/systemd/. Because this path is outside the standard systemd configuration directory, systemd does not automatically reload it during boot. As a result, the NTP service starts without the override configuration applied.
Add the below crontab entry for root user (using sudo crontab -e) to automatically reload the systemd configuration and restart the NTP service on each reboot:@reboot systemctl daemon-reload && systemctl restart ntp
When the offbox agent restarts, it checks the controller's version for compatibility via HTTPS internally. From 6.1.0, the offbox agent uses the controller IP address or the worker VM's IP address as the source IP address. If ACL has a rule to block those IP addresses, the offbox agent would fail in retrieving the controller's version information and then prevent the other agent's process from starting.
if ACL is enabled in the Apstra Controller, please add allowed rules for the controller's IP address and worker VM's IP address.
When the Junos EVPN Next-hop and Interface count maximums parameter in the staged->Fabric settings->Fabric-policy is enabled, Apstra introduced modifying the default hardware settings for VXLAN routing's resource (next-hop and interfaces) for QFX5110, QFX5120, EX4650, and EX4400 devices in the rendered configuration () starting with version 4.2.0. Whenever configuration changes in VXLAN routing's resource, JUNOS triggers PFE automatic restarts to reflect new changes with service impact. The typical scenarios would be when the device becomes deployed, undeployed, or the device is in NOS upgrade. To prevent unnecessary PFE restarts in those scenarios, the configuration for VXLAN routing's resource needs to be included in the pristine configuration.
If the Junos EVPN Next-hop and Interface count maximums parameter in the staged->Fabric settings->Fabric-policy is enabled, add the below configuration into the device's pristine configuration.QFX5120 and EX4650 VXLAN routing's resource
forwarding-options { vxlan-routing { next-hop 45056; interface-num 8192; overlay-ecmp; } }
QFX5110 VXLAN routing's resource
forwarding-options { vxlan-routing { next-hop 32768; interface-num 8192; overlay-ecmp; } }
EX4400 VXLAN routing's resource (add overlay-ecmp if Junos EX-Series Overlay ECMP is also enabled)
forwarding-options { vxlan-routing { next-hop 16384; interface-num 6144; overlay-ecmp; } }
Customers may encounter the following Server-side Validation Error in the Web UI when the pristine configuration contains multiple system stanzas which is not a expected behvavior:
"Cannot parse config: system already parsed."
According to ScotchInventoryAgent logs, POST requests to update the pristine configuration failed with a 422 Unprocessable Entity error, indicating a validation issue:
2025-02-17 23:31:18,730 680:INFO:aos.scotch.libs.scotch_flask:request: POST /api/systems/AN10555621/pristine-config HTTP/1.0 34074 bytes 2025-02-17 23:31:18,737 680:INFO:aos.scotch.libs.scotch_flask:response: 422 55 bytes 0.007347 seconds
Background of the issue:
1. In Apstra 4.2.x, gRPC was introduced to support Telemetry Streaming, and as a result, having two system blocks in the pristine configuration was expected in that release. 2. Starting from Apstra 5.0.0, enhancements were made to automatically merge multiple system stanzas in the pristine configuration during the NOS upgrade process. 3. If a customer chooses to remain on their current NOS version for an extended period, multiple system stanzas can exist in the pristine configuration without causing issues.
If a customer chooses to remain on their current NOS version for an extended period and needs to forcefully update the pristine configuration, they should manually merge the system stanzas within the pristine configuration using the UI and then perform a Force Update.
For further assistance, please contact Juniper Apstra Support.
PTX10002-36QDD configured with EVPN/vxlan on Junos 23.4R2-S5-EVO have ipv6 neighbors fail
Upgrade Junos OS to version 24.4R1-EVO or later, where the issue is resolved.
Note: Junos 24.4R1-EVO and later are not part of the Apstra 6.1 qualified NOS list. Use of these versions should be validated in a lab environment prior to deployment.
The device console may display cosmetic Python traceback error messages indicating an unreachable network when Sonic ZTP is provisioning a device. In the event that the device modifies the VRF of its management interface and is unable to send logging messages to the ZTP server, this may occur.
If the device completes the ZTP process, these messages in the console can safely be ignored.
Interface telemetry uses gRPC periodic mode to collect interface status data from the JUNOS and EVO devices. Apstra (the gRPC client) needs the delivery of all interface status information every round (the default period is 120 + delta seconds). False alerts for interface anomalies results from the devices occasionally sending incomplete information (just for a partial set of interfaces) inside a round, and the interface status for absent interfaces would be represented as missing status.
The current workaround to avoid false alerts for interface anomalies is to disable gRPC temporarily until the fix is incorporated (shown in the below example) or keep gRPC enabled with changing collection mode of interface telemetry into native polling via REST API call. If further assistance is needed, please reach out Juniper Apstra Support team
[ Example to disable gRPC globally ] To disable the gRPC service, change grpc_enabled = 0 in the /user/root/etc/aos/aos.conf file and then restart AOS service in the Apstra Controller (sudo systemctl restart aos).
[telemetry_global_config] # Python multithreading enable/disable knob for telemetry collection multithreading_config = 1 # Execution timeout for extensible telemetry collectors command_timeout = 120 # Knob to enable/disable gRPC based service collectors grpc_enabled = 0
When transitioning between ipv4 or ipv6 on the underlay for the Apstra managed fabric, Qfx5120, ex4650, ex4400 devices must be rebooted to avoid issues with VTEP programming
After transitioning the Apstra fabric between ipv4 <> ipv6 address families you will need to reboot Qfx5120, ex4650, ex4400 devices to avoid issues with VTEP programming
Integrated DCI feature(vxlan stitching) was introduced in Apstra 4.2.0. In versions 4.2.x and later, customers using this feature may encounter an issue where all VTEP loopback addresses from the fabric including those from non-border leaf devices are being advertised to external routers over BGP in the default routing zone.
This affects only VXLAN DCI Stitching deployments(Stitching requirement for VTEP loopbacks for only border leaf nodes vs OTT requirement for all VTEP loopbacks in the fabric). Even when customers configure routing policies to export only loopback of border leaf nodes, Apstra backend logic automatically includes all VTEP loopbacks. Due to the current design, Apstra does not differentiate between border and non-border leaf roles in this context, resulting in the unintended advertisement of all fabric loopbacks to external peers.
Engineering has confirmed this as a bug. The expected behavior is to advertise only the loopback addresses of border leaf switches to external routers in the default routing zone. There is no official workaround to modify this behavior through standard configuration. Engineering is actively working on a fix to address this issue in a future release. The only option is to use a custom configlet to override Apstra default export logic. Please reach out to Apstra Technical Support for assistance.
Validation logic for a security zone assumes the description includes valid ASCII characters. If the description includes violating characters, the upgrade from 5.1.0/6.0.0 to 6.1.X would fail with a error related with security zone.
Validation Logic for security zone's description 1. Description should not start or end with space 2. Description should include only printable ASCII characters (ranging from 33 to 126), excluding double quotation (") character, less than (<) character, and backslash (\) character.
Please remove or replace violating ASCII characters from the security zone's description before upgrading
User-defined import / export route-targets with ":0" are rejected with validation errors.
{ "rt_policy": { "import_RTs": { "0": "Type 0 RD X:Y must be in format 2-byte ASN:4-byte value. Provided value: \"65500:0\"" } }
When the virtual infra manager is removed from the Apstra controller, Apstra should have cleared any data related to the virtual infra manager. Because it's not cleared, when the same virtual infra manager is added back to Apstra later, old data is still used together with the new collected data from the virtual infra manager's collector. In some scenarios, when old, uncleaned data has an error condition, it can trigger continuous error even if newly collected data doesn't have an error condition.
If the virtual infrastructure manager requires re-onboarding (removing and then adding back) from Apstra, the user must take the actions listed below.1. Remove virtual infra manager from Apstra Controller (External Systems/Virtual Infra Managers).2. Restart the AOS service.3. Add the virtual infra manager back to to the Apstra
From Apstra 6.1.0, users can reboot a device directly from the Apstra UI by selecting the device and choosing Reboot action. However, when a device is in Drain, Ready or Undeploy mode, the reboot with force option is not available in the UI, even though the underlying functionality should permit it. As a temporary workaround, customers can use the Apstra REST API to initiate an agent reboot while a device is in Drain, Undeploy or Ready mode. This API method works as expected.
Customers can reboot the device agent using the REST API as follows:
1) Navigate to Platform > Developers > REST API Explorer. 2) Select POST /api/system-agents/{agent_id}/reboot. 3) Set the force field to true. 4) Click Try it to execute the request.
Because the Jinja comment is not recognized as a valid Jinja expression during rendering, it is submitted as normal commands to the device, causing the JUNOS/EVO device to reject the invalid commands and deployment to fail.
To make the template "Jinja-aware", the user needs to include a no-op Jinja construct at the top of the configlet. This will bypass the stringent line-by-line set/delete checks. The workaround entails introducing a dummy control block or expression directly after the Jinja comments.
configlet example with error
{# v1.0 - 22 Jan 2026 - Author: ... - Initial version #} {# Objective: Example of non-working configlet #} set system time-zone Europe/Luxembourg
configlet example with workaround
{# v1.0 - 22 Jan 2026 - Author: ... - Initial version #} {# Objective: Example of working configlet #} {% if 1 > 0 %}{% endif %} set system time-zone Europe/Luxembourg
With current cluster design, worker nodes may fail to recover their configuration state after a temporary loss of SSH connectivity to the controller. This condition can be observed in the UI by navigating to Platform > Apstra Cluster > Nodes > Worker, where the following error may be displayed:
Configuration Error: ssh: connect to host 10.28.17.4 port 22: Connection refused
When the controller (ClusterManagerAgent) attempts to push configuration to worker nodes, it retries SSH connections up to three times. If all attempts fail, due to connection refused or no route to host, the node is marked with a FAILED Configuration State. Once this state is set, the system does not automatically retry configuration, even if SSH connectivity is later restored. Although worker nodes may resume sending keepalives and transition back to an active operational state, the configuration state remains in failed state indefinitely. Due to this the overall node state may continue to appear FAILED despite restored connectivity.
Manually trigger a configuration synchronization using one of the following methods:
1. Navigate to Platform > Developers > REST API Explorer 2. Execute REST API: POST /api/cluster/worker/sync [OR] 1. Restart AOS from the controller VM: systemctl restart aos
When sFlow collector is configured with mgmt_instance, the configuration will be ignored in the ACX platform with a warning such as the below example.sflow {polling-interval 10;sample-rate {ingress 10000;egress 10000;}source-ip 10.217.6.15;collector 10.217.0.165 {udp-port 6343;#### Warning: statement ignored: unsupported platform (ACX7024X)##routing-instance mgmt_junos;
##
## Warning: statement ignored: unsupported platform (ACX7024X)
Please use a non-management instance for exporting sFlow until the ACX platform supports a management instance for sFlow export.
The NXOS rollback feature on the N9K-C93600CD-GX device has significant limitations when the devices' interfaces are broken out.Ports 1-24 in the model are organized into four-port groups: (1, 2, 3, 4), (5, 6, 7, 8), (9, 10, 11, 12), (13, 14, 15, 16), (17, 18, 19, 20), and (21, 22, 23, 24). When port 1 is broken out as 4x10G or 4x25G, port 3 is automatically broken out in the same mode, and vice versa. When any port in the quadruple is split into 2x50G, all four ports are automatically split in the same mode. Similarly, ports 26-28 are organized in pairs of two, i.e. (25, 26) and (27, 28). Both ports in the pair must operate in the same breakout mode.
In most cases where a breakout (or more than one) exists, rollback fails to generate a working rollback patch. The reason for this is that the breakouts cannot be reversed if the remaining broken-out interfaces in the same port group have not been shutdown first. For example, to negate the breakout of port 1, the broken-out interfaces of port 3 must be shutdown, and vice versa. It appears that the rollback logic shuts down the interfaces associated with the port whose breakout is being reverted (port 1 in the previous example), but fails to shut down other broken-out ports in the same port group (port 3).
The safer way for the N9K-C93600CD-GX to be used with AOS is for the customer to avoid using breakouts altogether on the device.No issue with rollback when ports 29-36 have been broken out has been observed. Breakouts on these ports can be rolled back In the case that the last interface of a port-group is the only one used and broken out, would the nxos rollback feature (and rollback to pristine) be successful. However this is highly discouragedIn any other case the only way to reverting to pristine would be to manually shudtown all broken down interfaces before reverting to pristine (or using the rollback to a pristine config)
The QFX10008 device with dual routing engines currently doesn't support FIPS Level. Even when the additional configuration (IPSEC) is configured after FIPS is enabled, FPC doesn't come up online correctly, which prevents normal operation.
When FIPS is enabled, interface and MAC telemetry services encounter gRPC connection errors for JUNOS and EVO devices running off-box agents. The issue is related to the device's inability to properly initialize the SSL certificate directory during FIPS activation, preventing JSD from locating the required certificates at startup.
Recommend NOS upgrade to the fixed version, JUNOS >= 23.4R2-S7 and EVO >= 23.4R2-S7-EVO.
A new default limit for the number of IP addresses per MAC per bridge domain in EVPN (mac-ip-limit) was added in Junos and Junos Evolved Releases 24.2R1, 23.4R2, and 23.2R2. 200 IPs per MAC is the default setting.
Clients who use EVPN fabrics, such as those with MAC-VRF deployments in fabrics managed by Apstra, might observe that MACs linked to more than 200 IP addresses cease to learn new IPs in the EVPN MAC-IP table. The Junos software release introduced this expected behavior.
The fabric can support more IPs per MAC while preserving per-bridge-domain enforcement by setting mac-ip-limit globally using an Apstra Configlet. No software fix is required.
To adjust the limit in an Apstra-managed fabric, create a Configlet in Apstra with the below command, specifying the desired limit:
set protocols evpn mac-ip-limit <desired-value>
Import the Configlet into the blueprint and apply it to the relevant switches.
Note: Although the command is global in Junos, the limit is enforced per MAC per bridge domain, including inside MAC-VRFs.
In Junos EVO version 23.4R2-S5-EVO and 23.4R2-S6-EVO, devices do not export sFlow packets through the management interface. This issue affects all QFX device models.
You can resolve this issue using one of the following approaches:
1. If you require sFlow export through the management interface, please use the Apstra-qualified Junos EVO release 23.4R2-S4-EVO instead of 23.4R2-S5-EVO.
2. If you are using Junos EVO release 23.4R2-S5-EVO, configure sFlow to export through a non-management (in-band revenue) interface instead of the management interface.
2026-06-15: Added AOS-61764
2026-06-08: Added AOS-61624
2026-05-26: Updated AOS-59416, AOS-60637
2026-05-15: Updated AOS-54864
2026-05-14: Added AOS-61108
2026-05-06: Added AOS-57295
2026-04-21: Added AOS-52788, AOS-60637
2026-04-14: Added AOS-60612
2026-04-13: Updated AOS-60011,Added AOS-59984
2026-04-09: Added AOS-54864,AOS-60011
2026-04-03: Added AOS-55780
2026-03-23: Added AOS-60167,AOS-60183,AOS-60193
2026-03-18: Removed AOS-51029,Updated AOS-59821,Added AOS-59842,AOS-60093,AOS-60114
2026-03-16: Added AOS-60061
2026-03-05: Added AOS-59821,AOS-59931
2026-03-03: Added AOS-59726,AOS-59869,AOS-59895
2026-02-22: Added AOS-59758,AOS-59777
2026-02-18: Removed AOS-58460, Added AOS-58991,AOS-59596
2026-02-11: Added AOS-59269,AOS-59579,AOS-59663
2026-02-03: Added AOS-59198,A0S-59416
2026-01-28: Updated AOS-56175,AOS-56282
2026-01-27: Added AOS-59296
2026-01-20: Initial Publishing.