SSR devices with LTE as one of the WAN interfaces sometimes raise alarm mentioning the interface is down. Alarm looks like below
============================================================= 2024-05-29T06:29:45.665Z Intf lte (5) operationally down ============================================================= Type: alarm.add Node: ro1 Action: add Category: network-interface Interface: 5 Process: stateMonitor Serial: 12 Severity: critical Shelved Status: not-shelved Uuid: e267a13-5d47-471a-9877-c461bd2bfd Value: 2
In case of lte interface down alarms, consider checking below points.
Check if the USB device is connected and still detected
[root@128t-ro1 ~]# lsusb Bus 001 Device 003: ID 1199:9071 Sierra Wireless, Inc. Bus 001 Device 002: ID 8087:07db Intel Corp. Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Scan the path /sys/bus/usb/drivers/qmi_wwan/*/net to detect the QMI supported interface names. Validate this interface names within the SSR configuration.
[root@128t-ro1 ~]# ls /sys/bus/usb/drivers/qmi_wwan/*/net/ /sys/bus/usb/drivers/qmi_wwan/1-3:1.8/net/:wwp0s21u3i8 /sys/bus/usb/drivers/qmi_wwan/1-4:1.10/net/:wwp0s21u4i10 /sys/bus/usb/drivers/qmi_wwan/1-4:1.8/net/:wwp0s21u4i8 /sys/bus/usb/drivers/qmi_wwan/2-1:1.8/net/:wwp0s21u1i8
Verify the journal logs / kernel logs for any USB soft disconnects
[Mon May 6 22:31:55 2024] qcserial ttyUSB2: Qualcomm USB modem converter now disconnected from ttyUSB2
In such case make sure it is connected back later
[Mon May 6 22:31:59 2024] usb 2-1: Qualcomm USB modem converter now attached to ttyUSB2
Incase of USB disconnects, reinit script will be triggered by highway as an attempt to recover. This is most case will bring back the interface up in few mins.
For cases where reinit could not recover will fail mostly with python traceback error in the log, Something like below
"May 07 00:04:24 128t-ro1 highway[4040]: Script ""/etc/128technology/plugins/network-scripts/lte/reinit"" failed execution stderr: Traceback (most recent call last): File ""/etc/128technology/plugins/network-scripts/lte/reinit"", line 142, in <module> main() File ""/etc/128technology/plugins/network-scripts/lte/reinit"", line 74, in main lte.initialize_connection() File ""/etc/128technology/plugins/network-scripts/default/qmi/lte_device.py"", line 321, in initialize_connection self._reinitialize_data_connection() File ""/etc/128technology/plugins/network-scripts/default/qmi/lte_device.py"", line 140, in _reinitialize_data_connection status, output = network.start() File ""/etc/128technology/plugins/network-scripts/default/qmi/qmi_network.py"", line 163, in start self._start_init() File ""/etc/128technology/plugins/network-scripts/default/qmi/qmi_network.py"", line 72, in _start_init self._update_profile() File ""/etc/128technology/plugins/network-scripts/default/qmi/qmi_network.py"", line 96, in _update_profile if self.profile_index and self._should_recreate_profile(): File ""/etc/128technology/plugins/network-scripts/default/qmi/qmi_network.py"", line 107, in _should_recreate_profile profile = lte_helper.get_profile(cli, self.profile_index) if cli else None File ""/etc/128technology/plugins/network-scripts/default/qmi/lte_helper.py"", line 45, in get_profile profile_list = get_profile_list(cli) File ""/etc/128technology/plugins/network-scripts/default/qmi/lte_helper.py"", line 41, in get_profile_list return cli.run_command(command).get(""profile list"", []) AttributeError: 'NoneType' object has no attribute 'get' stdout:"
In such scenario, consider checking the health of the modem using AT commands "AT+CGMM" , "AT+CGMR" etc and see if it responds back.
If modem is not responding an additional step you can consider is physically disconnecting modem and connect it back again or get in touch with modem vendor.