This article explains as to why false error messages (like Voltage read failed) might be seen on MX10003/MX10008 platforms. The message point to an error that the system detected when trying to read the voltage from a component.

We could be observing errors related to I2C read failure on several components but the components maybe working fine. 

Feb 11 16:32:22 router fpc0 mpcs_i2c_single_io: MPCS(0) ctlr 0 group 4 addr 0x4d prio 0 flags 0x10 failed status 0x1

Feb 11 16:32:22 router fpc0 I2C Failed device: LTC3887-1:U15202 :EA1_VDD0V9R2, group 0x33 address 0x4d

Feb 11 16:32:22 router fpc0 pmbus_npcfpc_smb_write: npc_smb_write failed for group 0x33 addr 0x4d cmd 0x0 bytes 1

Feb 11 16:32:22 router fpc0 ltc3880_pmbus_access_setup: Could not enable channel (channel 0)

Feb 11 16:32:22 router fpc0 pmbus_read_volt: summit-mx3ru-mpc - LTC3887-EA1-VDD0V9R2-CH0: pmbus read failed for cmd 0x8b

When not getting these messages continuously at same time point. That is this error occurs once for every time.

Then this maybe regarded as false alarm message and cosmetic problems since following readings are successful (no messages are reported again). Also, no service impact. And the component for which i2c error is reported is normal functioning.

This issue is tracked under PR1405787.

Cause of these false alarm is that its internal i2c bus can face congestion. So the software might not be able to communicate on the first attempt. This PR tweaks the retry and tuning of the alerting mechanism. That is, if the i2c bus can fail the read operation. There is a retry operation to handle this specific error. Please note, if this is not a false alarm and actual hardware problem, we will observe these messages repeating at same point of time. 

Jan 25 18:59:12 router fpc0 mpcs_i2c_single_io: MPCS(0) ctlr 0 group 4 addr 0x4d prio 0 flags 0x10 failed status 0x1

Jan 25 18:59:12 router fpc0 I2C Failed device: LTC3887-1:U15202 :EA1_VDD0V9R2, group 0x33 address 0x4d

Jan 25 18:59:12 router fpc0 pmbus_npcfpc_smb_write: npc_smb_write failed for group 0x33 addr 0x4d cmd 0x0 bytes 1

Jan 25 18:59:12 router fpc0 ltc3880_pmbus_access_setup: Could not enable channel (channel 0)

Jan 25 18:59:12 router fpc0 pmbus_read_volt: summit-mx3ru-mpc - LTC3887-EA1-VDD0V9R2-CH0: pmbus read failed for cmd 0x8b

Jan 30 14:03:38 router fpc0 mpcs_i2c_single_io: MPCS(0) ctlr 0 group 4 addr 0x4d prio 0 flags 0x10 failed status 0x1

Jan 30 14:03:38 router I2C Failed device: LTC3887-1:U15202 :EA1_VDD0V9R2, group 0x33 address 0x4d

Jan 30 14:03:38 router fpc0 pmbus_npcfpc_smb_write: npc_smb_write failed for group 0x33 addr 0x4d cmd 0x0 bytes 1

Jan 30 14:03:38 router fpc0 ltc3880_pmbus_access_setup: Could not enable channel (channel 0)

Jan 30 14:03:38 router fpc0 pmbus_read_volt: summit-mx3ru-mpc - LTC3887-EA1-VDD0V9R2-CH0: pmbus read failed for cmd 0x8b

This message is considered a false alarm, and no further messages will follow. It could either evolve into a real issue or remain a false issue, and this PR resolves the false alarm. The PR focuses on improving the retry logic and fine-tuning the alerting mechanism. While the addition of the retry mechanism will not eliminate PMBus read errors, the final value may be accurate after several read attempts.