This article explains how one faulty ADC can impact adjacent ADC's & how to better handle such issues.

There are many SPMB I2C error logs from the customer setup even before the faulty ADC was detected. I'm reviewing these logs determine if the issue is with the ADC or the SPMB which is affecting other FRUs.

Also, as you suggested isolating the ADC initialization process such that it does not affect other FRUs, is actually not possible in my opinion, since the SPMB is itself responsible for detecting the ADC presence & reporting the same to Chassid. From logs need to make sure if faulty ADC led to SPMB issues or its the other way around.

From 22OCT logs I found the following - 

ADC0 -->
37618 Oct 22 03:36:16  Router-MX2K8-001-RE0 : %PFE-6: spmb0 cmty_adc_offline_req: adc 0 adc_i2c=0x0 reason=1 fault_reason=1
37622 Oct 22 03:36:16  Router-CPP1-MX2K8-001-RE0 : %PFE-6: spmb0 cmty_adc_offline_req: adc 0 reason and fault reason overridden to  reason=1 fault_reason=0
 
ADC1-->
37649 Oct 22 03:36:16  Router-CPP1-MX2K8-001-RE0 : %PFE-6: spmb0 cmty_adc_offline_req: adc 1 adc_i2c=0x0 reason=1 fault_reason=3
37653 Oct 22 03:36:16  Router-CPP1-MX2K8-001-RE0 : %PFE-6: spmb0 cmty_adc_offline_req: adc 1 reason and fault reason overridden to  reason=1 fault_reason=0

fault_reason are defined below - 

 13 typedef enum ty_flt_reason_ {
 14     TY_FLT_UNKNOWN = 0,
 15     TY_FLT_POWER_ON,                 /* Failure during power up */
 16     TY_FLT_LINK_ERRORS,              /* Failure due to link errors */
 17     TY_FLT_PIO_ERRORS,               /* Failure due to PIO errors */
 18     TY_FLT_ASIC_INTERNAL_ERROR,      /* Failure due to ASIC fatal interrupt */
 19     TY_FLT_I2C_BUSY_ERROR,           /* Failure due to I2C busy condition */
 20     TY_FLT_ASIC_CELLDRP_ERROR,       /* Failure due to ASIC cell drops */
 21     TY_FLT_PMBUS_ERROR,              /* Failure due to PMBUS errors */
 22     TY_FLT_IDPROM_ERROR,             /* Failure due to IDPROM errors */
 23     TY_FLT_MAX_POSSIBLE_ERRORS       /* New faults should be defined before this */
 24 } ty_flt_reason_t;
fault_reason=1 --> Failure during power up, fault_reason=3 --> Failure due to PIO errors, however both these are overridden by fault_reason=0 --> reason unknown.

In the code if an ADC is marked as faulty then in the ADC state machine an offline request is sent for that ADC, which is also seen from the logs and that ADC is brought down. It should not be doing anything with other FRU states.

As confirmed by Engineering its already been confirmed that its hardware faulty due to which adjacent ADC were also misbehaving.

We can confirm that the issue occurs specifically when a ADC is inserted in slots adjacent to a faulty ADC slot.

As a workaround, we can advise the customer to avoid placing any FRUs in the slots adjacent to a known faulty ADC.