ARTICLE - $154 Million- Color it Gone!

We have yet to come across an organization that didn’t have its share of Latent Organizational Weaknesses (LOWs).  A preliminary report from NASA indicates that human error, coincident with a LOW (or few) were responsible for the recent loss of the $154 million Mars Global Surveyor spacecraft.  Care to guess what the LOWs were? 

The preliminary report, issued by NASA on April 13, 2007, stated, “…the mission team followed existing procedures and processes.  However, these procedures and processes were inadequate to catch the errors that occurred.”

What happened?

In September 2005, ROUTINE updates were made to the High Gain Antenna (HGA) parameters in each of the spacecraft’s two redundant control systems.  The updates were made by different individuals at different points in time (each apparently having a different understanding of the required “precision” of the input parameters).

In June 2006, a full memory readout revealed the discrepancy between the redundant control systems, and an upload was commenced to fix the problem.  Unfortunately, the “fix” was loaded to the wrong memory address within the spacecraft’s “brain” (even though procedures were apparently followed).  This resulted in a disabling of the solar array positioning units and corrupted the pointing direction of the High Gain Antenna.

On November 2, 2006  a command sent to the spacecraft by NASA caused the solar array to attempt to rotate beyond its hard stop.  The onboard fault protection system responded, and in its own attempt to correct, positioned the spacecraft such that one of the batteries was exposed to direct sunlight.  The battery overheated.  The power management software interpreted this as an overcharge condition, and terminated charging voltage to the battery.  The solar arrays were unable to maintain adequate voltage with the single remaining battery.  Ten to twelve hours later (five to six orbits), all signals (and the spacecraft) were lost.

Latent Organizational Weaknesses.  Were you able to pick them out?

It appears we had a daisy chain of events, initiated by human error resulting from inadequate procedures and processes.  And, oh by the way, the report continues on to include, (1) design weaknesses, (2) lack of periodic reviews to ensure that spacecraft control parameters remained adequate, (3) lack of assessment of risks associated with personnel turnover, and (4) lack of uniform training for personnel.

Sound familiar?

Nasa’s report went on to say, “Many of the factors contributing to this anomaly would not have occurred if these operating procedures and processes had been made more thorough, or if the mission team had recognized the inadequacies and had addressed them.”

The good news is that the spacecraft was on its fourth mission extension, having served us well for the past ten years.

The Message?

Latent Organizational Weaknesses are everywhere.  We work in organizations created by people, using processes and procedures created by…people.  People are fallible.  This is a great reminder of Practicing Perfection^® Precept #2:

84 to 94 percent of all human error can be directly attributed to process, programmatic, or organizational issues    

The great news is that by employing the strategies and tools involved with Practicing Perfection^® and The HU Factor™, we can eliminate LOWs and we can prevent catastrophe in their presence.  Two specific elements that might have precluded the NASA incident are Questioning Attitude and Proactive Accountability. 

To read a copy of NASA’s preliminary report, click here.

Published: Wednesday 24 April 2007