Software Quality: What you don’t know CAN hurt you

On Tuesday of this week, Jim Lentz, president of Toyota Motors Sales USA, testified before the House Energy and Commerce Committee about the car manufacturer’s problems and recalls.  Yes, it was Mr. Lentz’s turn in the proverbial hot seat.

 

During pointed questioning by House members, Mr. Lentz said that the repairs prescribed by Toyota might “not totally” solve the problem of unintended sudden acceleration in its vehicles. He also admitted that Toyota is still trying to determine the source problem, including the possibility – previously denied – that the vehicles’ electronics systems (software!) might be at fault.

 

My point here is not to join the chorus of Toyota-bashers. Rather it is to point out that major software quality problems seem to be occurring with increasing frequency across more industries today – and they are definitely happening more frequently in the embedded industry. 

 

It’s strange because most companies have made significant investments in the people, processes and technologies they need to manage software quality levels in their projects. Yet we see this massive problem for Toyota (although the jury is still out on the software question). But other, confirmed, high-profile software problems in embedded devices have happened just this past year alone with major aerospace, networking, smart phone and medical device companies. All of these episodes were painful for the companies and executives involved.

 

So what is really going on here? Why does it seem like management and product development teams have blinders on when it comes to certain aspects of quality in their products? 

 

The answer is that a confluence of industry trends is causing companies to race ahead of their software quality testing capabilities. Primary among them are:

 

• Skyrocketing software content – analysts’ estimate that software content in embedded devices is doubling every two years.
• Architectural complexity – 32- and 64-bit architectures and multi-processors are now the norm, with multi-core coming soon to a device near you. 
• Iterative/‘agile’ development– one long development cycle has been replaced by lots of shorter ones, making testing a non-stop exercise aimed at a moving target.
•  Short-fuse delivery schedules – Product development teams grappling with their need to deliver complex products with feature sets that really ‘wow’ the customers in much tighter timeframes.

Executives at embedded product companies are discovering a significant gap between their perceived and actual level of delivery quality. They are realizing with these trends their existing testing tools and methods alone are not going to be enough. That’s why when it comes time to make that critical call on product readiness, they are increasingly off balance and anxious. 

 

These issues, as well as new game-changing strategies for dealing with them and countering the risks, are detailed in a new executive white paper from Wind River’s Device Test product division.  There's a new technique for embedded testing that lets teams gather actionable intelligence directly from their device under test and use this information to optimize their quality processes.

 

The paper is entitled Don’t Crash at the Finish Line – Overcome Skyrocketing Device Complexity to Deliver Products On-Time and On-Quality. Drop me an email and I’ll send you a copy (paul.henderson@windriver.com).

 

Today the president and CEO of Toyota goes in front of Congress to get grilled about his company’s quality problems. With his family name on the company’s front door, Akio Toyoda likely isn’t worried about his job or career. But for any embedded industry executive whose name isn’t on the front grille or hood ornament of their products, it’s time to take a new look at testing. 

Comments

[Michael Barr]

Paul,

Thanks for a great post about an important issue and trend.

Embedded software is turning up increasingly in courtrooms. Did you hear about the Drager Alcotest 7110 breath analyzer source code? After a two-year fight, seven defendants in New Jersey DUI cases won the right to have experts review the source code for the firmware. Each side produced expert reports evaluating the quality of the code. Although the experts disagreed about the overall quality, several flaws were found in the process.

I wrote about related issues in the December issue of Electronic Design magazine:

http://electronicdesign.com/content.aspx?topic=faulty_code_will_lead_to_an_era_of_firmware_related_litigation&catpath=embedded-software

Cheers,
Michael Barr
Netrino, LLC