Compressed Schedules Driving Shorter Testing & Defect Resolution Requirements

Today I’m continuing my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog posting).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In part 2 of the survey we asked about schedule compression and what affect that was having on the device testing cycle. A majority of survey participants reported that market conditions have forced them to shorten their development schedules by as much as 18 months.

Many companies are turning to iterative methods to improve feedback within their product cycles. Some are using Agile techniques. However, most respondents are still heavily using manual techniques for system testing and especially for more complex testing of error conditions and performance validation. Defect isolation and repair cycles are typically measured in days per bug. Together these bottlenecks are increasing the need for new approaches to test automation.

Shorter Release and Build Cycles

A majority (64%) of respondents indicated that market conditions have forced them to shorten their development schedules. In terms of the length of their development cycles, a majority of respondents (57%) reported current cycle times of between six and eighteen months.  Of those, 44% reported having to cut their standard development cycles by six to twelve months. 

The survey also asked about the adoption of new Agile development methodologies. Of the respondents who knew, the majority of the participants (64%) reported that they have not adopted Agile methods.  However, 57% reported that they are running bi-weekly build cycles or less with 16% of these actually doing nightly builds.

So despite most teams not considering themselves formally Agile, most teams are using iterative processes.  Surprisingly, 18% indicated that they don’t operate on a formal, fixed schedule for their build cycles. 

Difficult to Evaluate Device Behavior at Runtime

A high majority of participants (over 81%) reporting that their teams test each new software build on the embedded device itself.  This shows that respondents recognize the importance of testing software on the integrated device, rather than just conducting unit tests. Surprisingly, 81% of respondents cited manual testing tools as part of their testing process today, so presumably, manual testing within this iterative environment is becoming a challeng.

Regarding the time it takes their teams to isolate and repair defects found during system integration, more than 62% reported that it takes them a full day or longer.  Over 25% said it takes them more than 2 days. This is clearly a bottleneck to the process and an expense that needs to be minimized.

The survey also asked how users were testing failure modes, error conditions and performance. Fifty seven percent (57%) of respondents said they were using manual methods to verify failure modes and 8.3% said they did not verify these conditions at all. Forty one percent (41%) of those who knew said their tests were not able to measure performance of their devices at runtime.

Numerous participants also commented on the difficulty of understand precisely what is happening within their device under test at runtime.

 “Multi-core products are creating performance challenges for the architects and more corner cases which are difficult to test.”

“Software must be tested thoroughly  under all possible operating conditions.”

“(It’s) difficult to "see inside" linked programs.”

In a somewhat surprising finding about their test environments, more than half of all participants (56.9%) reported that their software development and testing processes are not governed by a safety, security or other certification standards. Of the participants who responded in the affirmative, adherence was fragmented across many different standards.

Next time I’ll review respondents’ feedback about quality measurement and how lack of quality visibility is eroding management confidence.