Tools, Testing, & Virtual Systems

Inadequate Management Visibility into Quality is Eroding Confidence

By Paul Henderson

Here's the next installment in my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog postings).  If you'd like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In this section of the survey we asked participants about how they measure software quality today, the metrics most often cited by survey respondents were reactive in nature such as tracking customer-reported failures and open defects rather than metrics that can help them prevent defects.

Survey participants clearly indicated that they have a strong desire to be more proactive, but their responses also reveal a significant disparity between their intentions and their actual testing. The survey uncovered a significant disparity between testing goals and reality (as measure by test coverage analysis), where a minority of respondents actually have access to the information they need to assess quality. Respondents further report that often the software quality information they do receive is later proved to be wrong, leaving them to make product readiness decisions on information that turned out to be inaccurate.

Respondents seem to know that you can't fix what you can't measure.  They also know that the answer lies in more thorough testing done more often. But the common metrics in use, especially the customer-reported problems, appear to count how many “horses have already left the barn.”  The disparity between testing goals and actual results, the management blind spots regarding software quality levels, and the decision making based on bad information are all contributing to a crisis in confidence in software quality in the embedded industry. 

Metrics Not Providing Management Confidence

When asked if their exiting tools give management sufficient software quality information to make release readiness decisions with confidence, 75% of respondents said that they were at best only “somewhat confident.”  With 19.1% of respondents, their uncomfortable answer was flatly “No, not confident.” 

It seems that a high percentage of participants have learned to be skeptical about the software quality information upon which they must make decisions.  Of those that responded that they knew one way or the other, 57% said that they have made release readiness decisions based on inaccurate or insufficient software quality information. 

One telling group of data points is the responses to a question about which metrics respondents use to measure the software quality in their products.  By far, the two most frequent answers were:

- The tracking of the number of open defects

- The numbers of defects reported by customers

Notably, both of these measures are reactive; they essentially gauge the number of defects already introduced into their products.  The other, less-used metrics include test coverage metrics, numbers of tests executed, etc. were more proactive measures that can be controlled and managed by the companies. 

Inadequate Test Coverage Information

Their lack of confidence and reactive management approach can also be linked to the low use of test coverage and code coverage tools. These tools provide insight into whether software was actually tested (or not) and how thorough test suites are. They should be used in both unit testing at the developers desktop as well as in the functional system testing in the QA lab.  

Overall only a minority of participants much use of code coverage and test coverage tools. This is evidenced in respondents' answers to the question as to whether they track which parts of their code is actually exercised by their tests.  Over 60% said they either don't track test coverage at all or only do so during development unit testing. Only about 32% said they measure and track what code is exercised in system tests of the fully integrated device. 

Perhaps the most telling response in this section of the survey was in response to questions about test coverage goals and actual test coverage results with their current projects.  Nearly 65% of participants reported having goals for test coverage of their code bases of over 70%.  But only 35% of respondents believe they are at greater than 70% code coverage with the project they are currently working on. 

Next time I will cover respondents' feedback on the high cost of poor quality and how they measure this.

Compressed Schedules Driving Shorter Testing & Defect Resolution Requirements

By Paul Henderson

Today I'm continuing my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog posting).  If you'd like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In part 2 of the survey we asked about schedule compression and what affect that was having on the device testing cycle. A majority of survey participants reported that market conditions have forced them to shorten their development schedules by as much as 18 months.

Many companies are turning to iterative methods to improve feedback within their product cycles. Some are using Agile techniques. However, most respondents are still heavily using manual techniques for system testing and especially for more complex testing of error conditions and performance validation. Defect isolation and repair cycles are typically measured in days per bug. Together these bottlenecks are increasing the need for new approaches to test automation.

Shorter Release and Build Cycles

A majority (64%) of respondents indicated that market conditions have forced them to shorten their development schedules. In terms of the length of their development cycles, a majority of respondents (57%) reported current cycle times of between six and eighteen months.  Of those, 44% reported having to cut their standard development cycles by six to twelve months. 

The survey also asked about the adoption of new Agile development methodologies. Of the respondents who knew, the majority of the participants (64%) reported that they have not adopted Agile methods.  However, 57% reported that they are running bi-weekly build cycles or less with 16% of these actually doing nightly builds.

So despite most teams not considering themselves formally Agile, most teams are using iterative processes.  Surprisingly, 18% indicated that they don't operate on a formal, fixed schedule for their build cycles. 

Difficult to Evaluate Device Behavior at Runtime

A high majority of participants (over 81%) reporting that their teams test each new software build on the embedded device itself.  This shows that respondents recognize the importance of testing software on the integrated device, rather than just conducting unit tests. Surprisingly, 81% of respondents cited manual testing tools as part of their testing process today, so presumably, manual testing within this iterative environment is becoming a challeng.

Regarding the time it takes their teams to isolate and repair defects found during system integration, more than 62% reported that it takes them a full day or longer.  Over 25% said it takes them more than 2 days. This is clearly a bottleneck to the process and an expense that needs to be minimized.

The survey also asked how users were testing failure modes, error conditions and performance. Fifty seven percent (57%) of respondents said they were using manual methods to verify failure modes and 8.3% said they did not verify these conditions at all. Forty one percent (41%) of those who knew said their tests were not able to measure performance of their devices at runtime.

Numerous participants also commented on the difficulty of understand precisely what is happening within their device under test at runtime.

 “Multi-core products are creating performance challenges for the architects and more corner cases which are difficult to test.”

“Software must be tested thoroughly  under all possible operating conditions.”

“(It's) difficult to "see inside" linked programs.”

In a somewhat surprising finding about their test environments, more than half of all participants (56.9%) reported that their software development and testing processes are not governed by a safety, security or other certification standards. Of the participants who responded in the affirmative, adherence was fragmented across many different standards.

Next time I'll review respondents' feedback about quality measurement and how lack of quality visibility is eroding management confidence.