Industry Investing in Better Device Runtime Visibility During Testing

Here’s the final installment in my series about our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog postings).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In this section of the survey we asked participants about what test tools they use today and where they are investing in test automation. Given the high cost of product failure, accelerating complexity and reduced schedules the industry is turning to more test automation in 2010 to help address these problems. The top investment are moving to new tools that can help test teams and their management better understand how well they are testing, better focus their efforts on the areas needing testing, and reduce cycle time through more automation.

Despite Existing Tools, More Automation Required

Participants were asked various questions about their present software quality testing environment and their plans and priorities for new investments to improve their testing operations in the coming year. Test automation remains top of mind for respondents with 68% of those who knew saying they plan to automate more of their test processes this year.

As for the tools currently in use in their testing processes today, respondents not surprisingly reported that they use a wide variety of systems and techniques.  Results did show that manual testing, conducted by 81.7% of respondents, remains a substantial part of the device test processes at most organizations today.  Other well-represented categories included defect tracking systems, static or dynamic code analysis, and performance and timing measurement. There was a strong showing by simulators, which were reported as being used by over 50% of survey participants. This category includes both device simulation environments as well as traffic, load, and other external simulators that drive device behavior during the test process.

The Top 10 List for 2010

Regarding their plans for new investments, there is a clear need for more automated testing among respondents. The top 10 rank order of tools for planned investment reported was;

1)            Automated test equipment

2)            Code and test coverage analysis

3)            Simulators

4)            Runtime diagnostics tools

5)            Static or dynamic code analysis

6)            Performance and timing measurement

7)            Test execution engines

8)            Script authoring tools

9)            Test script capture/replay tools

10)          Fault injection & runtime behavior monitoring

Runtime Visibility of Device Under Test Needed

Notably, in the top 10, at least 5 of these tools are focused on helping testers understand more effectively what is happening in their code at runtime. Respondents indicated that purchasing tools for these runtime analytics is a higher priority that obtaining more traditional tools for manual testing, defect tracking or lifecycle quality management.

The survey also asked an open-ended question about other trends or investments that should be considered. A significant number of respondents expressed the need for better host-based testing and device simulation so to streamline testing early in the process before hardware available and to reduce the cost of testing assets.

“The biggest problem we have is usually not able to test for a new ASIC before the ASIC is taped out…. It would be nice if there are tools that could be used to develop production code alongside the simulator that we can use for testing before an ASIC arrive, and run in the high level system.”

“There is a strong need for host based "device dependent code" testing suite or methodology.”

“Need better simulations of embedded devices as real hardware can be expensive.”

Conclusions

I hope this series was interesting for you. We found it ratified some of our anecdotal evidence while also surfacing additional areas where we will need to focus our investments and services. It does show that industry trends and market forces have pushed the requirements for software quality testing well beyond the capabilities of the tools and processes currently in use.  The problem is getting worse with more code crammed into devices, architectural complexity off the charts, and squeezed development and test cycles. This is the new normal.

This disparity is causing serious operational problems for embedded products companies, negatively impacting not only their brands but also their bottom-line results. Ignoring the problem is no longer an option. Leaders are seeking new test automation tools that can help them with the unique problems of embedded device testing while helping them deliver on-time, on-budget, and on-quality.

One respondent summarized it best:

“We spend too little time testing and too much time hoping.”  

It’s time to take action.