A Crisis of Complexity – Industry Report on Growing Challenges in Embedded Testing

I’ve been talking a lot with embedded device companies around the world over the last few years and I am hearing growing concerns about software testing. I’ve mentioned several of these concerns in previous blogs. I wanted to get more quantifiable data and get some feedback that could help us shape our products and services to help. So I decided to run a survey to gather important data from our community.

The focus of this survey was to gain a detailed snapshot of how executives, development managers, QA and test team leaders and other involved staff currently view the embedded device software quality test landscape.  Recent changes, new challenges and strategies for managing them were of particular interest.  So I fielded a four-part survey to individuals who work for embedded products companies. In total, nearly 35,000 individuals in North America were invited to participate in the survey via emails. Participation was also promoted on the Wind River Website and on executive blog.  The survey was conducted between April 2nd and May 13th, 2010 using an online survey tool. 

Huge Response
Amazingly, I received nearly 900 responses (889 total) from folks in the industry. While 47.8% of respondents identified themselves as software developers, another 46.4% of participants have roles in management and test (including executives, directors, managers, project leads, quality officers, etc).  The sizes of companies from which responses were received were diverse and reflective of the industry in general. Respondents split with about 50% coming from companies with 1500 or less employees and 50% coming from companies with larger than 1500 people. About 41% work at companies with more than 5000 employees. A total of 54.1% of participants identified themselves as Wind River customers; over 40% of respondents said they were not customers

In this blog and 4 subsequent postings I will review some of the highlights of the summary report I created from the aggregate results. If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

Key Takeaway

Overall the survey shows that embedded device market dynamics, including skyrocketing software content and new architectural complexity, are challenging development and test teams in every industry segment. Quality Assurance groups are faced with increasingly short delivery schedules, despite having significantly more work to do. Management does not have the quality visibility they need to make confident release readiness decisions. The penalty for late-cycle quality surprises and shipped defects is extremely high. Companies across all segments are investing in new test tools with an emphasis on more automation, better measurement and new runtime analytics technologies that can help them cycle faster, and improve the thoroughness of testing for their complex embedded software.

Product Complexity is Driving Significant New Testing Challenges

Software not only makes up a huge portion of the device features, most participants reported huge growth rates of software content. More than 54% of participants indicated that the software they are incorporating in their devices increases between 25% and 100% annually. 

Ninety six percent (96%) of respondents also validated that their devices are no longer ‘fixed function’ devices but instead are platforms for which they are required to periodically upgrade the software with new features and updates. Forty seven percent (47%) report that they upgrade the software in fielded devices on an “as needed” basis while 48% update on a regular basis ranging from less than 6 months to more than 18 month cycles.

Participants identified several key issues as roadblocks to on-time delivery of software. The top 5 operational factors ranked by respondents as the biggest challenges to their ability to deliver software on time were:

1. Changing requirements
2. Becoming familiar with new technology (processors, operating systems, etc)
3. Integrating and debugging devices
4. Complex system design
5. Time allotted for testing

 

The survey also asked about top challenges affecting their software testing processes. Participants identified a range of issues with the 4 most frequently cited as:

1. The amount and complexity of code they are required to test
2. Complexity of system architecture
3. Insufficient test automation
4. Moving to new operating systems

 

Insight into why these trends are challenging can be gained from participants’ responses to a question about which architectural trends are contributing to testing complexity.   Not surprisingly, changing operating systems was the top-rated architectural challenge followed closely by transitions to multi-core processors. 

Other often-cited challenges included the use of advanced user interfaces, advanced networking functions and the switch from single to multiple processors.  Although it was not ranked in the top five, the use of virtualization technologies, cited by 23.9% of respondents, is clearly a growing challenge in the software quality testing world.  

Over 80% of the respondents felt that testing on the embedded device itself is critical and they stated that they actually test each new build on the device.  But many respondents provide comments about how challenging embedded device testing is.

 In my next blogs I will cover additional findings from the survey relating to ever-tighter release cycles, lack of visiblity and confidence in quality, and new directions in tools and test automation.