Tools, Testing, & Virtual Systems

Diab Compiler Adding Support for new Infineon TriCore AURIX Microcontrollers

By Graham Morphew

Infineon Technologies recently announced their next generation family of 32-bit TriCore™ microcontrollers for automotive applications.  The new architecture is called AURIX™ (AUtomotive Realtime Integrated neXt generation architecture) and it supports multi-core architectures of up to three independent TriCore CPUs.  The feature set of this new AURIX family is a perfect match for powertrain applications, electric vehicles as well as steering, braking, airbag and advanced driver assistance systems.  The Diab Compiler is used in these same applications because it meets the extreme code optimization needs of these types of applications as developers try to maximize performance within the tight memory constraints of a microcontroller’s on-chip memories.  It is this alignment of target markets that brought Wind River and Infineon together to form a partnership to to ensure the Diab Compiler is highly optimized for TriCore, so that our joint customers can meet the demanding specifications of their OEM customers.

The Wind River Diab Compiler team has been working hard to support this exciting new TriCore architecture and its new features, including up to 100 percent additional performance compared to the TC1798.  AURIX support and new optimizations will be coming soon in the next Diab Compiler release, including an integrated assembler for the AURIX’s powerful Generic Timer Module (GTM).  Key customers and early adopters are already using a preview of this release to evaluate the AURIX capabilities for some big upcoming automotive projects. 

AURIX has been designed to meet the highest safety standards and Wind River has also been making investments to support safety standards.  Investments in ISO 26262 and Automotive SPICE will add to our extensive proven-in-use track record in automotive Electronic Control Units (ECUs) with many of the world’s major automotive manufacturers. 

It’s this combination of advanced optimization technology, collaboration with semiconductor partners, a focus on safety and Wind River’s world class service and support that make the Diab Compiler so successful in the automotive industry.

 

For additional information from Wind River, visit us on Facebook.

Integration and Testing the Integration

By Jakob Engblom

The US National Highway Traffic Safety Administration (NHTSA) recently released a deep report into last year's issue with "unintended acceleration" on certain Toyota cars. They actually employed a team from NASA who analyzed the throttle control software using a wide range of cutting-edge tools. Reading their report gives a good idea for how embedded control software is developed, and the challenges inherent in validating it.

The conclusion is that the software is not a likely cause of the problems. The real meat of the report (and its appendixes) is just how this conclusions is reached, and what it says about the tools that could be applied to develop embedded software with greater confidence about its correctness.

The NASA team used several types of static analysis, both code-semantic analysis looking for coding errors and static timing and stack-depth analysis. They also applied model checking, and built models of the environment that the control code interacted with. What they did not do was to simulate the actual code being used, since a simulation environment was not really available that could run the complete software setup.

The source code is complex and has many dependencies that make full-scale simulation outside its native hardware environment difficult. Even within the ... compilation and debug environment, the practice ... has been to only use a software simulation environment for “one-shot” unit testing, i.e., one input vector yielding one output vector. Any further testing beyond the unit level was done ... on hardware platforms with integrated software loads.

What this appears to be saying is that the developers of the code created unit-testing scaffolds for the control system and ran these on some small simulation setup. The simulated testing of the actual compiled code did not extend to the complete system, and most likely did not include running the actual target OS, dealing with interrupts, and other asynchronous events. This problem is elegantly solved with a virtual platform like Simics, where you replicate the hardware platform inside a host machine. You can then run the real code just like on the physical machine. No other solution really gives you that ability, running the whole software stack with all its bits and pieces integrated.

Using virtual platforms for such testing of functionality has several advantages, in particular for reproducing complex issues. Considering the effort spent on code validation, unit testing, and static analysis, I would expect the code to be pretty solid in units. However, when the code is integrated into the environment with an OS, with hardware and interrupt handlers, there is a whole new set of issues that appear.

Virtual platforms are great for integration testing. They are much more available than hardware. When rare issues appear, they are trivial to capture, reproduce, and communicate. With multiple communicating processor cores, you really want the debug power of a virtual platform. You also want to test what happens when the hardware starts to act up, with faults such as bad sensor readings or interrupts arriving quicker than expected.

In the end, all software has to be tested on the real hardware. There is no getting around that, you "fly what you test and you test what you fly". Using a virtual platform alongside environment simulations and static analysis tools will, however, make that testing much less painful. More bugs will have been removed prior to committing to hardware, and the overall code quality will be better. More than likely, the software development will also have taken less time and been less stressful for the developers.

 

Industry Investing in Better Device Runtime Visibility During Testing

By Paul Henderson

Here's the final installment in my series about our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog postings). If you'd like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.com and I will send it to you.

In this section of the survey we asked participants about what test tools they use today and where they are investing in test automation. Given the high cost of product failure, accelerating complexity and reduced schedules the industry is turning to more test automation in 2010 to help address these problems. The top investment are moving to new tools that can help test teams and their management better understand how well they are testing, better focus their efforts on the areas needing testing, and reduce cycle time through more automation.

Despite Existing Tools, More Automation Required

Participants were asked various questions about their present software quality testing environment and their plans and priorities for new investments to improve their testing operations in the coming year. Test automation remains top of mind for respondents with 68% of those who knew saying they plan to automate more of their test processes this year.

As for the tools currently in use in their testing processes today, respondents not surprisingly reported that they use a wide variety of systems and techniques. Results did show that manual testing, conducted by 81.7% of respondents, remains a substantial part of the device test processes at most organizations today. Other well-represented categories included defect tracking systems, static or dynamic code analysis, and performance and timing measurement. There was a strong showing by simulators, which were reported as being used by over 50% of survey participants. This category includes both device simulation environments as well as traffic, load, and other external simulators that drive device behavior during the test process.

The Top 10 List for 2010

Regarding their plans for new investments, there is a clear need for more automated testing among respondents. The top 10 rank order of tools for planned investment reported was;

1. Automated test equipment
2. Code and test coverage analysis
3. Simulators
4. Runtime diagnostics tools
5. Static or dynamic code analysis
6. Performance and timing measurement
7. Test execution engines
8. Script authoring tools
9. Test script capture/replay tools
10. Fault injection & runtime behavior monitoring

Runtime Visibility of Device Under Test Needed

Notably, in the top 10, at least 5 of these tools are focused on helping testers understand more effectively what is happening in their code at runtime. Respondents indicated that purchasing tools for these runtime analytics is a higher priority that obtaining more traditional tools for manual testing, defect tracking or lifecycle quality management.

The survey also asked an open-ended question about other trends or investments that should be considered. A significant number of respondents expressed the need for better host-based testing and device simulation so to streamline testing early in the process before hardware available and to reduce the cost of testing assets.

“The biggest problem we have is usually not able to test for a new ASIC before the ASIC is taped out…. It would be nice if there are tools that could be used to develop production code alongside the simulator that we can use for testing before an ASIC arrive, and run in the high level system.”

“There is a strong need for host based "device dependent code" testing suite or methodology.”

“Need better simulations of embedded devices as real hardware can be expensive.”

Conclusions

I hope this series was interesting for you. We found it ratified some of our anecdotal evidence while also surfacing additional areas where we will need to focus our investments and services. It does show that industry trends and market forces have pushed the requirements for software quality testing well beyond the capabilities of the tools and processes currently in use. The problem is getting worse with more code crammed into devices, architectural complexity off the charts, and squeezed development and test cycles. This is the new normal.

This disparity is causing serious operational problems for embedded products companies, negatively impacting not only their brands but also their bottom-line results. Ignoring the problem is no longer an option. Leaders are seeking new test automation tools that can help them with the unique problems of embedded device testing while helping them deliver on-time, on-budget, and on-quality.

One respondent summarized it best:

“We spend too little time testing and too much time hoping.”

It's time to take action.

The High Cost of Poor Quality – Brand, Market, Budget

By Paul Henderson

I'm continuing my series on our embedded device industry software testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog posting).  If you'd like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In this section of the survey we asked participants about how they measure the high cost of poor quality. Respondents told us that the true cost of poor quality is much higher than program budget. The majority of respondents showed that the true cost of poor quality is measure by damage to company brand and lost revenue due to missed market windows.

Despite these risks, participants are suffering from repeated schedule slips due to late-cycle quality surprises, though a significant number make the decision to ship defective product anyway an fix it later. When they do ship defects, companies are finding the cause is not due to standard features that they tested for, but instead from unanticipated uses of the product, which naturally were not tested.

Substantial Budget Devoted to Testing, Yet Late Programs

Participants were asked about how much of their total project total resources are spent on testing. More than 77% of the respondents reported that testing activities typically consume over 20% of their total project budgets.  Nearly 21% reported testing costs totaling over 40% or their total project budgets.

Participants were asked what percentage of their overall test budget is allocated for testing maintenance releases and software upgrades.  A large majority (74.8%) said that they dedicate more than 10% to this lower profile but still evidently important testing activity.

While 20-40% of project budget for testing is significant, the responses subsequent questions point out that this hard cost of the testing department resources is only a small part of the cost of quality – or rather the cost of poor quality.

Participants were asked about how poor quality was effecting their development cycle and how they responded to and measured this effect. Seventy two percent (72%) of respondents reported that over the last year their projects had been disrupted due to late-cycle discovery of critical software quality problems. The frequency of these disruptions was surprisingly high, with most respondents that had problems reporting that this happened more than once per project in the last year.

As to the question of how they handled these issues, 57.6% of those that experienced disruptions said that they delayed their products' ship dates in order to fix the problems and retest the products.

Unanticipated Usage the Most Common Cause of Defects

When they did ship products with software defects that were discovered later by customers, it was most often the unusual and unanticipated use that exposed the problems:

1)            Products used in unanticipated ways (58.1%)

2)            Untested edge conditions (48.9%)  

When asked how their companies measured the cost to the company when products were not delivered at planned quality, the top three measurements were significant business drivers:

1)            Higher than expected program costs

2)            Damage to brand due to shipped defects

3)            Missed market window for the product

Next time I will review respondents' feedback on what test tools they are using today and where they are investing for new test automation.

Inadequate Management Visibility into Quality is Eroding Confidence

By Paul Henderson

Here's the next installment in my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog postings).  If you'd like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In this section of the survey we asked participants about how they measure software quality today, the metrics most often cited by survey respondents were reactive in nature such as tracking customer-reported failures and open defects rather than metrics that can help them prevent defects.

Survey participants clearly indicated that they have a strong desire to be more proactive, but their responses also reveal a significant disparity between their intentions and their actual testing. The survey uncovered a significant disparity between testing goals and reality (as measure by test coverage analysis), where a minority of respondents actually have access to the information they need to assess quality. Respondents further report that often the software quality information they do receive is later proved to be wrong, leaving them to make product readiness decisions on information that turned out to be inaccurate.

Respondents seem to know that you can't fix what you can't measure.  They also know that the answer lies in more thorough testing done more often. But the common metrics in use, especially the customer-reported problems, appear to count how many “horses have already left the barn.”  The disparity between testing goals and actual results, the management blind spots regarding software quality levels, and the decision making based on bad information are all contributing to a crisis in confidence in software quality in the embedded industry. 

Metrics Not Providing Management Confidence

When asked if their exiting tools give management sufficient software quality information to make release readiness decisions with confidence, 75% of respondents said that they were at best only “somewhat confident.”  With 19.1% of respondents, their uncomfortable answer was flatly “No, not confident.” 

It seems that a high percentage of participants have learned to be skeptical about the software quality information upon which they must make decisions.  Of those that responded that they knew one way or the other, 57% said that they have made release readiness decisions based on inaccurate or insufficient software quality information. 

One telling group of data points is the responses to a question about which metrics respondents use to measure the software quality in their products.  By far, the two most frequent answers were:

- The tracking of the number of open defects

- The numbers of defects reported by customers

Notably, both of these measures are reactive; they essentially gauge the number of defects already introduced into their products.  The other, less-used metrics include test coverage metrics, numbers of tests executed, etc. were more proactive measures that can be controlled and managed by the companies. 

Inadequate Test Coverage Information

Their lack of confidence and reactive management approach can also be linked to the low use of test coverage and code coverage tools. These tools provide insight into whether software was actually tested (or not) and how thorough test suites are. They should be used in both unit testing at the developers desktop as well as in the functional system testing in the QA lab.  

Overall only a minority of participants much use of code coverage and test coverage tools. This is evidenced in respondents' answers to the question as to whether they track which parts of their code is actually exercised by their tests.  Over 60% said they either don't track test coverage at all or only do so during development unit testing. Only about 32% said they measure and track what code is exercised in system tests of the fully integrated device. 

Perhaps the most telling response in this section of the survey was in response to questions about test coverage goals and actual test coverage results with their current projects.  Nearly 65% of participants reported having goals for test coverage of their code bases of over 70%.  But only 35% of respondents believe they are at greater than 70% code coverage with the project they are currently working on. 

Next time I will cover respondents' feedback on the high cost of poor quality and how they measure this.

Compressed Schedules Driving Shorter Testing & Defect Resolution Requirements

By Paul Henderson

Today I'm continuing my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog posting).  If you'd like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In part 2 of the survey we asked about schedule compression and what affect that was having on the device testing cycle. A majority of survey participants reported that market conditions have forced them to shorten their development schedules by as much as 18 months.

Many companies are turning to iterative methods to improve feedback within their product cycles. Some are using Agile techniques. However, most respondents are still heavily using manual techniques for system testing and especially for more complex testing of error conditions and performance validation. Defect isolation and repair cycles are typically measured in days per bug. Together these bottlenecks are increasing the need for new approaches to test automation.

Shorter Release and Build Cycles

A majority (64%) of respondents indicated that market conditions have forced them to shorten their development schedules. In terms of the length of their development cycles, a majority of respondents (57%) reported current cycle times of between six and eighteen months.  Of those, 44% reported having to cut their standard development cycles by six to twelve months. 

The survey also asked about the adoption of new Agile development methodologies. Of the respondents who knew, the majority of the participants (64%) reported that they have not adopted Agile methods.  However, 57% reported that they are running bi-weekly build cycles or less with 16% of these actually doing nightly builds.

So despite most teams not considering themselves formally Agile, most teams are using iterative processes.  Surprisingly, 18% indicated that they don't operate on a formal, fixed schedule for their build cycles. 

Difficult to Evaluate Device Behavior at Runtime

A high majority of participants (over 81%) reporting that their teams test each new software build on the embedded device itself.  This shows that respondents recognize the importance of testing software on the integrated device, rather than just conducting unit tests. Surprisingly, 81% of respondents cited manual testing tools as part of their testing process today, so presumably, manual testing within this iterative environment is becoming a challeng.

Regarding the time it takes their teams to isolate and repair defects found during system integration, more than 62% reported that it takes them a full day or longer.  Over 25% said it takes them more than 2 days. This is clearly a bottleneck to the process and an expense that needs to be minimized.

The survey also asked how users were testing failure modes, error conditions and performance. Fifty seven percent (57%) of respondents said they were using manual methods to verify failure modes and 8.3% said they did not verify these conditions at all. Forty one percent (41%) of those who knew said their tests were not able to measure performance of their devices at runtime.

Numerous participants also commented on the difficulty of understand precisely what is happening within their device under test at runtime.

 “Multi-core products are creating performance challenges for the architects and more corner cases which are difficult to test.”

“Software must be tested thoroughly  under all possible operating conditions.”

“(It's) difficult to "see inside" linked programs.”

In a somewhat surprising finding about their test environments, more than half of all participants (56.9%) reported that their software development and testing processes are not governed by a safety, security or other certification standards. Of the participants who responded in the affirmative, adherence was fragmented across many different standards.

Next time I'll review respondents' feedback about quality measurement and how lack of quality visibility is eroding management confidence.