Tools, Testing, & Virtual Systems

The Role of Tools in Improving Embedded Software Security / Part 3: Mapping the Tools to Activities

By Bill Graham

In the previous posts in this series, Part 1: Automation is the Key and Part 2: Security Improvement and the Software Development Lifecycle, I talked about the connection between the typical embedded device development process and the 5+1 improvement framework for embedded security.  Figure 1 is an illustration of this connection (and discussed in more detail in a previous post).

Figure 1: Mapping the security improvement framework and the device software lifecycle

In this post, I’m going to map out the tools that can help improve security and where they fit in the development lifecycle. Tools have different parts to play at each phase of the lifecycle, in some cases they are used throughout, and in others, their role is key in one phase and less so in others. In terms of security improvement, many of the tools and processes mentioned are not unique. In fact, the ideal situation is where security is considered a first class citizen in the priorities of a project.  In this respect, tools for requirements, modeling and change management are meant to be used in the same fashion and don’t represent unique capabilities required in the 5+1 improvement framework. Again, the better the requirements and risk management, the better the security solution (assuming that security is given the right priority.)

Figure 2 illustrates roughly where each class of tool plays a role in the embedded device lifecycle. In each case, the unique capabilities required for improving security are called out but these don’t preclude the tools used in other areas.

Figure 2: The mapping of major tools categories in the software lifecycle

The Tools

System Simulation

High fidelity, system-wide, comprehensive simulation is an invaluable software development too. State of the art simulators are more than just processor emulators, today they provide full networked system, board, peripheral and processor simulation. The key benefit of system simulation is pulling in your integration and test cycles much early than previously possible. Secure design and system vulnerability can be tested earlier and more often. In addition, simulators offer unique features that real hardware can’t do and the ability to arbitrarily inject faults at any part of the system. An example of a system simulation tool is Wind River’s Simics product.

Integrated Development Environment

The Integrated Development Environment (IDE) is the developer’s desktop. Most developers’ tools integrate into the IDE in order to be visible and usable from this desktop. All of the tools discussed here are integrated with the IDE. IDE’s also provide many tools that are useful in improving security. For example, memory (runtime) analysis tools are critical for finding buffer overflow errors that are the root cause of many security vulnerabilities. System behavior and analysis tools are critical for tracing complex defects – essential for detecting and debugging security issues. An example of an IDE is Wind River Workbench.

Static Analysis

Static analysis tools analyze source (and sometimes binary) code to find defects, security vulnerabilities and other programming issues. These tools can work on existing code bases and on new source as it is written by the developer. The key advantage of using static analysis is that security vulnerabilities are found as soon as the code is written – the cheapest and least riskiest time to find and fix them. In addition, vulnerabilities can be found in “hidden” parts of the code – code that is not executed during traditional testing. Static analysis tools can be run individually within each developer’s IDE and singularly on the project build providing a distributed and centralized analysis capability. Examples of static analysis tools include Coverity’s Security Advisor, Klocwork’s Insight, Programming Research’s QA-C/C++,  and Grammatech’s Codesonar products.

Build and Continuous Integration Tools

Modern iterative and agile development methods require more frequent build and test cycles. Proper tools support for high frequency build and test in order for the productivity and quality gains of new methods to pay off. These tools provide the same benefits for security assessment and testing – security vulnerabilities can be found, fixed and test in a short cycle. As development methods evolve to include secure development, the return on investment for state of the art build and integration tools increases. An example in this area of tools would Electric Cloud’s ElectricAccelerator and ElectricCommander products.

Test Automation and Management Tools

Increased testing effort of developing an embedded device is the largest factor in its schedule and cost impact of improving security. Typically, testing for security vulnerabilities requires additional testing that differs from functional testing in that is attempting to break the system. Most functional testing is looking at most common usage patterns of the device. Security testing is often looking at edge cases, erroneous input, and misuse. In addition, test automation tools are required to improve test efficiency and effectiveness. Security testing will add risk, cost and effort to the testing effort for an embedded device. However, innovation in testing tools has made this impact significantly smaller. An example tool in this category would be Wind River Test Management.

Robustness and Fuzz Testing Tools

Due to the nature of security testing – often requiring out of range and erroneous data - new tools are required. Fuzz testing tools are designed to send malformed data to the device to induce errors. These tools use knowledge of various communication protocols (including specialized embedded control protocols) to create incorrectly formed packets, bad data and various extreme cases. Often this data is enough to induce runtime errors in the device – exposing potential vulnerabilities. These vulnerabilities are previously unknown (so called zero-day vulnerabilities) are especially critical to find and fix. Fuzz testing can be applied well before the device is in final form. Leveraging system simulation, fuzz testing can used in a simulated environment as effectively as it is in the lab.  An example of a fuzz testing product is the security content package for Wind River Test Management (with technology licenses from Codenomicon).

Risk and Threat Management Tools

It’s likely that few embedded development projects are currently using automated risk and threat management tools. These tools are often used in highly safety critical projects where risk management processes are stringent and detailed traceability is required. However, these tools can prove valuable to any team that needs to maintain detailed information about their risk and security threat management plan. Automating the management security threats from early threat assessments all the way through the device lifecycle is important as a way to organize and retain the details, record the threat analysis, drive security test and fix priority, and end-to-end traceability. In addition, as security requirements start to become part of various industrial standards, threat management tools become increasingly essential. An example of this would be RSA Archer Threat Management.

Application Lifecycle Management Tools

Application Lifecycle Management (ALM) tools cover a wide variety of tools that are aimed at improving software development productivity and quality while managing and limiting schedule and budget risk. Requirements and change management, modeling and analysis tools, test management, product portfolio management and more. These tools are important for productivity, quality, compliance and security. Although not specific to security, incorporating security requirement management, secure design and security testing into the ALM scope is critical. In other words, the tools manage the security design lifecycle as they do the mainstream software development lifecycle. Examples of the ALM tools would be the IBM Rational suite of tools.

Summary

Tools are key to developer productivity and critical in improving security in embedded devices. In this age of increased scrutiny of security (and certainly other quality characteristics) and increased pressure for more features and tight schedules and budgets, the need for tools is greater now then ever. In this post I’ve mapped the tools to the embedded device development lifecycle and to the 5+1 step security improvement framework and what role each class of tool plays in the security improvement framework. I’d like to hear if there are other classes I’ve missed in the list and what tools you’ve found to be the most useful for improving your product security.

 

For additional information from Wind River, visit us on Facebook.

Using Static Analysis to Improve Product Quality, Earlier and Cheaper

By Bill Graham

Fixing bugs is expensive. Fixing bugs is more expensive the later you leave them, in fact, its been shown to cost a magnitude higher with each major phase of development. The famous defect cost chart from Capers Jones shows the cost of a bug going from $25 at the coding phase to $16,000 in development.  Not only that, but most of the bugs are introduced into the system at this stage yet very little of them are found. Logic says we should fix them earlier to save money.

  (Source: Applied Software Measurement, Capers Jones, 1995)

 

Static analysis tools such as those offered by our partner Coverity Inc.  provide an easy, non-obtrusive way to detect defects in the source code – as it’s being coded. These tools can find all sorts of defects even before the code is submitted to version control or compiled into your integration builds. Significant and dangerous errors such as buffer and integer overflows, format string vulnerabilities and poor coding techniques can be found in your source code.  Static analysis tools are integrated into your build environment as part of your nightly integration builds, for example.  What is also extremely powerful is the integration of these tools into our Wind River Workbench. Developers can quickly and easily check for bugs in the code in the integrated development environment. A quick click and the current code file you are working on is checked for bugs.  Developers are understandably skeptical but success rate for static analysis tools is high and the technology has improved to where false positives (detected bugs that turn out to not be errors) are very low, typically less than 5%.

Using tools like Coverity’s Static Analysis in addition to automated test tools (Wind River Test Management, IPL’s Cantata++, for example) are great ways to improve the graph we see above.  Imagine the return on your investment – every bug you find and fix at the developer’s desktop saves you $1000 if you are lucky to catch it during integration or even $16,000 if it makes it into customer hands.

 

For additional information from Wind River, visit http://www.facebook.com/WindRiverSystems.  

It's a Beautiful Day

By Emeka Nwafor

I have been totally absorbed by the media coverage of the incredible rescue of the thirty-three Chilean miners that were trapped about a kilometer below the earth's surface for months. The images of jubilation as "Super Mario" and the other miners exited the pod will be imprinted in our minds forever. A real Feel Good Story.

Many thanks to the scientists, engineers, and rescue specialists from across the planet for playing their part in bringing up the 33 miners and 4 rescue workers and for making this A Beautiful Day.

Between reports of the rescue of the individual Chilean miners, CNN's report on the "Best Jobs in America" caught my attention, my curiousity, and briefly diverted my focus from the rescue (and my day job). Of the top-100 jobs, I was pleasantly surprised to see that "Software Architect" ranked at the top - #1 in America. "Product Management Director" and "Software Engineering/Development Director" were in the top-20 - this was a pleasant but credible surprise. "Telecommunications Network Engineer", "Test Software Development Engineer", "R&D Manager", and "Systems Engineer" were all in the top-50. Very interesting considering that these rankings are based on criteria like compensation, job growth, and quality of life.

I proceeded to dive deeper into some of the detailed job descriptions - the "what they do", "what to like", and "what not to like" details. Just looking at the top-5, there were a few things that alarmed me. Interestingly, "Management Consultant" comes in at #3 - the third best job in America. Hmmm... I have respect for what Management Consultants do - I even know a few of them. But the third best job in America?!

Clearly, I have a bias. When reading the Management Consultant job details, to me it read like a niche role when compared side-by-side with the likes of the "Software Architect" description or the "Systems Engineer" description (#49). To me it feels even more "niche-like" when I think of this in the context of the innovative embedded devices and applications and "smart products" that embedded developers across the planet are developing to make our lives better when interacting with information on the web, interacting with our cars, interacting with our medical specialists, or making life safer when travelling in planes and trains.

So, what could the "Top Jobs in America" list look like in 5 years?

Like "Management Consultant", will we see new job descriptions that better characterize what is going on in the growing embedded software development space? Will jobs like "Human Device Interaction Engineer", "Device Performance Therapist", "Software Restoration Engineer", or "Software Archaeologist" make it into the top-25 jobs in America list? In 5 years, I wonder if these jobs would crack the top-10 "Benefit to Society" jobs list and join  the ranks of the "Anesthesiologists" and "Rehabilitation Services Directors" that are there today.

I think so.

There needs to be more celebration about the great things that we are doing with embedded software. I think we need to tell more stories about how our innovation matters to society. Here's a little story to illustrate why I think this might happen...

During my last visit to the doctor's office, we wrapped up the visit with our usual discussion around the latest and hottest hi-tech gadgets. The discussion this time around expectedly was focused on the iPad. My doctor shared a story about how the iPad had made one of his patients lives better. This patient had suffered a stroke that limited his use of one of his arms and had impacted his vision. Before the iPad, the patient routinely complained about difficulty in using his computer because of the effects of the stroke - difficulty with the mouse and keyboard, difficulty in reading the screen, and headaches. Life for this patient got better with the iPad. Browsing the web was easier using one hand and the multitouch interface. The ease of magnifying text and images when reading resulted in fewer headaches.

For me, this too is a Feel Good Story.

I am sure that there are many more like them for almost every embedded device application that's out there. I always look forward to hearing about the cool applications Wind River's customers are designing and the stories about how these embedded devices help to make A Beautiful Day in our lives.

Intuitive & Efficient Debug of Multi-core Systems

By Brian Finkel

I reflect on a recent visit with my 84-year old Aunt Honey. As you might expect, she is not an avid cell phone user. She wanted help entering contacts into her phone. She has a basic phone so we had to enter text using the “9-up” keypad. Her objective is to make phone calls by selecting someone’s name instead of typing the telephone number.   

The event had me thinking about the way we used to make telephone calls – using an address book or a phone list and then typing in the number. In my youth, things weren’t so bad – everyone lived in the same town – with the same area code and only a few prefixes. For each friend you only had to remember the 4 digits and perhaps which prefix to use. Fast forward a few years and with the advent of cell phones, IP telephony, new area codes, and geographically-dispersed friends and family – you now need to remember 10 digits. 10 digits is beyond most people’s working memory (which is typically 7 digits). Address books and phone lists were required. 

[International Readers: The North America phone number is 10 digits. 3-digit Area Code + 3-digit Prefix + 4 digit Number]

Enter the cell phone. And the ability to program phone numbers and contact names into the phone’s memory. That enabled me to Dial-by-Name. Dial-by-Name is intuitive -- my Aunt Honey is a person not a 10-digit number. Dial-by-Name is efficient – no more rummaging through the desk drawer for the phone list, no more typing a wrong number, no need to memorize 10 digits. That saves me time! Dial-by-Name is intuitive and efficient compared to the old way.

Before I compare Dial-by-Name to what we do at Wind River, I’d like to take a minute to give kudos to the mobile phone developers for creating this seemingly simple user experience that we now take for granted.

Look at your cell phone. Under the covers things aren’t so simple. Your phone is a complex wonder of miniaturization enabled by numerous technology advances: low-power CPUs and communications chips, small-form factor batteries, cost-effective LCD displays, and thousands of lines of code ranging from device firmware to applications. There are dozens of components that had to be developed, integrated, and finely tuned:  the phone dialer application, the contact list application, the CPU, the baseband communications chip, the keypad, speaker, LCD display, headphones, Bluetooth, and more – all working together in harmony. It took 100’s of engineers, project managers, and quality assurance professionals (and a little luck) to deliver Dial-by-Name on your phone. So congratulations to the mobile phone developers out there for making our lives simpler!

So why does this all matter to you, to me, and to Wind River?

I assume if you are reading this blog you are probably closely tied to the embedded developer community. No doubt, you and your colleagues are using or considering multi-core SoCs.  These devices are becoming increasingly complex – integrated peripherals, co-processors, 3 levels of cache, and so-on. And to realize the benefits of these devices, you are considering multiple operating systems running on the same CPU potentially in a SMP (Symmetric Multi-Processing) or virtualized system configuration. Your world is complicated.  As an example, if you are designing network infrastructure gear, you are probably consolidating data plane and control plane functions onto a single board, blade, or SoC running a combination of Linux, an RTOS, and/or a low-level executive, partitioned across multiple cores or processors.   

As an embedded developer, how do you take control of a complex system like this? How do you debug and resolve issues that are prone to arise in this complex multi-core system environment?

As tools developers at Wind River, we strive to deliver solutions that are intuitive and efficient. 

Here are some of the challenges we think about on a daily basis:

• How do you simultaneously visualize and interact with the resources for 8 cores in a single IDE? 
• How do you provide visibility to multiple operating systems each running its own set of processes, threads, and applications?   
• How do you debug guest operating systems running on a hypervisor?
• How do you provide a flexible GUI so that different types of users – hardware engineers, device driver developers, application developers, system integrators – can view and control system elements important to the task at hand?
• What is the best way to represent an embedded system on a 1280 x 1024 display using graphical elements such as colors, fonts, tabular views, scroll bars, icons, expandable lists, progress indicators, windows, and more?   

In order to deliver innovation, we embrace a customer-oriented approach in developing our tools.  Through our local technical support organizations, we reach out to you to understand the problems you are trying to solve. We implement your usage models and desired “workflows” within our development and quality assurance processes. This approach is yielding innovation. 

A recent example is the SMP Debugging capability delivered with our On-Chip Debugging (JTAG) solution. When using Workbench 3.2 and our Wind River ICE 2 JTAG debugger, you can logically represent an OS running on multiple SoC cores as a single system. This enables an intuitive debug experience -- operations such as breakpoint management, kernel visibility, and run-control are managed at the system-level. Similar to how Dial-by-Name abstracts the 10-digit phone number, we’ve abstracted the details on each of the cores. It’s intuitive because it’s the way you think about your system. 

We look forward to hearing your feedback on how we can make your life easier through an intuitive and efficient development tools experience.     

Happy developing!

Brian Finkel is a Senior Product Line Manager for Wind River Development Tools and On-Chip Debugging Solutions. One of Brian’s current initiatives is to deliver a world-class debug experience for developers implementing complex multi-core and multi-OS systems.   Brian has 15-years experience in product management, technical marketing, program management, and business development roles at Wind River, Marvell Semiconductor, Intel, and IBM. Brian holds a Bachelor of Science Degree in Electrical Engineering from Rutgers University.

A Few Words About the Future

By Amit Nandi

I recently started reading a book called “I Live in the Future & Here’s How It Works” by Nick Bilton.  I got interested in the book after reading an excerpt online at the New York Times, where Bilton is the lead writer for their technology blog. This is where my life started imitating art.

While I was waiting for the e-book to download, I read on to the second half of the excerpt. Bilton describes a nifty rule he calls “1, 2, 10”; numbers that describe the distance between your eyes, the screen of a mobile device, computer screen and television screen, respectively.  He asserts that content will eventually follow us from screen to screen and place to place. While this exact future hasn’t happened yet, a reasonable facsimile of it was taking place in a race between the downloading of his book both on my computer and my iPod touch.  Having the content accessible on a large monitor during the day and at bedtime is really handy when you’re writing to a deadline.

While Bilton covers a wide expanse of topics ranging from the evolution of content delivery to relationships in social networks, there’s a conversation it can start about how we can enable the user experience in a world with interacting embedded systems. 

We’re in the infancy of understanding truly user-centric design as it relates to multiple intelligent devices interacting with each other, providing a seamless experience, as our location, focus and interest changes. Technologies like Bluetooth have been out for a decade; consumer GPS longer than that, and both in cars for years - yet I still can’t find my keys in the morning.  My set top box can’t remember where I stopped watching last night’s movie when I power it back on, let alone downscaling it for me to watch it on a trip.  It would be delightful to listen to the new Arcade Fire album during my evening commute, picking up where I left off at the office.

Bilton himself explored similar scenarios to the above with a colleague using a cellphone, embedding an RFID chip inside, attaching an RFID reader to a computer and writing some code to track articles he was reading on NYTimes.com.  Their proof of concept passed articles automatically back and forth.  A person who prototypes the future in addition to writing about it is a rare beast.

If journalists can do that, surely those of us who are technology insiders can dream and design as well. We are in the age of ubiquitous computing; we might as well get cracking at providing the technology to enable the future that Nick Bilton writes about.

Amit is the Product Line Manager for Wind River Workbench development tools. He has over 16 years of Software Development and Product Management experience and is currently working to accelerate the time to market for VxWorks and Wind River Linux developers. Amit holds a bachelor's degree in aerospace engineering from Carleton University and a master's degree in business administration from Queen's University.

Wind River and IBM Attack Software Quality

By Paul Henderson

As I've mentioned before, we've been working with IBM Rational for some time around quality management automation. Both companies see the skyrocketing software content and architectural complexity in the embedded device market as creating a tipping point where companies will not be able to continue with business as usual.

Product development teams will need to take a more managed and automated approach to quality that spans across the lifecycle and access into the devices under test. This is particularly true in markets that require strict adherance to standards and compliance regulations.

We put together a joint whitepaper on this subject downloadable from here. And we are also having a joint web seminar next week on Tuesday Aug 31 at 2pm EDT. You can register for this event here.

In this webcast you will hear how IBM Rational and Wind River are teaming to deliver a lifecycle approach to quality management by linking our respective products, IBM Rational Quality Manager and Wind River Test Management.  We will discuss seven key strategies for optimizing development – and how you can eliminate the friction in your delivery which can lead to missed cost, delivery and quality targets.

Key Highlights:

• Best practices to help facilitate effective collaboration around quality
• How a requirements-driven approach can help you deliver the right solution
• Ways to support automated and integrated testing via the ecosystem
• Processes to accelerate defect detection and resolution for efficient delivery

I will be joined in the Webcast by Moshe Cohen, Sr. Director of Products, IBM Rational, and Karla Ducharme, Market Manager for Aerospace & Defense, IBM Rational. Please join us if you are available or watch the replay when it is available at http://www.windriver.com/seminars/web-seminars/ after the event.

 

Compressed Schedules Driving Shorter Testing & Defect Resolution Requirements

By Paul Henderson

Today I'm continuing my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog posting).  If you'd like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In part 2 of the survey we asked about schedule compression and what affect that was having on the device testing cycle. A majority of survey participants reported that market conditions have forced them to shorten their development schedules by as much as 18 months.

Many companies are turning to iterative methods to improve feedback within their product cycles. Some are using Agile techniques. However, most respondents are still heavily using manual techniques for system testing and especially for more complex testing of error conditions and performance validation. Defect isolation and repair cycles are typically measured in days per bug. Together these bottlenecks are increasing the need for new approaches to test automation.

Shorter Release and Build Cycles

A majority (64%) of respondents indicated that market conditions have forced them to shorten their development schedules. In terms of the length of their development cycles, a majority of respondents (57%) reported current cycle times of between six and eighteen months.  Of those, 44% reported having to cut their standard development cycles by six to twelve months. 

The survey also asked about the adoption of new Agile development methodologies. Of the respondents who knew, the majority of the participants (64%) reported that they have not adopted Agile methods.  However, 57% reported that they are running bi-weekly build cycles or less with 16% of these actually doing nightly builds.

So despite most teams not considering themselves formally Agile, most teams are using iterative processes.  Surprisingly, 18% indicated that they don't operate on a formal, fixed schedule for their build cycles. 

Difficult to Evaluate Device Behavior at Runtime

A high majority of participants (over 81%) reporting that their teams test each new software build on the embedded device itself.  This shows that respondents recognize the importance of testing software on the integrated device, rather than just conducting unit tests. Surprisingly, 81% of respondents cited manual testing tools as part of their testing process today, so presumably, manual testing within this iterative environment is becoming a challeng.

Regarding the time it takes their teams to isolate and repair defects found during system integration, more than 62% reported that it takes them a full day or longer.  Over 25% said it takes them more than 2 days. This is clearly a bottleneck to the process and an expense that needs to be minimized.

The survey also asked how users were testing failure modes, error conditions and performance. Fifty seven percent (57%) of respondents said they were using manual methods to verify failure modes and 8.3% said they did not verify these conditions at all. Forty one percent (41%) of those who knew said their tests were not able to measure performance of their devices at runtime.

Numerous participants also commented on the difficulty of understand precisely what is happening within their device under test at runtime.

 “Multi-core products are creating performance challenges for the architects and more corner cases which are difficult to test.”

“Software must be tested thoroughly  under all possible operating conditions.”

“(It's) difficult to "see inside" linked programs.”

In a somewhat surprising finding about their test environments, more than half of all participants (56.9%) reported that their software development and testing processes are not governed by a safety, security or other certification standards. Of the participants who responded in the affirmative, adherence was fragmented across many different standards.

Next time I'll review respondents' feedback about quality measurement and how lack of quality visibility is eroding management confidence.

 

A Crisis of Complexity – Industry Report on Growing Challenges in Embedded Testing

By Paul Henderson

I've been talking a lot with embedded device companies around the world over the last few years and I am hearing growing concerns about software testing. I've mentioned several of these concerns in previous blogs. I wanted to get more quantifiable data and get some feedback that could help us shape our products and services to help. So I decided to run a survey to gather important data from our community.

The focus of this survey was to gain a detailed snapshot of how executives, development managers, QA and test team leaders and other involved staff currently view the embedded device software quality test landscape. Recent changes, new challenges and strategies for managing them were of particular interest. So I fielded a four-part survey to individuals who work for embedded products companies. In total, nearly 35,000 individuals in North America were invited to participate in the survey via emails. Participation was also promoted on the Wind River Website and on executive blog. The survey was conducted between April 2nd and May 13th, 2010 using an online survey tool.

Huge Response
Amazingly, I received nearly 900 responses (889 total) from folks in the industry. While 47.8% of respondents identified themselves as software developers, another 46.4% of participants have roles in management and test (including executives, directors, managers, project leads, quality officers, etc). The sizes of companies from which responses were received were diverse and reflective of the industry in general. Respondents split with about 50% coming from companies with 1500 or less employees and 50% coming from companies with larger than 1500 people. About 41% work at companies with more than 5000 employees. A total of 54.1% of participants identified themselves as Wind River customers; over 40% of respondents said they were not customers

In this blog and 4 subsequent postings I will review some of the highlights of the summary report I created from the aggregate results. If you'd like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

Key Takeaway

Overall the survey shows that embedded device market dynamics, including skyrocketing software content and new architectural complexity, are challenging development and test teams in every industry segment. Quality Assurance groups are faced with increasingly short delivery schedules, despite having significantly more work to do. Management does not have the quality visibility they need to make confident release readiness decisions. The penalty for late-cycle quality surprises and shipped defects is extremely high. Companies across all segments are investing in new test tools with an emphasis on more automation, better measurement and new runtime analytics technologies that can help them cycle faster, and improve the thoroughness of testing for their complex embedded software.

Product Complexity is Driving Significant New Testing Challenges

Software not only makes up a huge portion of the device features, most participants reported huge growth rates of software content. More than 54% of participants indicated that the software they are incorporating in their devices increases between 25% and 100% annually.

Ninety six percent (96%) of respondents also validated that their devices are no longer ‘fixed function' devices but instead are platforms for which they are required to periodically upgrade the software with new features and updates. Forty seven percent (47%) report that they upgrade the software in fielded devices on an “as needed” basis while 48% update on a regular basis ranging from less than 6 months to more than 18 month cycles.

Participants identified several key issues as roadblocks to on-time delivery of software. The top 5 operational factors ranked by respondents as the biggest challenges to their ability to deliver software on time were:

1. Changing requirements
2. Becoming familiar with new technology (processors, operating systems, etc)
3. Integrating and debugging devices
4. Complex system design
5. Time allotted for testing

The survey also asked about top challenges affecting their software testing processes. Participants identified a range of issues with the 4 most frequently cited as:

1. The amount and complexity of code they are required to test
2. Complexity of system architecture
3. Insufficient test automation
4. Moving to new operating systems

Insight into why these trends are challenging can be gained from participants' responses to a question about which architectural trends are contributing to testing complexity. Not surprisingly, changing operating systems was the top-rated architectural challenge followed closely by transitions to multi-core processors.

Other often-cited challenges included the use of advanced user interfaces, advanced networking functions and the switch from single to multiple processors. Although it was not ranked in the top five, the use of virtualization technologies, cited by 23.9% of respondents, is clearly a growing challenge in the software quality testing world.

Over 80% of the respondents felt that testing on the embedded device itself is critical and they stated that they actually test each new build on the device. But many respondents provide comments about how challenging embedded device testing is.

In my next blogs I will cover additional findings from the survey relating to ever-tighter release cycles, lack of visiblity and confidence in quality, and new directions in tools and test automation.

What's New in Wind River Test Management 3.3?

By Paul Henderson

Today we announced the latest version of Wind River Test Management, Release 3.3, our test automation system for monitoring, executing and managing embedded device software testing. Wind River Test Management lets teams optimally execute complex tests while dynamically gathering information from the production software under test as it is running, without requiring special pre-instrumented software builds. This approach allows teams to adopt new white-box test techniques that give testers visibility into the operation of the device and help them determine the thoroughness of the tests, quickly identify defects and performance bottlenecks, and focus efforts on sections of software that are most in need of testing.

Release 3.3 is a major new release that adds a number of significant new features. You can learn more and download several new whitepapers from www.windriver.com/products/test_management.

Here's a quick summary of the key new features

A 10X improvement in scalability enables runtime test coverage for very large apps</p>

Wind River Test Management's dynamic instrumentation technology allows 'white box' access to the device under test. This instrumentation is used to gather runtime test coverage, test-to-code traceability information, performance, as well as to enable fault injection, diagnostics, 'hot patching' or other runtime execution control. Instrumentation is applied 'on-the-fly' to a device under test, without stopping it, rebooting, or reloading the software image.

In Release 3.3, the engine that generates the dynamic instrumentation for runtime test coverage has been redesigned. Now coverage instrumentation can be generated in 1/10th the time and without using a compiler. It also now supports very large applications allowing as many 10 times as many functions to be instrumented. The memory footprint has also been reduced by almost 90% allowing better use of device memory for resource-constrained devices.

New reboot persistence maintains coverage and performance data across complex tests

Wind River Test Management already supported persistence of analytics instrumentation during reboot, but Release 3.3 now adds persistence of analytics data too. For example, sometimes device testers want to run a test sequence that requires intermediate reboots (or causes errors that result in reboot) and they would like to gather test coverage or performance data across the entire test sequence. Release 3.3 provides a new mechanism to provide instrumentation data persistence across reboots for these complex device testing scenarios.

New reports enable better management of Agile projects and test coverage trends

Wind River Test Management provides key support for change-based testing within iterative or Agile development projects. The product recently introduced key features to help make testers more effective in these constantly changing environments. The system analyzes code changes in binary builds—the most accurate way to identify changes and their impact on your application—and it leverages the system's test-to-code traceability map to automatically determine what regression tests need to be re-run based on changes.

Release 3.3 now adds a new Build Change Summary report that shows exactly what code has been changed/added/deleted between builds, allowing managers to understand code churn and do change-based risk assessment in iterative or Agile processes. Other new reports such as Coverage Trends and Functions Not Covered allow test managers to understand how thorough their tests are and monitor coverage progress throughout the development lifecycle. These new reports add to existing and improved summary reports for Test Coverage, Test Execution Results and lab device asset utilization. Custom reports can be added to the system using 3rd party design tools.

New hybrid test type enables sophisticated device test sequences and facilitates migration from manual to automated testing

Previous Wind River Test Management releases offered either manual OR scripted test cases. Release 3.3 instead now moves to a hybrid test case type that can consist of BOTH manual and scripted test steps. These tests can then be executed in either an "Interactive" or "Automatic" mode. Test teams can utilize automated scripts within a manual test process, executing these interactively with manual steps within a managed environment.

This hybrid test feature also helps provide a migration path from manual to fully automated testing as test teams gradually replace manual steps with scripts. The fully "Automatic" test execution mode is offered for Test Cases that contain only automated test scripts and can run to completion as batch operations without tester involvement (e.g. overnight regression testing). Wind River Test Management is an open system that can run tests built in any scripting or programming language.

New component-based test model improves test modularity and maintainability

Embedded device testers often have test scripts that are used over and over in multiple test cases. Test suites can be simplified if these common tests are stored only in one location and then included by reference elsewhere. It can also be advantageous to modularize tests along with related 'setup' and 'cleanup' sequences. For example, a set of scripts may be required to initialize test equipment, reboot a device, or restore a device to a 'clean' state for further testing. Note, however, that users typically don't want mix application test coverage or other runtime analytics data with analytics from setup and cleanup sequences.

Release 3.3 implements a new modular test case structure and user-defined, reusable, scripted Test Components that let test developers build more self-contained and modular tests. A reboot or login script that is used repeatedly need only be defined once then used in multiple test cases throughout the system. This new test model enhances maintainability as the Test Component need only be updated in one location. The component-based test model has been designed so that its unique runtime analytics information, such as test coverage, is only gathered from the main test functions, not from ancillary setup or cleanup sequences. Modularity also helps Wind River Test Management better optimize test execution based on changed code. Better compartmentalization allows the system to more easily 'mix and match' tests from any source, knowing that each test case is self contained.

Improved test execution and runtime analytics management for multicore devices and multi-device test beds

Wind River Test Management features a Virtual Lab Manager to control, provision and manage a company's test lab assets. Release 3.3 expands existing capability to now control and run tests against multiple cores or devices at the same time. The system also intelligently manages the built-in runtime analytics to intelligently gather 'white box' data from the live device under test. Test coverage analysis, performance measurement, fault injection or dynamic diagnostics instrumentation can be deployed across multiple cores or processors and information is then collected in synchronous with tests.

Release 3.3 also now allows the definition, reservation and management of "Test Beds"—collections of devices and associated test equipment—allowing easy management of varied devices, simulators and instruments embedded testers often need. The Test Bed feature enables design of sophisticated tests that operate on a group of devices and may, for example, communicate with each other as part of a test. Release 3.3 provides the means to send both tests and instrumentation to each of the devices in the Test Bed and manage the test results.

New file attachment feature enables data-driven and GUI testing

Device test developers often prefer to segment reusable or changing test data from the actual test scripts by moving test datasets into external files. Graphical User Interface (GUI) test designers often capture images of correct screens so that testers can compare current versus expected results. Device testers may also need to send datasets, scripts or commands directly to the device as part of a test sequence.

All these scenarios are now supported with the new file Attachments feature in Release 3.3. Test developers can attach any type of file to a test case or to the individual test steps within an interactive test sequence, allowing development of sophisticated, data-driven tests for all types of devices. Attachments can also be useful for test teams that are migrating from manual test processes captured in Word or Excel files to a more automated approach. These existing test descriptions can be attached to Test Management test cases, allowing users to refer to familiar documents while running their tests, thereby speeding adoption of the automated system.

Expanded system customization and usability

Often companies or projects will have their own standard test templates that include unique attributes they want to associate with test cases. Release 3.3 now allows users to create their own test case templates by adding custom fields to the base test case definition. The new release also allows users to customize test execution status descriptions, search test cases with standard or custom fields, and modify test run duration metrics. Usability was also improved in many areas including the ability to easily view HTML device logs or 3rd party test tool results captured by the system.

Where can you get Release 3.3?

Wind River Test Management Release 3.3 is available worldwide on June 11 from your local Wind River representative for new customers and existing users under a Wind River subscription license or support agreement. Release 3.3 is a mandatory upgrade that supersedes all previous releases, Update Packs and patches. Please note you will have to migrate your existing test data to Release 3.3 with the included migration tool. Please backup your current data and thoroughly read the included Version 3.3 Release Notes about database migration before installing any software. Users should check the Wind River Support Downloads page periodically for the latest updates, service packs and patches. See www.windriver.com/support and www.windriver.com/products/test_management for more information.

Workbench and Jazz

By Emeka Nwafor

Today I managed to carve out some quality hands-on time to play with Wind River Workbench integrated with Rational Team Concert - also known as Jazz. While playing with this development environment, I found myself wishing that "I had these types of development tools back when I was doing embedded development environment".

If you are not familiar with Rational Team Concert (RTC), think of it as a collaborative development platform that unifies planning, tracking, automation of software development processes, team collaboration, work item management, and reporting (think dashboards) with traceability across all artifacts that participate in software delivery processes. By artifacts, I mean things like source code, builds, defects, requirements, log files, change sets, etc... In short, RTC is all about connecting team members.

When integrated with Workbench the integrated development environment provides developers with the ability to connect-with and optimize an embedded devicewhile connecting team members that are participating in the software delivery activities. I strongly believe that to be successful, organizations need to effectively manage both device technology and people collaboration concerns.

There is much that can be said about the environment and the integration opportunities. More of this will come out over time. For now, there are 3 simple things that have got me jazzed up about Workbench and RTC:

1. Clean Integration with Eclipse...

Folks close to me will acknowledge that over the last 8 years, I have had a sinusoidal love-struggle-love-struggle-love-struggle feeling about Eclipse. Hopefully, that is behind me now. Building Workbench on the Eclipse platform is the only way we could have enabled this type of integration for embedded device development. The benefit to a Workbench / RTC user is a cleanly integrated usage experience and traceability of artifacts - like Workbench's Memory Analysis log files - with Rational Team Concert's work-item automation engine. A need that I hear from customers is about how they can improve the collaboration between the platform integration teams and the distributed teams developing the device's application logic. The RTC collaboration infrastructure can help here by sharing much of the system build, test case, and environmental information that must be conveyed the developers to help shorten the "time-to-fix" when defects are found.

Another impressive discovery was that RTC is not just for Java. Many of change management related process templates simply just work with the Eclipse CDT. That was a really nice discovery...

2. It is a Very Pleasant Development Environment...

The Jazz platform brings in a number of widgets that simply make it a delightful environment for software development. Things like snapshots, thumbnail views of images, RSS, and chat all add a little sparkle to the Workbench IDE.

3. RTC/Jazz Doesn't Get in the Way!

As we were thinking about the integration of Workbench with RTC, I have to admit that I was a little nervous about the process automation and reporting aspects of the solution. Not the case. The process automation framework is highly customizable; you can enable as much or as little as you want and get value out of the solution without it getting in the way.RTC puts a lot of emphasis on agile workflows. I really like having out-of-box samples and templates to enact the processes of an agile team, but in using it I discovered that teams that are simply doing milestone or time-boxed based planning would benefit from the solution.

I will be demoing the integration of Workbench and RTC for VxWorks 6.8 at the IBM Innovate Conference in Orlando, Florida. The demo will preview some integration work done by Doug Schaefer and enabled by the IBM Ready for Rational Software team - the group responsible for validating that commercial offerings meet IBM's integration criteria and will be well behaved when inserted in our customer's environments. A special "thank you" to IBM Rational software's Pat McCarthy for his patience in running me through RTC bootcamp.

My goal is to use the demo to provoke some conversation on other integration points. If you can't make the demo and are interested in trying out this integration on your own, drop me comment and I'll post the set-up instructions.