Wind River Blog Network

Test Automation Meets Simulation

By Paul Henderson

I'm seeing increasing interest from many companies in using simulation environments with test automation systems to accelerate the testing process. Specifically, putting Wind River Test Management together with Wind River Simics is getting creative juices flowing in industry thought leaders.

Why? Well, development teams have started to realize the benefits of simulation systems for speeding and validating system and software design, and for accelerating software development and debug in advance of hardware availabilty. And even when hardware is available, systems like Simics provide tremendous access and control to speed analysis and diagnsotics.

Continue Reading >>

Transporting Bugs with Checkpoints

By Jakob Engblom

I have a paper about "Transporting Bugs with Checkpoints" to be presented at the S4D (System, Software, SoC and Silicon Debug) conference in Southampton, UK, on September 15 and 16, 2010. The core concept presented is to leverage Wind River Simics checkpointing to capture and move a bug from the bug reporter to the responsible developer. It is a fairly simple idea, but getting it to work efficiently does require that some things are done right.

Continue Reading >>

Wind River and IBM Attack Software Quality

By Paul Henderson

As I've mentioned before, we've been working with IBM Rational for some time around quality management automation. Both companies see the skyrocketing software content and architectural complexity in the embedded device market as creating a tipping point where companies will not be able to continue with business as usual.

Product development teams will need to take a more managed and automated approach to quality that spans across the lifecycle and access into the devices under test. This is particularly true in markets that require strict adherance to standards and compliance regulations.

We put together a joint whitepaper on this subject downloadable from here. And we are also having a joint web seminar next week on Tuesday Aug 31 at 2pm EDT. You can register for this event here.

Continue Reading >>

It’s Time for Testers to Step Up

By Paul Henderson

RTC Magazine recently published an article that I wrote called "Time to Rethink Software Testing for Embedded Devices". In it I describe some of the new techniques that are possible, and I believe necessary, to delivery high quality device software for embedded devices.

• When staying 'positive' doesn't pay
• Getting negative with white box testing
• Focusing on the 'deltas' with change-based test automation

Continue Reading >>

VxWorks = Secure

By Nikhil Chauhan

A recent report describes potential security vulnerabilities in devices running VxWorks. Researcher HD Moore claimed during a recent talk ( slides) that a quarter million devices accessible directly from the Internet were found to be vulnerable.

VxWorks has a very strong track record of offering secure products. However, we also realize that vulnerabilities can affect VxWorks, even if very infrequently. In those cases, Wind River will act quickly to address any issues. Regarding recent vulnerabilities, Wind River responded rapidly with patches and remediation steps in conjunction with a public announcement by the CERT Coordination Center on August 2, 2010. Once CERT notified Wind River, Wind River immediately assessed the alert and was instructed by CERT to release a synchronous public response. We're confident that our customers know that Wind River is committed to supporting its products with the highest quality and security standards.

Continue Reading >>

Test Driven Development Meets Continuous Integration

By Paul Henderson

In my last posting I mentioned I'd be running a webinar with James Grenning on Agile testing. James is a recognized expert and frequent speaker on the topic of software development and one of the original authors of the Manifesto for Agile Software Development.

We talked about the case for agility where today's embedded software projects are inevitably faced with changing requirements and market conditions that cause unplanned, mid-course corrections. The result is what went in is often not what was expected to come out. Testing folks are the tail trying to wag the dog as they try to test in quality at the end of the project.

Continue Reading >>

Agile Testing for Embedded Devices

By Paul Henderson

I am running a Web seminar on agile testing with James Grenning. James is a recognized expert and frequent speaker on the topic of software development. Founder of Renaissance Software Consulting, he provides training, coaching, and consulting to corporate and government clients worldwide. James is one of the original authors of the Manifesto for Agile Software Development and is presently leading efforts to introduce Agile development practices to the challenging world of embedded systems.

Iterative software development methods like Agile are being adopted by many software and information technology (IT) organizations across the industry. Quick design-develop-test cycles let teams respond better to changing requirements while providing timely feedback on features and quality. Agile methods help companies produce higher-quality software faster and at lower costs.

Continue Reading >>

Industry Investing in Better Device Runtime Visibility During Testing

By Paul Henderson

Here’s the final installment in my series about our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog postings).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.com and I will send it to you.

In this section of the survey we asked participants about what test tools they use today and where they are investing in test automation. Given the high cost of product failure, accelerating complexity and reduced schedules the industry is turning to more test automation in 2010 to help address these problems. The top investment are moving to new tools that can help test teams and their management better understand how well they are testing, better focus their efforts on the areas needing testing, and reduce cycle time through more automation.

Continue Reading >>

The High Cost of Poor Quality – Brand, Market, Budget

By Paul Henderson

I’m continuing my series on our embedded device industry software testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog posting).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.com and I will send it to you.

In this section of the survey we asked participants about how they measure the high cost of poor quality. Respondents told us that the true cost of poor quality is much higher than program budget. The majority of respondents showed that the true cost of poor quality is measure by damage to company brand and lost revenue due to missed market windows.

Continue Reading >>

Inadequate Management Visibility into Quality is Eroding Confidence

By Paul Henderson

Here’s the next installment in my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog postings).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.com and I will send it to you.

In this section of the survey we asked participants about how they measure software quality today, the metrics most often cited by survey respondents were reactive in nature such as tracking customer-reported failures and open defects rather than metrics that can help them prevent defects.

Continue Reading >>

Compressed Schedules Driving Shorter Testing & Defect Resolution Requirements

By Paul Henderson

Today I'm continuing my series on our embedded device software industry testing survey conducted in April-May 2010 with almost 900 respondents (see previous blog posting).  If you’d like a copy of the full report in pdf, please drop me an email at paul.henderson@windriver.comand I will send it to you.

In part 2 of the survey we asked about schedule compression and what affect that was having on the device testing cycle. A majority of survey participants reported that market conditions have forced them to shorten their development schedules by as much as 18 months.

Continue Reading >>

True Concurrency is Truly Different (Again)

By Jakob Engblom

A recent article at Ars Technica describes yet another security flaw in Windows. Nothing much new in that respect, but this is indeed an interesting attack in that it is enabled by using multicore hardware. It is not practical on a single processor, demonstrating once again how multicore is fundamentally different from multitasking on a single processor.

The attack targets software that hooks Windows kernel code to do additional work, such as anti-virus software. The idea is to get bad data into a hooked system call, by replacing input data after it has been validated by the hook but before it gets consumed by the kernel.

Continue Reading >>

A Crisis of Complexity – Industry Report on Growing Challenges in Embedded Testing

By Paul Henderson

I’ve been talking a lot with embedded device companies around the world over the last few years and I am hearing growing concerns about software testing. I’ve mentioned several of these concerns in previous blogs. I wanted to get more quantifiable data and get some feedback that could help us shape our products and services to help. So I decided to run a survey to gather important data from our community.

The focus of this survey was to gain a detailed snapshot of how executives, development managers, QA and test team leaders and other involved staff currently view the embedded device software quality test landscape. Recent changes, new challenges and strategies for managing them were of particular interest.  So I fielded a four-part survey to individuals who work for embedded products companies. In total, nearly 35,000 individuals in North America were invited to participate in the survey via emails. 

Continue Reading >>

What’s New in Wind River Test Management 3.3?

By Paul Henderson

Today we announced the latest version of Wind River Test Management, Release 3.3, our test automation system for monitoring, executing and managing embedded device software testing. Wind River Test Management lets teams optimally execute complex tests while dynamically gathering information from the production software under test as it is running, without requiring special pre-instrumented software builds. This approach allows teams to adopt new white-box test techniques that give testers visibility into the operation of the device and help them determine the thoroughness of the tests, quickly identify defects and performance bottlenecks, and focus efforts on sections of software that are most in need of testing.

Release 3.3 is a major new release that adds a number of significant new features. You can learn more and download several new whitepapers from www.windriver.com/products/test_management.

Continue Reading >>

ESC Silicon Valley

By Mark Hermeling

Looking forward to my trip to ESC Sillicon Valley next week. It is promising to be a busy show, especially since ESC is now combined with the Multicore expo. I just leafed through the agenda (in the form of a Nxtbook) and found a large number of sessions that I want to attend, experience show though that I'll probably be too busy talking to customer to attend sessions, which is a good problem to have of course.

I am hosting a 4 hour session (with several of my colleagues) on Multicore Demystified on Tuesday afternoon 2.30-7pm (there will be refresments!) in the Hilton Plaza Room. Do stop by either the session, or our booth at Multicore Expo for a chat if you want to brain storm about your next generation devices.

Continue Reading >>

Time to Get Positive about Negative Testing

By Paul Henderson

I spend a lot of time talking to device development and test groups and I continue to be surprised by a number of things. One is how little test automation is in place if you look across embedded device companies and industries.

But another surprise is how little companies are doing in the way of “negative” testing. By this I mean testing that tries to break the system, validate fault and exception handlers or otherwise force the device in to an unusual state or “edge condition”.

Continue Reading >>

Community Survey: Device Software Testing and Your Changing Requirements

By Paul Henderson

If you’re like me, you’re amazed at the pace of change occurring in the embedded industry. Our corner of the technology world is racing through changes – in platforms, functionality, development approaches and business requirements – faster than ever. 

From a product management standpoint, it’s a challenge to stay on top of these new developments, and how their impact varies from one industry to the next. We must understand how these changes are playing out in your shop, and how they are impacting you and your teams.

Continue Reading >>

Get Real, Go Virtual

By Michel Genard

I am going to make a bold statement: Go virtual to create reality. Seems crazy enough?

According to Wikipedia, reality means "the state of things as they actually exist.” The concept of existence can be elusive, because to exist, something doesn’t need to be real; it just needs to have its own relevance. Often we humans use simplified terms to explain complex concepts or situations and we describe phenomena in mathematical equations. We do it all the time in many domains such as philosophy, physics, nuclear, and chemistry, and it works.

Continue Reading >>

Software Quality: What You Don’t Know CAN Hurt You

By Paul Henderson

On Tuesday of this week, Jim Lentz, president of Toyota Motors Sales USA, testified before the House Energy and Commerce Committee about the car manufacturer’s problems and recalls.  Yes, it was Mr. Lentz’s turn in the proverbial hot seat.

During pointed questioning by House members, Mr. Lentz said that the repairs prescribed by Toyota might “not totally” solve the problem of unintended sudden acceleration in its vehicles. He also admitted that Toyota is still trying to determine the source problem, including the possibility – previously denied – that the vehicles’ electronics systems (software!) might be at fault.

Continue Reading >>

Quit Bugging Me: Revalidated

By Mike Deliman

One day, I get this phone call. "We're working with the system, we see the calls that update the exception handlers early on - connecting the clock routines, etc. Then not very much farther on we see the system has run past where tasking should be running but it's not. 

When we check, we're not getting any clock interrupts, and the clock exception handler isn't there any more. How could the system do that? Is there any way the OS could remove it's clock connections?"

Continue Reading >>

Case study: Ultra Datel Safety-Critical Avionics Upgrade Using COTS

By Paul Parkinson

I recently had the privilege of working with one of our partners, LDRA, and one of our customers, Ultra Datel, on writing a case study of their experiences of a mid-life upgrade of an existing avionics system.

What caught my attention was the fact that the existing system was uncertified, and the upgrade involved migrating the existing system to a commercial-off-the-shelf (COTS) and undertaking DO-178B Level B safety certification.

Continue Reading >>

What Exactly are You Testing, and How?

By Emeka Nwafor

"What exactly are you testing? How are you testing it?"

I'm sure that these are amongst the questions that product, business, and technical managers across the embedded software industry have asked themselves on several occasions throughout their careers. I know I have.

The importance of testing embedded software isn't new. In my "youth", I remember our VP of Product Development circulating a white paper that discussed the devastating impact resulting from the deployment of an untested "simple" patch to some switching software and declaring that "this cannot happen to us".

Continue Reading >>

Quit bugging me: surprise NaN!

By Mike Deliman

You've got a complex system. There are dozens of tasks, a handful of interrupt routines. Your system runs fine, for hours or sometimes days on end. The loading on your system is somewhat "bursty" in that average loading is only about 20% of peak load, and peaks only happen once every hour for less than 5 seconds. Every once in a while, for whatever reason, you suddenly get a divide by zero, not-a-number, or other unanticipated erroneous results on one of your floating point calculations.

In your system you only have a few tasks that employ floating point. You're pretty sure none of your ISRs do, all they do is copy or move data around, they shouldn't have to compute anything. Yet for some reason, occasionally any one of the tasks running floating point will go belly-up with unexplainable results. It just looks like some kind of random corruption.

Continue Reading ››

Quit Bugging Me: Induction

By Mike Deliman

Working on a customer problem once, we had an interesting phenomena. Upgrading a system with a large VME cage and several boards, the customer replaced older processor boards with what were then "new" boards. The old boards ran at (I think) 33 MHz, the new ones at more like 133MHz.

The overall system included motor control functions and sensor feedback, there were D-to-A, A-to-D, and other boards in the system. The problem was, with a straight-and-simple rebuild of software for the new boards, the system was giving anomalous readings. Checking the registers for values from the A-to-D boards, there were numbers showing up that were not possible. Readings indicated the motors were on and moving their load when in fact the motors were not connected. Switching back to the old computer boards eliminated the problem. We added some hardware to examine VME bus activity, to see if the new CPUs were reading the bus wrong. We had to put the test board between the CPUs and the sampling boards.

Continue Reading ››

Targeting Your Assets

By Paul Henderson

Embedded folks know that managing your target device is a key part of the development process. While you can do a lot on the host, the 'rubber meets the road' when you run your software on the actual target hardware at full speed.

Developers using IDE's like Wind River Workbench have a nice target management environment that fully supports cross development. But testers don't have this luxury. In testing, the whole process of managing devices is slow and painful.

Most companies have a lab full of equipment covering new and old devices of all configurations. Often with new devices, the prototype hardware needed by the team is scarce and expensive. Plus, these labs are spread across the world -- Americas, Europe, China, India.... How do you know where the equipment is, and how do you get access to the device or board you need remotely - now?

Continue Reading ››

Carrier Grade Starts with Finding and Fixing Software Defects Before Deployment

By Jessica Schieve

Did you know 25% to 50% of deployed device failures are due to software defects? This statistic should be nowhere near this high, considering there are a few very good solutions out there today to help reduce the number of defects in software code.

In the networking and telecom industry, if equipment fails and brings down the network it can cost a service provider or operator millions of dollars in lost revenue or even lost customers (read this white paper to learn more). If you are an equipment provider and the failure is associated with your application, the lost revenue cost could very well be passed on to you. And count on the network failure to be all over the headline news like the one that hit the New York Times.

Continue Reading ››

The Software Managers’ Blind Spot....

By Paul Henderson

I’m amazed at the consistency of input I’m getting from device software execs around the world about their lack of visibility into their testing process. Everybody is spending huge amounts of time and money (30-50% of their projects) on testing, but I haven’t found anyone yet that is comfortable with the result. I hear statements like this consistently:

“We have thousands of test cases and we’re afraid to not run them. In fact we are adding more all the time. But we really don’t know what software we are actually testing.”

Continue Reading ››

Runtime Technology Turning Heads in Device Test

By Paul Henderson

When Wind River introduced Sensorpoint runtime instrumentation technology a few years ago it was well received as an enabler for the embedded device diagnostics process. Recently however, with the introduction of the Sensorpoint-enabled Wind River Test Management System, this same ability to instrument device software binaries – dynamically, on-demand and on-the-fly without stopping or reloading the device -- has been turning heads in device test and quality assurance groups too. The need to get runtime access to device software internals while under test is turning out to be a ‘killer app’ for device test teams. Wind River is now providing four runtime analytic capabilities as out-of-the-box features in its new product.

It’s often difficult for testers to measure the quality of their test plan, optimize test execution, or know what device software has actually been tested. Source-based code coverage tools are helpful, but they are often too cumbersome and intrusive to be used by testers on a regular basis in actual target devices. Wind River Test Management uses Sensorpoint dynamic instrumentation to detect and record specifically what software functions within a build were tested or – more importantly – were NOT tested. This runtime code coverage capability can be enabled and disabled by testers on-demand for specific modules and test runs, letting testers and their managers know how much of their device is actually being tested for each test run or cumulatively over a set of runs.

Continue Reading ››

Software Certification

By Alex Wilson

I wonder why it is always the case that the period of time just before year end is suddenly very hectic! There must be a universal law about that, something like Murphy's Law...

Anyway, I have just returned from 2 weeks of travel around Italy, Sweden and the US visiting customers and presenting our solutions. I always enjoy meeting folks and seeing what they can do with our software as I love to see the latest and greatest devices; and so I have been negligent in posting to my blog for that time and noticed over the last few weeks some noticeable events have taken place.

Continue Reading ››

Mitigating Risk

By Mike Deliman

The phone rings.  A customer has an application destined for a high-risk environment, and somewhere in test they've found a new unanticipated condition.  A "Tiger Team" is formed - a group of experts who've "been here before", who understand the priority and the risks, and know how serious it is. Many times a Tiger Team may give the go / no-go, the final answer that either saves a mission, or drops all that work into the dust-bin.  Such a team itself represents a *lot* of work, frustration, and time - investigations may run on days, weeks, or months, as long as they reach their conclusion on-time.  And there's the one facet that can't be changed by any engineering practice:  Time.

Continue Reading ››