When embedded software testing is a life or death issue

By Ido Sarig

Ido-headshotWhen I was in high school, there was a popular TV show called “Moral Dilemma” that aired in my native country. It featured a panel of professionals such asdoctors and lawyers who discussed “life or death” issues related to their daily work:  Would you represent and defend a client charged with murder after he confided in you that he indeed killed his victim? A kidney has become available for transplant – do you operate on the rich, 85-year-old patient who promised to bequeath a million dollars toward the creation of an organ bank at your hospital, or provide the kidney to the teenager who was just accepted into the country’s most prestigious college?

The show portrayed these professions as dealing with the most important moral issues imaginable, and by implication, relegated other professions to a lower rung on the ladder. As someone growing up from a family of engineers, I knew engineers can have significant roles to play and I definitely had respect for other professions not represented on this TV show. However, it did get me to thinking: Were there also life or death issues in software?

Fast forward 30 years, and I know that there are indeed life or death scenarios in the world of engineering and software. Embedded software is a vital element in the majority of today's sophisticated medical devices—and when it fails, disastrous results follow.

The other day, I was reading  about a (yet another!) FDA Class I recall of infusion pumps.  This particular recall was ordered before any malfunction actually impacted a patient, but previous instances were not so fortunate. Problems with infusion pumps have been linked to more than 56,000 reported complications over the past five years, including at least 500 deaths.

The irony (if you can call it that) in this particular recall is that these specific infusion systems feature a wireless radio enabling remote programming that was being marketed as a patient-safety solution to manual programming errors - yet it was precisely this wireless radio component which failed and triggered the recall. According to the FDA, when used in a wireless network environment that utilizes Temporal Key Integrity Protocol (TKIP) authentication, a particular brand of compact flash can potentially induce a memory leak that can cause the Management Processor to become non-responsive. This causes normal operation to stop, preventing the pump from delivering the medicine with no visual error warning to alert the user that the pump is not working.

How does this happen? A key element to consider in this situation could be insufficient medical device testing.  As device software grows in functionality and complexity, the process to test and verify that it works correctly becomes increasingly complex and difficult. Medical devices manufacturers have a nearly endless matrix of combinations that require testing.  Making matters worse, market dynamics force device manufactures to shrink their product development cycles, leaving less time than ever for comprehensive testing programs.

In this case, it would have been critical to test not only the Motorola Compact flash which failed vs. some-other-vendor's flash, but also the use of TKIP vs. other authentication protocols [the device supports at least 4: WEP (40 and 128 bit keys), WPA-PSK or WPA2-PSK), 3 different wireless LAN standards (802.11 a/b/g), and 2 ways of obtaining IP addresses (Static or Dynamic (DHCP)]. So, there are 24 different configurations for each type of compact flash just to test basic wireless connectivity issues, and this is before we even get the functional testing of the various infusion modes, dosage programming modes, error code checking etc…

And this is where the moral issues come into play. Exhaustively testing every possible combination of software and hardware would likely take many months, and would have to be repeated every time a new software version is released by the developers, which happens quite frequently during the development cycle.

The manufacturer does not operate in a vacuum: there will always bemarket pressures and competitors releasing new models as well. But is it right to release a medical device that has not been thoroughly tested in order to better compete? And how do we know enough testing has been done to sufficiently cover all the complex functionality that goes into modern infusion pumps?

Luckily, we can reduce some of this uncertainty by using tools like Wind River Test Management  – which provide us with clear indication of which parts of our code has not been tested at all, help us focus our testing efforts on those risky parts of the code, and provide management with insight into the quality of the device they are about to ship.

I recently discussed some of these issues at a Webinar titled. “Medical Device Software Testing: A Life-and-Death Challenge.” You can view a recording of it here, and learn what you can do today to improve the quality of software in your medical devices.