We intuitively know that defects in the field are much more expensive than those found in development. In fact, the difference is bigger than you think. If you find a bug in-house you save orders of magnitude in time and money. Unfortunately, the occasional defect does find its way out into the a shipping product and when something goes wrong, shipping back to the factory for diagnosis and repair is the usual remedy. This is costly to the producer and the consumer. In other cases, the system just might be too big to ship anywhere and require onsite technicians – also costly and time consuming. Is returning a product or sending service people the only way? Take a look at this diagram (from www.agitar.com). Cost to fix bugs rises exponentially as you move down the development timeline. After you're shipping your product, the cost per defect has increased 640 times!
An interesting feature that has come from our Test Management product line, and is now part of our standard VxWorks offering (as of version 6.8), is the post mortem kernel debugging capability. This feature captures all the debug information needed form the VxWork kernel, running tasks and Real Time Processes (RTPs) when a system exception is raised. These "core dump" information is stored on the target system and can be transfered back the factory via a network, customer transfer (email, ftp), or in person. This debug capability is available locally during development as well. Once you have the debug information you can use Workbench to pinpoint the error and code path to that error. You can also capture a system viewer trace to help you understand system behaviour such as task and process interactions before the exception occurred. Presumably patches could be sent directly to the device or installed by the customer, saving returns and service trips.
Leveraging remote diagnostics and debugging helps minimize fielded product defect costs by keeping your products with your customers and your service people in-house. We still want to minimize bugs in final products but if it does happen, let's try to make it cheaper. Additional benefits include improved customer satisfaction and loyalty, market perception and product reputation.
If you're interested in seeing this in more detail, join me on Jan 19th for a webinar which will include a demonstration of this capability.