Today we announced the latest version of Wind River Test Management, Release 3.3, our test automation system for monitoring, executing and managing embedded device software testing. Wind River Test Management lets teams optimally execute complex tests while dynamically gathering information from the production software under test as it is running, without requiring special pre-instrumented software builds. This approach allows teams to adopt new white-box test techniques that give testers visibility into the operation of the device and help them determine the thoroughness of the tests, quickly identify defects and performance bottlenecks, and focus efforts on sections of software that are most in need of testing.
Release 3.3 is a major new release that adds a number of significant new features. You can learn more and download several new whitepapers from www.windriver.com/products/test_management.
Here's a quick summary of the key new features
A 10X improvement in scalability enables runtime test coverage for very large apps
Wind River Test Management’s dynamic instrumentation technology allows ‘white box’ access to the device under test. This instrumentation is used to gather runtime test coverage, test-to-code traceability information, performance, as well as to enable fault injection, diagnostics, ‘hot patching’ or other runtime execution control. Instrumentation is applied ‘on-the-fly’ to a device under test, without stopping it, rebooting, or reloading the software image.
In Release 3.3, the engine that generates the dynamic instrumentation for runtime test coverage has been redesigned. Now coverage instrumentation can be generated in 1/10th the time and without using a compiler. It also now supports very large applications allowing as many 10 times as many functions to be instrumented. The memory footprint has also been reduced by almost 90% allowing better use of device memory for resource-constrained devices.
New reboot persistence maintains coverage and performance data across complex tests
Wind River Test Management already supported persistence of analytics instrumentation during reboot, but Release 3.3 now adds persistence of analytics data too. For example, sometimes device testers want to run a test sequence that requires intermediate reboots (or causes errors that result in reboot) and they would like to gather test coverage or performance data across the entire test sequence. Release 3.3 provides a new mechanism to provide instrumentation data persistence across reboots for these complex device testing scenarios.
New reports enable better management of Agile projects and test coverage trends
Wind River Test Management provides key support for change-based testing within iterative or Agile development projects. The product recently introduced key features to help make testers more effective in these constantly changing environments. The system analyzes code changes in binary builds — the most accurate way to identify changes and their impact on your application – and it leverages the system’s test-to-code traceability map to automatically determine what regression tests need to be re-run based on changes.
Release 3.3 now adds a new Build Change Summary report that shows exactly what code has been changed/added/deleted between builds, allowing managers to understand code churn and do change-based risk assessment in iterative or Agile processes. Other new reports such as Coverage Trends and Functions Not Covered allow test managers to understand how thorough their tests are and monitor coverage progress throughout the development lifecycle. These new reports add to existing and improved summary reports for Test Coverage, Test Execution Results and lab device asset utilization. Custom reports can be added to the system using 3rd party design tools.
New hybrid test type enables sophisticated device test sequences and facilitates migration from manual to automated testing
Previous Wind River Test Management releases offered either manual OR scripted test cases. Release 3.3 instead now moves to a hybrid test case type that can consist of BOTH manual and scripted test steps. These tests can then be executed in either an “Interactive” or “Automatic” mode. Test teams can utilize automated scripts within a manual test process, executing these interactively with manual steps within a managed environment.
This hybrid test feature also helps provide a migration path from manual to fully automated testing as test teams gradually replace manual steps with scripts. The fully “Automatic” test execution mode is offered for Test Cases that contain only automated test scripts and can run to completion as batch operations without tester involvement (e.g. overnight regression testing). Wind River Test Management is an open system that can run tests built in any scripting or programming language.
New component-based test model improves test modularity and maintainability
Embedded device testers often have test scripts that are used over and over in multiple test cases. Test suites can be simplified if these common tests are stored only in one location and then included by reference elsewhere. It can also be advantageous to modularize tests along with related ‘setup’ and ‘cleanup’ sequences. For example, a set of scripts may be required to initialize test equipment, reboot a device, or restore a device to a ‘clean’ state for further testing. Note, however, that users typically don’t want mix application test coverage or other runtime analytics data with analytics from setup and cleanup sequences.
Release 3.3 implements a new modular test case structure and user-defined, reusable, scripted Test Components that let test developers build more self-contained and modular tests. A reboot or login script that is used repeatedly need only be defined once then used in multiple test cases throughout the system. This new test model enhances maintainability as the Test Component need only be updated in one location. The component-based test model has been designed so that its unique runtime analytics information, such as test coverage, is only gathered from the main test functions, not from ancillary setup or cleanup sequences. Modularity also helps Wind River Test Management better optimize test execution based on changed code. Better compartmentalization allows the system to more easily ‘mix and match’ tests from any source, knowing that each test case is self contained.
Improved test execution and runtime analytics management for multicore devices and multi-device test beds
Wind River Test Management features a Virtual Lab Manager to control, provision and manage a company’s test lab assets. Release 3.3 expands existing capability to now control and run tests against multiple cores or devices at the same time. The system also intelligently manages the built-in runtime analytics to intelligently gather ‘white box’ data from the live device under test. Test coverage analysis, performance measurement, fault injection or dynamic diagnostics instrumentation can be deployed across multiple cores or processors and information is then collected in synchronous with tests.
Release 3.3 also now allows the definition, reservation and management of “Test Beds” – collections of devices and associated test equipment – allowing easy management of varied devices, simulators and instruments embedded testers often need. The Test Bed feature enables design of sophisticated tests that operate on a group of devices and may, for example, communicate with each other as part of a test. Release 3.3 provides the means to send both tests and instrumentation to each of the devices in the Test Bed and manage the test results.
New file attachment feature enables data-driven and GUI testing
Device test developers often prefer to segment reusable or changing test data from the actual test scripts by moving test datasets into external files. Graphical User Interface (GUI) test designers often capture images of correct screens so that testers can compare current versus expected results. Device testers may also need to send datasets, scripts or commands directly to the device as part of a test sequence.
All these scenarios are now supported with the new file Attachments feature in Release 3.3. Test developers can attach any type of file to a test case or to the individual test steps within an interactive test sequence, allowing development of sophisticated, data-driven tests for all types of devices. Attachments can also be useful for test teams that are migrating from manual test processes captured in Word or Excel files to a more automated approach. These existing test descriptions can be attached to Test Management test cases, allowing users to refer to familiar documents while running their tests, thereby speeding adoption of the automated system.
Expanded system customization and usability
Often companies or projects will have their own standard test templates that include unique attributes they want to associate with test cases. Release 3.3 now allows users to create their own test case templates by adding custom fields to the base test case definition. The new release also allows users to customize test execution status descriptions, search test cases with standard or custom fields, and modify test run duration metrics. Usability was also improved in many areas including the ability to easily view HTML device logs or 3rd party test tool results captured by the system.
Where can you get Release 3.3?
Wind River Test Management Release 3.3 is available worldwide on June 11 from your local Wind River representative for new customers and existing users under a Wind River subscription license or support agreement. Release 3.3 is a mandatory upgrade that supersedes all previous releases, Update Packs and patches. Please note you will have to migrate your existing test data to Release 3.3 with the included migration tool. Please backup your current data and thoroughly read the included Version 3.3 Release Notes about database migration before installing any software. Users should check the Wind River Support Downloads page periodically for the latest updates, service packs and patches. See www.windriver.com/support and www.windriver.com/products/test_management for more information.