Bill Graham

The ultimate head start

I was very excited on hearing about Intel's acquisition of Virtutech and subsequent announcement that Simics was going to be part of our product line. Why? Because Simics gives you access to hardware platforms before they're even made (including multicore platforms.) In my experience, access to hardware platforms and having enough for everyone is a constant concern during development. Many developers are targeting their new product for new hardware platforms that are still in development and hoping for the stars to align so their software and the hardware platform delivery dates align. Now, if you are on this bleeding edge of technology what can you do to ensure success?

Clearly, if the hardware doesn't exist you need to proceed regardless and you can develop the application software to certain extent but their comes a point where you need to start integrating and testing the system. This is were Simics comes in, letting you build a virtual hardware platform when the hardware doesn't exist or there isn't enough to go around. What's also exciting is the ability to simulate systems at the processor, board and system level (e.g. boards in a chassis.) This virtual platform simulates the hardware very accurately to the point, for example, where the same VxWorks BSP can be used for hardware or virtual platform. This platform can therefore run VxWorks, middleware and applications plus with our tools just like any other target platform.

A virtual platform means your project can proceed full speed to meet time to market windows that are so tight in today's market. More importantly though, are the other advantages of the virtual platform such as the ability to do advanced debugging, check points and restores, real external connections to other systems, and fault injections. Plus, everyone on the team can use this virtual platform, something that is impossible with the typically short supply of prototype boards (if you have hardware at all). Once the hardware comes in you can quickly move to the real hardware platform and deliver your product. Avoiding that last minute integration "Big Bang" saves time, money and improves quality. More and earlier access to the target platform plus advantages of a virtual software-based platform equals the utlimate head start.

The Multicore transition: tools are key to success

We just announced a new release of our state of the art tools which includes an update to Workbench and Workbench OCD version 3.2, plus our Wind River Compiler suite (also known as the Diab compiler). This news reminded me of my days as a product manager for tools software and the need to talk about the importance of tools to project success.

For embedded software companies where so much emphasis is given to supported hardware, operating systems and middleware technologies, tools can get ignored in the fray. This is unfortunate because tools are key to project success because they can make the difference between meeting a deadline or missing it by many weeks. Tools are all about developer productivity and getting problems solved faster.

How do tools make us work better and smarter? By reducing the inherent complexity of the task at hand - by using visual tools rather than command line, for example.

By pinpointing errors and giving us all the trace information we need to find the root of the problem.

By providing hardware integration via JTAG with our On Chip Debugging (OCD) tools which gives the developer complete control via stop mode debugging on the device under test.

By providing an optimized compiler tuned for specific features of the target processors.

All of these capabilities become critical to productivity to the point that they become essential components of the development environment. Equally critical is the ability to support all of the major processor platforms - ARM, Coldfire, Intel, MIPS, and PowerPC architectures. In fact, our OCD product can be used for VxWorks, Linux and other operating systems - in other words, its a great JTAG debugging solution. Similarly our Wind River Compiler can be bought and used separately for any type of cross development project.

Although tools have been important in the past, I think the multicore transition requires even more tools support. Multicore adds complexity to the system architecture and true hardware parallelism, although powerful, is a new programming model for developers. Our tools group has been working very hard to support our customers through this transition by solving such critical issues like deadlocks, memory corruption, and core synchronization. Moreover, visualization of multicore behavior and hardware run-control operations on multiple target systems and individual CPU cores is critical to taking control of your product debug and optimization.

So, let's not forget about the tools - the keys to multicore success.

Systems huge and really tiny

"Diverse" is certainly as word that applies to embedded systems. In fact, it's difficult to really define embedded systems because they can be a windshield wiper controller, a home theater remote control or an Internet backbone switch that handles gigabytes of data per second. VxWorks has been used in all of these types of systems and can do this because it offers flexible solutions for our customers. As systems become larger and more complex customers are usually seeking more options such as middleware for networking or graphics. As systems get smaller, however, customers are usually seeking less options and the ability to configure systems exactly the way they want.

We introduced VxWorks source builds in VxWorks 6.7 which gives you the capability to build VxWorks from the ground up and customize what is included. There is also an easy to use kernel configuration tool in Wind River Workbench that makes setting up and building a kernel straightforward. Expanding on this for VxWorks 6.8, we introduced the concept of profiles which are predefined kernel configurations and in particular we created a small memory footprint for ARM processors. Why ARM and small? Well, generally ARM processors get used in small memory constrained systems built by customers looking to build low power and low cost devices. These customers are also looking to minimize memory usage. In fact, the minimum kernel configuration can fit into less than 80KB which, by today's standards, is very small. Based on this tiny memory footprint, customers can add services as they need while maintaining the minimum memory profile.

Why worry about small memory footprint when embedded systems can have gigabytes of memory? Memory is still relatively expensive and using ess memory means less device cost to build and hopefully better profit margin when it goes to market!

It's all about the evidence

An important distinction that gets drilled into people new to safety critical systems and certification of said systems is the difference between "certified" and "certification evidence." When certifying a safety critical system to a very strict standard such as DO-178B, the system as a whole is certified not individual components such as operating systems or middleware. Verification and validation of safety critical systems is very expensive and time consuming. However, to greatly reduce the burden of testing and test reporting,  the use of Commercial Off-The-Shelf (COTS) hardware and software is required. However, COTS hardware and software must adhere to the same strict requirements for verification and validation as the systems they are being used for. In order for VxWorks DO-178B to be used, we must provide certification evidence along with the product - providing proof that these strict standards were followed. For the customer, this means a huge cost savings because they spend much less time proving the air-worthiness of COTS components and spend the time verifying and validating their own applications.

Our recent update to the VxWorks DO-178B platform is a case in point. Wind River took the initiative to provide DO-178B certification evidence for the board support package for a COTS Single Board Computer (SBC) from Curtiss-Wright Controls - the DMV/SVME-183. This is a popular board for avionics systems and by providing certification evidence for this combination is a great time and money saver for customers. 

Also of interest in this new release is the addition of support for Intel Architecture IA-32 and Power Architecture e600 processors.

VxWorks is helping power the next generation of wireless networks

You are probably wondering what an RTOS has to do with wireless networks. Well, a lot, actually.  VxWorks is used extensively in telecom/datacom infrastructure including 3G and next generation networks 4G, LTE (Long Term Evolution) and WiMax. As you have likely heard, the current 3G networks are straining from the demand from new wireless smartphones such as the iPhone. Networking companies that build the infrastructure for these networks have clear marching orders - the next generation must handle more data and for more customers. We're hearing requirements for 10 to 20 times the current capabilities.

Given the high throughput and the need to react to signaling and rerouting instructions, real-time operating systems are common in this environment. VxWorks has been a platform of choice for networking companies because of its broad hardware support, multicore capabilities (both in Symmetric (SMP) and Asymmetric Multiprocessing (AMP) configurations), offload acceleration and advanced networking. Of note in our recent VxWorks 6.8 release, is updates to our mobile IP stack which is of interest to companies building next generation wireless infrastructure.

• Compression for  IP headers to efficiently use “over the air” bandwidth

• Includes RoHC (Robust Header Compression) RFCs 3095, 3096, 3843, 4815

• IPv4 and IPv6 implementations
• Support for mobile node, home agent, and foreign agent
• Comprehensive RFC support

It's exciting to be part of this transition to high speed mobile networks both as a consumer (of data) and a provider of software for the next generation.

Getting shipped products fixed; service and repair is not the only option

We intuitively know that defects in the field are much more expensive than those found in development. In fact, the difference is bigger than you think. If you find a bug in-house you save orders of magnitude in time and money. Unfortunately, the occasional defect does find its way out into the a shipping product and when something goes wrong, shipping back to the factory for diagnosis and repair is the usual remedy. This is costly to the producer and the consumer. In other cases, the system just might be too big to ship anywhere and require onsite technicians - also costly and time consuming. Is returning a product or sending service people the only way? Take a look at this diagram (from www.agitar.com). Cost to fix bugs rises exponentially as you move down the development timeline. After you're shipping your product, the cost per defect has increased 640 times!

An interesting feature that has come from our Test Management product line, and is now part of our standard VxWorks offering (as of version 6.8), is the post mortem kernel debugging capability. This feature captures all the debug information needed form the VxWork kernel, running tasks and Real Time Processes (RTPs) when a system exception is raised. These "core dump" information is stored on the target system and can be transfered back the factory via a network, customer transfer (email, ftp), or in person. This debug capability is available locally during development as well. Once you have the debug information you can use Workbench to pinpoint the error and code path to that error. You can also capture a system viewer trace to help you understand system behaviour such as task and process interactions before the exception occurred. Presumably patches could be sent directly to the device or installed by the customer, saving returns and service trips.

Leveraging remote diagnostics and debugging helps minimize fielded product defect costs by keeping your products with your customers and your service people in-house. We still want to minimize bugs in final products but if it does happen, let's try to make it cheaper. Additional benefits include improved customer satisfaction and loyalty, market perception and product reputation.

If you're interested in seeing this in more detail, join me on Jan 19th for a webinar which will include a demonstration of this capability.

VxWorks 6.8: Broadening our horizons

Today marks the announcement of our latest VxWorks release, version 6.8. It's difficult to summarize a major release for VxWorks because there are so many parts to a modern RTOS - it's not just a kernel anymore. Therefore, I'm going to do a series of posts that hopefully cover some of the highlights of the release. For customers interested in all the details, our online support site has that covered.

If I was going to highlight one thing for this release, it would be the significant increase in processor support for new multicore chips. The post title was chosen for this reason, we are broadening the horizons of our multicore solution with each release to keep pace with the hottest new embedded platforms. VxWorks 6.8 adds support for ARM 11 MPCore and Cortex A9, Cavium Octeon CN3xxx and CN5xxx, RMI XLR and XLS, Freescale QorIQ P2020 and Intel Core i7. This list is really a who's who of multicore processors and for a significant set of major processor architectures - ARM, Intel Architecture, Power PC, and MIPS.

Multicore is a significant factor in embedded systems and we are striving to keep VxWorks the product of choice for customers seeking a multicore RTOS solution. Part of keeping up with the trend is processor support but also improvements to Symmetric Multiprocessing (SMP) and Asymmetric Multiprocessing (AMP) capabilities. In fact, VxWorks is seeing significant success in fielded AMP-based systems which indicates that both types of solutions are needed and important (despite what you may hear from other vendors). Some of the key multicore improvements made in VxWorks 6.8 include:

• SMP core reservation - this means you can dedicate a core to a single process and only that process. VxWorks has already supported core affinity where you can assign a process/thread to a specific core but not the one and only process that runs on that core. The advantage of core reservation is the ability to build AMP-like architectures with an SMP system. For example, you can create a set of dedicated processing engines by reserving one core for each engine, thus ensuring only the processing engines run on each core.
• Major upgrade to Wind River Multi-OS Inter-Process Communication services (MIPC version 2.0). MIPC allows multiple operating systems to talk to each other using a sockets-like interface. MIPC can connect operating systems on the same chip, board, or network. The performance of MIPC 2.0 has been improved and it now includes support for virtual network and serial interfaces.
• Performance improvements for SMP and general (multicore and single core) architecture optimizations for ARM, MIPS and Intel Architecture.
• Spin locks for AMP architectures to provide a common, reliable synchronization mechanism for multiple VxWork instances in an AMP configuration.

Stay tuned for more posts on some of the other interesting highlights for VxWorks 6.8.

The Rise to Dominance of NAND Flash

Flash memory is the dominant form of non-volatile memory for embedded systems and has been since the EEPROM days. I remember using an ultraviolet lamp to erase the old EPROMs, those were the days. You could erase your EPROM and permanently damage your eyesight at the same time!

NOR flash, traditionally dominant in embedded systems, has provided as a relatively small amount of non-volatile memory for booting the device and retaining critical parameters during power-off. NOR flash was a good choice because it supports execute-in-place which removes the need to copy the entire image from flash to RAM to execute. Given the relatively small amount of storage needed in the past, NOR remained the dominant technology.

In the past few years the demand for NAND flash has increased significantly. According to Gartner the market has grown from $7 billion to $15 billion since 2004. The significant contribution to this growth is USB flash drives and digital music players. In fact, Apple alone can cause shortages in the NAND flash marketplace with devices like the iPhone and iPods. Why has NAND taken off as the dominant technology? One of the reasons is the difference in silicon-level geometry. NAND flash gates are about 60% smaller than the equivalent NOR gate. This allows more gates per die and therefore higher density and more storage capacity.

There are also other advantages to NAND flash in that it behaves more like a hard disk since the storage media is sector-based making it ideal for sequential data (mp3s, pictures, video). NAND flash also has very fast write and erase performance. NOR and NAND flash have similar sustained read speeds although NOR has the advantage in random access. However, its structural advantage is arising in much greater densities and better overall performance, NAND has become the technology leader in flash. Consequently, as the demand for higher capacity continued, NAND has become the market revenue leader as well.

Datalight FlashFX Pro
Embedded operating systems need to support flash file systems since many embedded systems use flash nowadays. Flash is treated like a block file system in the OS but requires specialized software at the lower driver level. In our case we rely on our partner, Datalight, with their FlashFX Pro software to provide this interface. The beauty of this solution is that the OS can treat the flash media as a regular block-based disk and the FlashFX Pro software deals with the complexity of the underlying media.

Why is there the need for a media interface like this? NAND is more complex to deal with than hard disk. For example, there is limited number of writes that a NAND device can support. Wear levelling and bad block management is critical to device reliability and longevity, however, it needs to be done properly and efficiently. Why FlashFX Pro? Datalight is the flash memory media management expert r and provides support for 100's of different brands of flash. The company’s proven solution provides high performance, reliability and broad hardware support - all key things for embedded devices.

Our recent reseller announcement with Datalight is great timing because flash memory file systems are table-stakes for embedded systems. We can now offer FlashFX Pro directly to our customers.

Busting the Myths around VxWorks #6: Safety

Software safety is related to many factors but I think the impression that an operating system is unreliable conjures up the perception it is unsafe as well. I’ve spent time in other posts dispelling myths surrounding VxWorks reliability. Rather than repeat myself I’ll point to my previous post on reliability – the points made there, apply here too. The proof point for the safety of VxWorks lies in the success of our customers.

VxWorks supports three of the most important and stringent safety standards in the world. Firstly for the commercial and military avionics market we have VxWorks ARINC 653 and VxWorks DO-178B. Both of these products are certifiable for commercial aviation, which is has the strictest safety standards for software and hardware. Interestingly, the technology in these products can be used for safety critical devices whether they need officially certification. Example would be industrial control and medical devices. In addition, we have the VxWorks IEC61508 platform for industrial and medical systems – the key certification for safety in these markets. In products where safety is a concern, VxWorks is usually a first choice. 

Busting the Myths Around VxWorks #5: Security

In reality this isn’t a myth, it’s more of an awareness problem. Our solution for the need for high robustness systems is the VxWorks MILS product. VxWorks MILS is designed to provide Multiple independent Levels of Security (MLS), a security concept that ensures that complete separation is provided between secure and non-secure portions of a system (and that it is impossible to subvert this to gain access to the classified partition in the system from the non-secure). 

We have designed the MILS product to conform to the separation kernel protection profile as defined by the Common Criteria (aka ISO 15408) and is currently undergoing official certification to Evaluation Assurance Level (EAL) 6+. Now I appreciate this appears like a lot of buzzwords but customers looking to build high robustness secure systems will understand the need for this type of product. 

Now why is VxWorks MILS (and MLS) important technology? It reduces costs. Rather than have redundant hardware to provide physical separation of classified and non-classified information, MILS provides a way to do this on one device. It also reduces the burden in the field (or battlefield) to allow operators to have just one device that does secure and non-secure communication rather than many. Our VP for Product Strategy and Marketing for VxWorks, Marc Brown, recently did an interview with Military Embedded Systems Magazine on this very topic.