Multi-core & Embedded Virtualization

The Advent of Multi-core MILS

By Paul Parkinson

It has been a rather long time since my previous blog post. I intended to post a blog before the holidays regarding our partner Curtiss-Wright's announcement about VxWorks MILS support on the VPX6-187, but I simply ran out of time! So here it is as my first blog post of 2013.

The Curtiss-Wright new release was actually quite a game changer. To the casual reader, it may just appear to be about the release of another board support package (BSP) on a leading-edge COTS processor card, but the fact that this BSP supports multi-core for a MILS architecture is what makes this announcement so significant.

When the MILS software architecture was proposed in 1984 by the computer scientist Dr. John Rushby, microprocessors at the time were inherently single-core, and computing performance advances were being driven by clock speed increases and die size, at a rate following Moore's Law.

Over the last decade, single-core processors with robust MMU partitioning have become sufficiently performant to enable viable MILS separation kernels (SK). In recent years, Wind River has implemented VxWorks MILS on single-core processor architectures (PowerPC MPC8548), and on multi-core processors, but only running on a single core (dual-core PowerPC MPC8641D, quad-core Intel Core i7).

However, with the VxWorks MILS Platform, Multicore Edition, the VxWorks MILS separation kernel now runs on all eight cores of the Freescale QoriQ P4080 processor concurrently. Yes that's right, I am talking about (MILS & multi-core) rather than (MILS XOR multi-core). This enables the integration of many MILS solutions on a single processor, creating reduced size, weight and power (SWaP) benefits - even high assurance security-critical systems have tough performance requirements.

The advent of multi-core processor architectures over the last few years has had a dramatic and disruptive effect on the embedded market. This has provided significant benefits in terms of reducing size, weight and power (SWaP), but has also presented some interesting challenges for safe and secure application domains.

For example, the global aerospace market has very mature processes for DO-178B safety-certification on single core processors. Similarly, the security community is approaching maturity for Common Criteria security evaluation on single core processors. But both segments are challenged when driving multicore systems through their respective processes.

There has also been an emerging trend in recent years of individual systems needing to meet both safety-certification and security evaluation, and now with the emergence of multi-core, this presents some new regulatory challenges, but the benefits will be worth the effort.

This convergence is discussed in more detail in the white paper 'Safety, Security and Multicore' which is available for download from the Wind River website.

For additional information from Wind River, visit us on Facebook.

Exciting Times - Virtually!

By Chris Ault

It's an exciting time for embedded virtualization at Wind River!  We were recently named the embedded virtualization market leader by VDC Research Group.  In VDC’s 2011 ‘Virtualization for Mobile and Embedded Systems’ report, Wind River achieved the highest share of the market in 2010 with greater than 30 percent of total market revenue.

In addition to this significant recognition, today we announced updates to our embedded virtualization offering. We released updates to our guest operating systems – VxWorks and Wind River Linux, as well as updates to Wind River Hypervisor and Wind River Workbench tools.

Why such fanfare? This synchronized release of integrated products and tools represents a unique advantage for developers working on embracing or developing targets based on embedded virtualization.

Some of the challenges developers face when working on virtualized embedded targets is the need to symbolically debug multiple operating systems. Yes, GDB is a great debugger, but it cannot be used for debugging the virtualization layer of the embedded target. It doesn’t have insight into the multi-OS communication constructs or shared memory regions on the target.

And development tools transcend debugging: developers need to be able to create system images that now contain multiple operating systems while retaining the ability to explicitly dictate the memory ranges, sizes, and the placement of the virtual machines within the physical memory range. I’ve written in the past that proper device Configuration is Key to Success with embedded targets. Having the ability to graphically view the hardware devices, and actually see the allocation among the virtual machines before the target image is built is a HUGE, in terms of speed of development and debugging.

 

Demonstrating our continued commitment to offer broad and complete hardware support, we have added ARM Cortex-A9 to Wind River Hypervisor's already extensive list of supported cpu architectures and hardware platforms.  This is particularly compelling for those developing small low-power controllers for example. Having the ability to support a virtualization layer on such small devices can yield very innovative designs and products, allowing for existing designs to remain intact while new functionality is partitioned and separated, but can be developed in a much more agile environment.

Another significant advance brought to the market in this release is an internal Ethernet switch. With such a switch, virtual machines internal to the device can communicate amongst themselves using standards-based protocols and drivers, greatly reducing development burden and time. But beyond that, in a world of increasing security concerns, imagine having the ability to embed a firewall or intrusion detection appliance directly within the device that is controlling the robotic assembler on the factory floor! Or directly within the set-top box hosting multi-media applications! This can greatly reduce costs in external networking gear while increasing device security. An internal Ethernet switch provides the ability to create multiple internal isolated networks, using standard protocols, and connect a subset of networks or virtual machines to external networks.

In this manner, it’s easy to connect a VxWorks virtual machine to a Windows 7 virtual machine. This allows manufacturers to retain the real-time characteristics of the VxWorks application and provide a graphics-rich Human-Machine Interface front-end to devices through a Windows 7 virtual machine.

I could go on and on, but I will reserve further discussion on the benefits brought by this release, and in embedded virtualization in general, to further blog postings. In the meantime, be sure to check out the product launch page for more information.

Embedded Virtualization Opens New Possibilities

By Mark Hermeling

One of the things that stands out in many customer visits that I do is that embedded virtualization often enables possibilities that the customer never thought possible. This of course is a good thing, it may enable them to put a new product to market that the customer had never thought of. In a few cases I have seen customers use virtualization to shift their business models from shipping custom hardware to shipping software only using COTS boards. Virtualization in this case is used to provide the customization that they formerly needed hardware for.

I went on a customer call earlier this year to a customer that was looking to go to multi-core. The customer was building a networking product and was using VxWorks on a single core Intel processor and wanted the additional processing power that a dual or quad-core could provide. They wanted to go to multi-core and came to Wind River to see what their options were. We discussed their business issues, came up with a set of requirements and decided that VxWorks SMP was their best bet.

The Symmetric MultiProcessing capability in VxWorks allows a customer to quickly utilize the additional cores, either by letting the operating system decide which process runs where, or by manually optimizing the configuration for whatever performance aspect is important to you (throughput, determinism, real-time response). 

The reason that we settled on VxWorks SMP was that the customer had already split their application into tasks, each tasks was very independent in processing and memory usage and there was minimal locking between the tasks. On top of this the customer did not need strong separation between the tasks. VxWorks SMP was the easiest and fastest migration path with minimal risk for them.

However, as part of the discovery process we also found out that this VxWorks based product (let's call it product A) works with another product B that uses Linux and there is an additional  management console C that uses Windows. Also as part of the discovery process we discussed embedded virtualization of course and how it could be used (for example) to combine products A, B and C onto a single quad core processor. This raised some eyebrows and there was some chatter between the participants, however at this point the company was selling each of them as independent products and they were okay with that. The immediate problem they needed to solve was to get their product A to multi-core.

Fast forward a couple of months. The company has been making great progress migrating their product A to a dual core processor. However, it turned out that the discussion on embedded virtualization had reached other levels of management level inside the company and they wanted to explore the possibility of introducing a new product combining A, B and C into a single box.

It turned out that the reason for the separation was scalability (as it often is). One management box C can be combined with several boxes running A and 2 boxes running B to build a particular configuration. The main market for these products was huge networking installations. However, for the bottom of the market this was complicated and expensive, a single box with a pre-integrated combination of A, B and C would open a new market segment of much smaller networking installations, something that without virtualization is either not feasible, or at the very least expensive to do.

Long story short, product ABC is now under development using a mixture of VxWorks SMP, Wind River Linux, Wind River Hypervisor and Microsoft Windows XP. The immediate target is a quad core processor, however, this can easily be extended to a higher core count product that includes multiple partitions running product A, a few B and a single C.

The funny thing (to me at least) is that we have a number of examples that are very similar in many other markets. The examples are the same in that the initial call with the customer is about migrating to multi-core, and how this can be easiest achieved and then morph into a combination of multi-core and virtualization to reach new markets to deliver new products.

It is a great proof point that indeed embedded virtualization opens up new possibilities and your Wind River representative is happy to talk to you about this to see how you can benefit from this exciting technology.

 

 

 

Virtualization on Handsets or not?

By Mark Hermeling

Here is an interesting article, continuing a discussion thats has been going on for a long time on mobile virtualization. It argues the case that a single mobile phone can run multiple operating systems using virtualization. A bit of googling will show that this is indeed already the case. However, there is a big caveat typically that you do not see exposed. 

Mobile phones have two big constraints: processing power and battery power. Typically one operating systems is the 'phone operating system' and the other OS is more general purpose. With 'phone OS' I mean that the OS has to handle a lot of the time critical tasks, sometimes even to the level of passing data off to/from the signal processing side for communication with the antenna and the wireless network.

The signal processing side is very time sensitive and hence it is important that the time constraints are met and data is delivered/picked up on the right time. This makes it that typically the virtualization layer will have to be aware of this and it makes the solution not a general virtualization solution, but a highly device customized solution.

The other solution (mostly used with Windows or Linux/Android type devices) is to just run everything on one OS. This of course works as well, but has limitations in the area of security, stability, upgradability and more. The biggest issue is that the phone side and the general purpose side run on the same kernel and in the same OS address space, this makes security much more difficult to guarantee. Also applying fixes to the OS to correct phone glitches may impact the general purpose side. It is all one big ball of spaghetti, difficult to understand and untangle when needed.

My vote is on the virtualization solution, the concepts mentioned above hold for not just mobile phones, but virtually (pun intended :) any device in the market today. Whether it is a multifunctional printer, a network switch, industrial control, automotive, aerospace and defence or another application, virtualization can help put multiple 'personalities' on your device for the various tasks that you may have to do (from real-time and safe&secure to general purpose).

Virtualization can help here, but it will have to be embedded virtualization, with that we mean that the virtualization layer can be optimized for a particular device and that the device-builder has all the tools to fine-tune the virtualization layer to the device, OS and application he is building. With fine-tune we typically mean the fine-tuning of sharing and/or partitioning of processing cores, memory and devices. Embedded virtualization provides the flexibility and the tools to perform this fine-tuning, for an example, have a look at the Wind River Hypervisor.

 

Migrating to Multi-core using Embedded Virtualization

By Mark Hermeling

So, let's assume that you are currently on a single-core board and you would like to explore the possibilities of delivering a next generation device using multi-core technologies. There are a lot of different ways to approach this. The most elaborate involve doing an architecture study into ways in which you could evolve your current device architecture. This study would need to be approached from both a technical, as well as a business level.

The funny thing with software is that 'virtually' anything is possible, but the trick is to find approaches that are not only technically possible, but that provide business benefits, usually in a short time-frame. Part of the equation is the amount of work required to build this next generation of device.

In the 'olden days' people would spend a lot of time deciding about which hardware to use for a project at the beginning of that project. Once the hardware is decided, the software is next.

Well, embedded virtualization is one of the leading technologies to support the migration to multi-core and embedded virtualization is a bit like hardware and a bit like software. There are three variables to decide on when planning to use embedded virtualization: how to partition or virtualize cores, memory and devices. And that is immediately where the complexity lies, there are many degrees of freedom. Doing an architecture study at the beginning of the project helps you define which parameters you are interested in such that you can then explore the proper degrees of freedom with a proof of concept for example.

So let's assume that the study has been done and we know what direct we want to go into. Now it is time to explore and do a proof of concept. Well, let's also assume that you are targeting an architecture that has hardware assist for virtualization. Examples of this are Intel processors with VT-x, or Freescale processors with the e500mc core, or the upcoming ARM Eagle. Well in these cases life is relatively easy. One can run unmodified guests, meaning unmodified operating system kernels, and run them directly on the new multi-core hardware on top of a hypervisor in a virtual board (also known as a virtual machine). In this case one would directly assign all devices directly to this one virtual board and the software will run as in a native environment, but now run on a virtual machine.

There may still be some changes required to the BSP, for one, the devices on the new platform may be different for example. However, this is by far the easiest and straight-forward way to migrate to a new multi-core based platform. The next step is to use the additional processing cores, memory and devices to create new content.

It is easy to add another virtual board with a new operating system, maybe add Linux to an already existing real-time operating systems. At this point, maybe one or more of the devices need to be shared or virtualized and maybe the new operating system needs to be connected to the initial virtual board using some form of IPC possibly.

Once this is done it is time to start building new content and to experiment with the new, multi-core environment based on the parameters laid down in the original study.

Now, let's take a step back and assume that we are not targeting a processor with hardware assist for virtualization. In this case, we can still do the exact same things explained above, but with the difference that we can not use unmodified operating systems inside the virtual boards, instead we need paravirtualized the operating systems we intend to use. Any operating system can be paravirtualized (the source needs to be available to make the changes though), and the Wind River operating systems (VxWorks and Wind River Linux) are already paravirtualized to run on top of the Wind River Hypervisor.

The rest of the workflow remain the same, we have to be a bit more careful with regards to device drivers. The operating system, the device drivers and the hypervisor have to collaborate to deliver the functionality of the system. There is a bit more work and coordination involved, but getting a basic software architecture up and running is still fairly straight forward.

All in all, there are well described paths to migrate a single core system to a multi-core, or combining multiple single core systems onto a multi-core. The true challenge comes when resources (such as devices, cores and memory) need to be shared, but I will leave that discussion to a later blog post.

Following these paths provides a quick way to evaluate how existing designs would run on top of an embedded virtualization layer in a proof of concept. Going through the process of making a proof of concept not only validates the findings from the architecture study, but it is also a learning experience for your software development teams in preparing them for the new world of multi-core.

Embedded Virtualization in Gaming

By Mark Hermeling

A few weeks ago I found myself in the Hard Rock Hotel in Las Vegas. My reason for being there was a seminar that Wind River was organizing together with Intel and AVNET specifically focussed on companies that build gaming. The main topics were graphics, manageability and of course, virtualization.

I am always eager to fold real-life examples into my presentations, hence I decided to try my luck at some of the slot machines in the Hard Rock Hotel under the disguise of 'domain study'. And I have to say, the state of slot machines has progressed significantly since the last time that I was in Vegas many years ago. The top of the line machines had multiple 30" inch displays with either static or moving imagery, they had player tracking, surround sound, some of them even had imagery over the rolls themselves.

Since this was a domain study, I started thinking what type of hardware would be running in these slot machines. Off the top of my head I could come up with the following sub-systems:

• Money collection: The part where you insert your dollar bills
• Money return: much less used, but important nonetheless
• The game: This is the part that is the most fun
• Machine management: The part that reports on whether the machine is doing okay, that connects money collection, return and the game to the central casino management
• Player tracking: where you slide your card and it keeps track of your preferences, and feeds your gambling habits back to the casino for perks. This is also where (in a more advanced scenario) you could order drinks, make reservations for restaurants and stuff.

So that is 5 separate subsystems and I expect that almost all of them run on their own dedicated hardware. That is 5 processors, memory, power supplies and so forth in that big box. Now, the slot machines are fairly big, so I don't think that space is an issue, but heat dissipation could be and maintenance certainly is. I was talking to another customer and they were using a number of 6% as the yearly failure rate of a single board. The higher the board count, the higher the potential of failure of course.

Technically I could integrate all of these 5 different subsystems in a single multicore chip using embedded virtualization. Each subsystem would run it's own OS inside of a virtual machine. The OS may be a real-time OS, or a general purpose or graphics OS such as Windows. The game would be the biggest compute power user, so it would need a dedicated core, possibly 2. The other parts are much smaller and can probably all share (time slice) a single core. From a device assignment, most of the devices would all be partitioned between the different virtual machines, the one exception possible be graphics. The virtual machines would also need to communicate between each-other which is very easy to set up within the multicore environment.

One thing to keep in mind is that these virtual machines will run at different levels of certification. The money collection machine for example may need to interact with credit card verification back ends, the game of course needs to be certified by the gaming commission and such. To this extent the embedded virtualization layer needs to be certifiable to at least the most stringent certification level of each of the subsystems.

Virtualization of the display is a fun topic. In principle it is not difficult, however, in this case there is a bit of added complexity with regards to the certification. It needs to be very clear to the user when he is gambling, versus when he is looking at player tracking, venue reservation or generic web-browsing and the data entered should never been able to be leaked into the other application. For instance, when the user needs to input credit card data for venue reservation, that should not be usable by the game or by the generic web browser, or very bad situations can develop.

The 'domain study' resulted of course in the casino making some money off me, but I had sufficient content to talk about in the seminar the next day. 

Wind River Hypervisor 1.2, which was officially announced last week, provides support for 'unmodified guests', meaning that one can run Microsoft Windows, a 3rd party operating system, or an in-house operating system unmodified on processors that support hardware assist for virtualization (such as the Intel P4080, or Intel processors with VT-x). This is a main enabler for consolidating multiple boards on a single multi-core processor, which is of course not limited to the gaming industry highlighted in this post, but also much desired in industries such as Networking, Industrial Automation, Medical, Automotive. Aerospace and Defence and so forth.

 

Embedded Virtualization and Multi-Core

By Mark Hermeling

VDC has released a recent report showing that multi-core is making fast inroads in embedded systems and we are seeing that at Wind River as well. There are exciting new possibilities with these chips, we have never had this much processing power in such small packages.

Take for example Freescale's new P4080 chip, 8 cores, 2 memory busses and a wealth of peripherals that are designed for multicore usage (such as the data path acceleration architecture (DPAA)). There are many things to consider with this new chip, I have blogged before about the many primary multi-core configurations available from SMP to uAMP, to sAMP to core virtualized. You have to consider partitioning, or virtualizing cores, memory and devices.

This may seem straight forward, but there are many things to consider. As mentioned, the P4080 has two memory busses, all cores have access to these busses at the same cost, however, if you run all cores with memory-heavy workloads on the same bus, you may impact performance, both from a throughput perspective as well as from a real-time and determinism perspective.

Many customers are wondering why, when and how to make the jump from multi-core. They are looking for information, recommendations and experience. Luckily there is help around.

First of all, there is a webinar this Wednesday, Oct 27 at 2pm ET with Curtiss Wright, Freescale and Wind River in which the three parties provide an overview of the P4080 and its possibilities on Curtiss Wright boards.

Secondly, Wind River has today announced the immediate availability of a new version of the Wind River Hypervisor with support for the P4080. Support for a many-core chip like the P4080 in the embedded space requires a hypervisor custom built for scalability and with embedded virtualization in mind. You can’t target this architecture by using an IT virtualization layer or by retrofitting an RTOS to do virtualization. Wind River’s support for the P4080 of course does not stop at the embedded virtualization layer, it extends to VxWorks, Wind River Linux, Wind River Simics and Wind River Workbench. The entire arsenal of products are available to get software our as fast and correct as possible.

Lastly, Wind River has released the Wind River Multi-Core Adoption Package, where multi-core experts from Wind River can work with your team to discover the business case for multi-core with your team and build a list of options as to how you can step into the multi-core era. The goal of this package is to firstly provide you with a list of options within the parameters that were discussed with your team on how you can best utilize multi-core for your business. If you decide that one of these options is the best choice for you, then Wind River is happy to assist you with to assist you with the implementation as well as a second step.

The multi-core disruption is an exciting time to start reconsidering how to architect your next generation of embedded devices.

 

Devices! Devices! Devices!

By Mark Hermeling

That is what embedded systems is all about: handling devices. An embedded software system has input devices and output devices and some amount of software running on operating systems, running on hardware that turn input into output. The trick is to handle the input in a fast an deterministic way and generate output in a fast and deterministic way. Determinism is important all the way through the system, from the device, to the hardware, to the operating system to the software running on the operating system.

Add virtualization to the picture and quickly understand that not all virtualization layers are created equal. Especially in the area of IT-virtualization the focus has been on abstraction of the physical environment, which includes devices. This means that often the software does not have access to the physical device, it has access to a device-driver inside the hypervisor that 'owns' the device.

And abstraction quickly impacts determinism and real-time aspects such as latency. A true embedded Hypervisor gives the system designer full control over how he wants to handle his devices such that the designer can minimize the impact in the areas where determinism is crucial for the embedded device.

A true embedded hypervisor (such as the Wind River Hypervisor) provides a device management infrastructure that provides multiple options as to the handling of devices:

The first and most simple option available is to partition. That is, given a set of devices, partition them between the virtual boards running on the processor. Say your hardware has 2 PCI busses and 2 virtual boards, give one PCI bus directly to each virtual board and assign the devices on the bus to the virtual board. Now you have a nicely partitioned system and the hypervisor does not need to intervene, hence there is absolutely no impact to the determinism in your system.

The second option is to share, assign the device directly to one of the virtual boards and the operating system in that virtual board owns the device. The software in that virtual board has direct access to the device and hence no impact to the determinism. If another virtual board needs access to the device you can export it through a inter-board messaging protocol. The second virtual board has access, but it is slower and non-deterministic. If this is okay, then we have an easy solution.

The third option is to virtualize the driver into the hypervisor. This is where we are starting to add a layer of delegation into the device handling path, so we need to make sure we write the driver carefully, consider core-interaction in the design (driver running on core 1, virtual board that uses it on core 2, hence more inter-core communication) and make sure that we keep our worst-case execution times in check. We also have to consider multiple users (which is why we are using virtualization, if there was only a single user we could use partitioning) and how it impacts our performance.

Give your Wind River representative a call if you are planning your next generation of embedded system and he can help you design a way to include virtualization into your system without impacting the determinism you require.

Type 1 versus Type 2 Hypervisors

By Mark Hermeling

I have always found the difference between Type 1 vs Type 2 hypervisors rather uninteresting. In short, a type-1 hypervisor is a hypervisor that has direct access to the hardware, where a type-2 hypervisor executes inside an operating system. Most hypervisors are type-1 hypervisors, including IT hypervisors such as VMWare, Xen, KVM and such. Type-2 hypervisors are applications like VMWare Workstation (or Fusion), Parallels, .... The distinction between type-1 versus type-2 is really not as useful as most people think. There is a good blog article by Anthony Liguori that describes this as well, complete with references, the article is a bit old, but still correct.

VxWorks in Education: University of São Paulo, Brazil - Part 3

By Bill Graham

In the third and final part of my interview with Professor Glauco Caurin we discuss multicore and virtualization and why they are working with us on their research projects.

Q: Are your students learning about multi-core processors  and programming? What about multi-OS systems including virtualization, i.e.  systems that have more than one OS on a single processor?

We are not teaching our students yet about the programming of multicore processors. But we have experience in the past with multiprocessing systems programming using X-METH.

It is in my medium-to-long term plan to insert multicore programming into our courses. It also depends on the time we take to have enough students that are skilled with the "basics".

Q: Are you  starting to use or design with multicore processors in your projects? Do you  think multicore and multi-OS systems are an important part of the  curriculum?

Regarding virtualization, we did some initial tests (not in real time) using virtualization and different OS for Human Machine Interaction. I am really impressed with this possibilities,

 I am really interessted in the future applications of this promising technology.

Q: Prof. Caurin, why do you choose Wind River as a partner in your courses and projects? What are some of the characteristics you were looking for in our software?

As I described, we have been working with RTOS since the 90’s especially with systems that were proposed in academic context at ETH (Swiss Federal Institute of Technology) like XMETH, XOberon, JBed and so on. We have also developed our own open source version of a Linux patch for Real Time applications here in Brazil. But my concern with the students and the research projects were related to their continuity [of software, OS, tools]. How is it possible to convince a new student to use a specific IDE? And further how could we take advantage of the work done previously. 

Every problem detected was solved with a new RTOS version [in-house developed]. Not rarely, the new version had commands and characteristics that were incompatible with the previous versions. Additionally, each new PhD student wanted to test a different open source RTOS, or even worse, some were suggesting they could make a better OS. 

As a consequence we were always starting from zero with every new robot project. 

Good results were achieved using Wind River Products and Tools, so we decided to focus on this software, and since then all the classes, students and researchers are invited to use them as a standard. We increase the information exchange between the students and our productivity. The availability of manuals and documentation was a strong point in our decision. I believe now we are able to propose and conduct a larger number of research projects in parallel.

Q: Do you plan to continue to use Wind River Products for teaching and research? Are there any interesting future projects or ideas you can share? 

Yes we are strongly investing in training human resources that will work in the next 5 years at our Lab. We are very excited with the possibility of new partnerships with different Universities in the USA. For example, the new project that we are starting together with the Newman Laboratory for Biomechanics and Human Rehabilitation at MIT. They have a lot of experience combining robot assistance and video games to motivate and improve stroke patients rehabilitation process. 

- end of interview - 

As you can see, there are some impressive things going on in the engineering program at USP! I wish to thank Professor Caurin for taking the time for this interview and for taking full advantage of our Wind River University program. If you have some exciting teaching and research going on with Wind River products let me know. If you are interested in using Wind River products in the class room and lab, take a look at our Wind River University Program.