Multi-core & Embedded Virtualization

Embedded Virtualization in Gaming

By Mark Hermeling

A few weeks ago I found myself in the Hard Rock Hotel in Las Vegas. My reason for being there was a seminar that Wind River was organizing together with Intel and AVNET specifically focussed on companies that build gaming. The main topics were graphics, manageability and of course, virtualization.

I am always eager to fold real-life examples into my presentations, hence I decided to try my luck at some of the slot machines in the Hard Rock Hotel under the disguise of 'domain study'. And I have to say, the state of slot machines has progressed significantly since the last time that I was in Vegas many years ago. The top of the line machines had multiple 30" inch displays with either static or moving imagery, they had player tracking, surround sound, some of them even had imagery over the rolls themselves.

Since this was a domain study, I started thinking what type of hardware would be running in these slot machines. Off the top of my head I could come up with the following sub-systems:

• Money collection: The part where you insert your dollar bills
• Money return: much less used, but important nonetheless
• The game: This is the part that is the most fun
• Machine management: The part that reports on whether the machine is doing okay, that connects money collection, return and the game to the central casino management
• Player tracking: where you slide your card and it keeps track of your preferences, and feeds your gambling habits back to the casino for perks. This is also where (in a more advanced scenario) you could order drinks, make reservations for restaurants and stuff.

So that is 5 separate subsystems and I expect that almost all of them run on their own dedicated hardware. That is 5 processors, memory, power supplies and so forth in that big box. Now, the slot machines are fairly big, so I don't think that space is an issue, but heat dissipation could be and maintenance certainly is. I was talking to another customer and they were using a number of 6% as the yearly failure rate of a single board. The higher the board count, the higher the potential of failure of course.

Technically I could integrate all of these 5 different subsystems in a single multicore chip using embedded virtualization. Each subsystem would run it's own OS inside of a virtual machine. The OS may be a real-time OS, or a general purpose or graphics OS such as Windows. The game would be the biggest compute power user, so it would need a dedicated core, possibly 2. The other parts are much smaller and can probably all share (time slice) a single core. From a device assignment, most of the devices would all be partitioned between the different virtual machines, the one exception possible be graphics. The virtual machines would also need to communicate between each-other which is very easy to set up within the multicore environment.

One thing to keep in mind is that these virtual machines will run at different levels of certification. The money collection machine for example may need to interact with credit card verification back ends, the game of course needs to be certified by the gaming commission and such. To this extent the embedded virtualization layer needs to be certifiable to at least the most stringent certification level of each of the subsystems.

Virtualization of the display is a fun topic. In principle it is not difficult, however, in this case there is a bit of added complexity with regards to the certification. It needs to be very clear to the user when he is gambling, versus when he is looking at player tracking, venue reservation or generic web-browsing and the data entered should never been able to be leaked into the other application. For instance, when the user needs to input credit card data for venue reservation, that should not be usable by the game or by the generic web browser, or very bad situations can develop.

The 'domain study' resulted of course in the casino making some money off me, but I had sufficient content to talk about in the seminar the next day. 

Wind River Hypervisor 1.2, which was officially announced last week, provides support for 'unmodified guests', meaning that one can run Microsoft Windows, a 3rd party operating system, or an in-house operating system unmodified on processors that support hardware assist for virtualization (such as the Intel P4080, or Intel processors with VT-x). This is a main enabler for consolidating multiple boards on a single multi-core processor, which is of course not limited to the gaming industry highlighted in this post, but also much desired in industries such as Networking, Industrial Automation, Medical, Automotive. Aerospace and Defence and so forth.

 

Primary Virtualization Use Case

By Mark Hermeling

This topic invariably comes up when talking to customers, unfortunately, there is not just one, but several primary use cases. There are multiple ways to look into the various use cases. The one I like best is to look at generic drivers. An alternative is to look at actual usages in the various industries.

Bill-Of-Material

By Mark Hermeling

Bill-of-material is something that is important in many devices and I have argued before that virtualization can help with this. Say you have a medical device like an MRI scanner, it is not uncommon that this device has three processors that collaborate. One processor, often in a separate box like an industrial PC, runs MS Windows and is the operator interface, possibly with touch-screen GUI. This is where the results are displayed.

What a month!

By Mark Hermeling

I have been back in Ottawa now for just over a week after my 5 week trip through Europe and am working through my notes from the past month. And what a month it has been, in an earlier post I jokingly declared 2010 the year of embedded virtualization and this is certainly ringing true.

Blog Article: Boundaries are disappearing

By Mark Hermeling

Ok, so I got a little bit of flak verbally, through email and Skype for unilaterally declaring 2010 the year of embedded virtualization. People seem to agree though, it is a hot technology, and a technology that can change the way that we develop embedded systems.

This post by George Zimmerman talks about how integration of technologies leads to inflection points in the adoption of new technology. And how the 'boundaries between computing devices are disappearing ... because of a combination of processing power, including multiple cores, virtualization, and network connectivity'.

These are all topics at the front of the minds of the customers I talk to, whether they build the devices that power the network or whether they build devices that connect to the network. From the place I sit I see a lot of excitement about new capabilities for existing devices. I think this excitement will lead into new development projects this year and devices in the market a while after that.

2010 Is The Year Of Embedded Virtualization

By Mark Hermeling

I believe that 2010 will be the year of embedded virtualization, all the signs point in the right direction. It always takes a while for new technology to grab the imagination of embedded device developers. Embedded developers are traditionally a conservative bunch, however, the benefits of virtualization can not be ignored, even by them.

Article: Multi-Core Slow Down

By Mark Hermeling

An interesting article by Dan Woods on Multi-core slowdown. The article tries to temper people's expectations with regards to multicore. The basic argument goes: A multicore processor has more raw processing power, but it requires the software load that runs on top of that processor to be able to use those cores, if not, the software could run at the same speed as single-core, or even slower.

One of the ways to use all the cores of course is multi-threaded programming in combination with an SMP operating system that can schedule over all the cores (SMP being Symmetric Multi Processing). Typically multi-threaded programs use multiple threads of execution and use synchronization primitives to make sure executions happens in the right order. The last thing you want is 2 threads 'simultaneously' modifying the same data structure. The thing is, one a single core processor, there is no true parallelism, the operating is pretending to do multiple things at once. When you introduce multicore processors you will have true parallelism, this may lead to problems.

One problem occurs when the application uses a lot of synchronization and the result is that the cores are continuously waiting on each other. You are now executing a largely serial application on a multicore part and Amdahl's law will dictate that you will have very little speed-up from the use of multiple cores.

Another possible problem is that your application does not use sufficient synchronization and the result is that the application will do things truly in parallel that were never meant to be executed in parallel. This often leads to data corruption and hard-to-find faults. Not a desirable situation either.

The statements in the article have more of an IT focus, however, they hold for embedded as well. Multicore could lead to a slow-down if the application is unable to use all this new processing power. However, in most cases there is a way to use the multiple cores and to make a difference, either in performance, or in the set of capabilities of a device.

One of the important points to consider when looking at multicore in embedded are the multicore software configuration. The most simple of these multicore configurations uses SMP which allows a single application to use multiple cores, if that application is written for it, as mentioned before. This seems to be the main focus of the original article.

However, other configurations allow customers to run multiple different operating systems on top of these multicore processors. There operating systems could be of the same type (say, VxWorks), or other types (say VxWorks and Linux). Each operating system runs in isolation and will use some of the raw processing power of that processor. Every operating system can of course run one or more applications.

Now, you may ask, why do I need to run multiple operating systems, can I not run multiple applications in a single SMP operating system? Yes you can, however, there are many situations where a single SMP OS is not the best solution because of safety, security, separation, robustness, performance or scalability reasons. Running multiple operating systems gives you, the user, better control as to how to use the power of your multicore processor.

One straight forward example of the use of multiple operating systems is that it allows a customer to use some of the cores in a multicore processor to add more functionality to an existing product using a different operating systems. For example, graphics capabilities using Linux added to the real-time control for an industrial application, or networking to a consumer device. This capability can provide benefits in many different industries, for example consumer applications, network switches, industrial applications, aero-space and defense, the examples are many.

Running more operating systems provides more flexibility such that you can provide more features or more performance to your customers and through that deliver a more competitive, higher value product.

Breadth and flexibility here are important. There is no silver bullet. SMP, uAMP, sAMP, virtualization, all are configurations in the tool-box of the embedded developer and can be used in many different combinations and situations.

There are a a couple of challenges that become immediately apparent:

• The first challenge is to use the right configuration at the right time (a screw driver does not work well if you need a hammer)
• The second challenge is to make sure that everything is integrated, the operating systems have to be optimized for the virtualization layer as well as the silicon
• The last, but certainly not the least is that with these additional configurations, we still need tools for development, analyze, debug and test, that need has not gone away

To summarize: a slowdown when using multicore is certainly possible, but there are ways to solve that slowdown:

1. Make your make your application aware of the multiple cores
2. Using the multiple cores to do new things, possible with new operating system

Primary Multicore Software Configurations

By Mark Hermeling

Many people ask the question as to what the best approach would be for them to go to multicore and/or virtualization. This is a great question to start a discussion as there is not a single silver bullet. I meant to post a quick diagram on the different multicore configurations before, but life has been busy since we announced the Wind River Hypervisor earlier this year. Busy in this case is certainly a good thing.

Addressing Core Issues

By Mark Hermeling

A good article from colleague Jens Wiegand on multicore in medical devices. Jens talks about consolidation and innovation, two driving factors in both medical and industrial devices. However, he also points out the flip side of the coin: certification. Virtualization can help provide strong separation on multicore, which makes certification manageable (the article goes into more depth).

Jens also touches on tooling for multicore development, an often-overlooked and under appreciated aspect. A single development environment that can be used to develop the entire device (real-time, UI, kernels, userland) as well as drive testing and debugging is a must to create highly efficient development teams.

Multicore and Unicore Virtualization

By Mark Hermeling

Sometimes I get a chance to browse the web a bit, trying to find interesting tidbits. I ran into the following post from Hollis Blanchard. Hollis is an active participant in the world of virtualization, particularly on the Power architecture.

Hollis makes a point that I have been trying to get out as well: virtualization is clearly technology to look into when you are building multicore systems, but also when you are building unicore systems.