Open Virtualization on Simics, Making Complex Work Easier – an Interview with Michael Barabanov

Having the right tools can make any job easier. In this interview, Michael Barabanov from our Wind River engineering team describes how by using Simics he approached some engineering issues differently, and was able to streamline his work with Wind River Open Virtualization and oVirt. The Wind River Open Virtualization is a virtualization solution for embedded systems built on Wind River Linux and an enhanced KVM. Systems built on Wind River Open Virtualization easily scale out to tens of virtual machines spread across multiple Intel-based multicore hosts, and working with such a system using only hardware can be rather painful. However, as Michael describes in this interview, when your teams have the tools they need to work on complex projects, they can be successful in ways you may not have thought of previously.

JE: Please introduce yourself!

MB: My name is Michael Barabanov, and I’m a Senior Member of Technical Staff at Wind River.  My background is real time systems, hypervisors, Linux and Android.  Presently I’m a member of the Wind River Open Virtualization team.

JE: How did you first encounter Simics?

MB: If I remember correctly I first used it for debugging an early version of the Wind River Hypervisor circa 2007. I was immediately impressed with Hindsight and even wrote a couple of context trackers in Python.

JE: Writing process trackers is pretty advanced usage, I think you are one of the few who ever did that. I guess it was in order to track the Wind River Hypervisor for debug?

MB: That’s right; I needed to differentiate the hypervisor and the guest execution contexts to be able to debug both at the same time.

JE: But Hypervisor and process trackers are not the main topic of the day. What we want to know is what you have been doing with oVirt and OVP?

MB: Currently, I use Simics to simulate a network of multiple machines that constitute a cloud environment based on oVirt virtualization infrastructure.  Simics helps me to debug pretty complex software running a network cluster, as well as demonstrate various use cases to others.

JE: What was the goal of building a Simics setup that runs oVirt and Wind River Open Virtualization?

MB: When there are multiple developers working on a project, making sure everyone has an available hardware target can be challenging. This is doubly (triply?) so if we talk about cloud software development and debugging.  Originally I wanted to be able to debug the whole cluster on my laptop.  I started with libvirt and KVM, but a nested virtualization setup was a challenge there. Simics support for precise emulation of timing and virtualization extensions made the difference.

JE: What did the setup look like?

MB: The basic setup is as shown below: a single oVirt engine node, which presents a web-based GUI to a web browser, is coupled with multiple oVirt nodes that run virtual machines. The web browser client is run on the host rather than inside of Simics, since there is little value in simulating that component of the system.

When it is running, it looks like this on my Linux host machine (please click to enlarge). Note the web interface to the oVirt engine in the middle, and the Simics Eclipse GUI running behind it showing the list of Simics target machines in the “Target Info” view on the right. The Simics checkpoints (as discussed below) are listed on the left.

JE: Are there any Simics features that helped you build this setup?

MB:  Simics hypersimulation technology made it faster to get oVirt up and running, since Simics could skip over the many programmed waiting times and time-outs in the system.

I used checkpoints to save the configured state at interesting points during the setup process, avoiding the need to manually repeat the setup actions or having to write a script.  Checkpoints were taken at point like “just after oVirt engine setup”, “just after oVirt node registration”, “just after storage domain creation”, “after VM creation”, and similar points of interest in the system setup history.

Simics integration with pcap allowed me to inspect network communications between multiple cooperating machines. Thanks to Simics deterministic nature, I could reliably capture execution trace of the software in the cluster and then use wireshark and other tools to understand what happened during a particular run.

JE: That is really nice. Is there anything more you can tell us about your use case?

MB: The simulated setup was also used to validate that the Wind River Linux and Open Virtualization implementation and oVirt setup was compatible with the upstream code. To do this, we also prepared some installations of Fedora 18 which were run in Simics alongside the Wind River Linux Open Virtualization nodes in various configurations, as shown below.

With this setup, it was easy to validate the behavior of the Wind River setups compared to the reference upstream configurations.

The flow went like this:

In each step, I changed one software component, thus performing a stepwise validation of the setup. If something was discovered to be broken or odd, it was easy to go back to the step before and do the same thing, comparing the behaviors. With simulation, running the two setups side by side is really easy. Trying to do the same using physical hardware would entail reinstalling the software stacks on the computers used, or having one physical computer available for each type of software stack that was being used. This is possible, but not very efficient.

JE: Did you use the Simics debugger for any part of your work?

MB: Not for this particular project, but yes I’ve used it before, for example to debug Wind River Real Time Core components that ran in the context of the Linux kernel.

JE: What can you tell us about your future plans for using Simics?

MB: I can’t share too many details, but I expect to continue to use Simics for simulating even more complex multi-system and multi-network setups, as well as debugging the kernel and userspace on individual nodes.

JE: Thank you for your time, this has been really interesting.

MB: Thanks Jakob.

Note. This blog post shows just one example of how we at Wind River use Simics ourselves to work more efficiently. Another example is how Simics is used to teach networking and device driver development (see also Chapter 10 of Simics Unleashed – Applications of Virtual Platforms ). For more on how you can simulate networks with Simics, please see my previous blog post.