Tools, Testing, & Virtual Systems

Systematically Exposing OS Kernel Races – An Interview with Ben Blum

By Jakob Engblom

Full-system simulators like Simics provide unparalleled insight into what is going on in a target system. Indeed, better insight is one of the main features of simulation that we get regardless of what we simulate and how. In addition, if we want to, we can also exert control over the target system to make it take different execution paths than it otherwise would. Earlier this year, Ben Blum at Carnegie-Mellon University  CMU presented a Master’s thesis that provides a very good example of just what can be achieved by combing the insight and control of a simulator with intelligence and domain knowledge. The system is called Landslide, and it is used to expose race conditions inside of operating-system kernels.

Landslide systematically explores the possible execution paths of the kernel in order to provoke latent bugs that might only happen very rarely in actual use of the operating system – but as we all know, such glitches are the ones that tend to be found by customers in critical situations and force engineering to spend months in reproduction attempts.

For this blog post, I interviewed the creator of Landslide, Ben Blum. 

Jakob Engblom: Please introduce yourself!

Ben Blum: I'm Ben Blum, a graduate student in the Computer Science Department at Carnegie Mellon (CMU) , advised by Garth Gibson. I spent last year doing this work as part of the Fifth Year Master's program in the Computer Science Department at CMU. Now I am staying on for several more years for a Ph.D. 

JE: What was the topic of your thesis?

BB: Systematic testing (or sometimes "exploratory testing" or "systematic exploration") is the idea of testing for race conditions in a concurrent system by forcing the system to execute as many different thread interleavings as possible.

Normally when you run a concurrent system, its threads run in an unpredictable pattern (governed by timer events or other device interrupts), and only one possible interleaving out of many is executed. And (by contrast), when you use conventional stress testing, you exercise a random set of thread interleavings, often a very small subset of the whole execution space. A systematic testing tool tries to enumerate and explore the whole state space of interleavings.

This approach is very powerful: it can find arbitrarily complicated race conditions, including TOCTTOU (Time of Check To Time of Use) bugs. Lighter-weight tools such as data race detectors or static analyses are often powerless against these, since simply adding locks around each access might satisfy such a tool but not actually solve the race.

But systematic testing is also very computationally expensive – you might be thinking "exponential explosion of possibilities", and you'd be right. A big part of the research involves figuring out how to reduce the coverage (and/or granularity) of possible interleavings in a way that still gives meaningful results. Sometimes we use sophisticated pruning algorithms; sometimes we use simple heuristics.

JE: How did you use Simics?

BB: I wrote my tool, called Landslide, as a Simics module. When Simics boots up a kernel, it also loads Landslide, which then gets to monitor and control the kernel's execution.

Being in a simulated environment gives a lot of power: Landslide sees every instruction and memory access the kernel runs, and so keeps fairly detailed internal representations of the kernel's state. There is about an order of magnitude slowdown in doing this (compared to regular Simics simulation speed), but it gets to know (for example) when each kernel thread is runnable and whether heap accesses are invalid.          

JE: And just what is Landslide and how does it work?

BB:  During the first run of a test case, Landslide identifies "decision points" - instructions during the execution at which it thinks that if the kernel had context switched to another thread, something interesting might have happened (the set of decision points defines the state space of thread interleavings). Annotations in the kernel (which I call "tell_landslide()") also help Landslide track what the kernel is doing (especially thread lifecycle events).

Then, when the test case finishes, Landslide chooses which decision point to execute differently (and chooses which thread to run instead), rewinds to that state, injects timer interrupts to force the kernel's scheduler to run that thread, and proceeds. (A pruning algorithm called Dynamic Partial Order Reduction drives the exploration - it chooses which interleavings to explore next in a way that skips over certain redundant ones.)

Simics has a really great feature that makes this possible: it supports setting bookmarks when the simulation is stopped and later doing reverse-execution back to that point. I used this to implement the "rewinding", and it turns out to be a lot faster than rebooting the kernel every time (especially in a depth-first search).

JE: Cool, so you actually make use of reverse execution to create a backtracking search through the state space of the target system. Kind of obvious once you see it, but also very clever.  Anyway, please continue.

BB: Finally, when Landslide finds a bug, it prints a "decision trace", which is a list of each thread switch that happened and a stack trace at each point. Landslide has a couple of bug-finding checks: tripped asserts, use-after-free heap accesses, deadlocks, and (heuristically) livelocks /infinite loops.

 

Which events to consider "decision points" was also a hard question. During the project, I used voluntary reschedules (such as calls to yield()) and synchronisation operations (such as calls to mutex_lock()/mutex_unlock()) as decision points. I felt this struck a good balance between fine-grained interleavings (more likely to find races) and coarse-grained interleavings (more feasible to explore the state space).

But I also wanted to harness the user's own intuition, to focus the search space more intelligently. So the user can also configure decision points of their own - they can write "tell_landslide_decide()" in the kernel's code to say "hey, put a decision point here!", and they can also use another config file to whitelist/blacklist certain modules of the kernel (maybe you don't want to waste any time exploring interleavings in the virtual memory subsystem).

 

---

Partial-Order Reduction Explained

Landslide uses an existing algorithm called Dynamic Partial Order Reduction to prune redundant subtrees out of the search space of different thread interleavings. The basic idea of the algorithm is to use a "memory independence relation", which identifies when certain state transitions of different threads have no conflicting shared memory accesses, to know when it wouldn't matter which order those transitions run in. In the picture above, the states are identical after "1" and "2", regardless of the order that these operations are done. Therefore, it is sufficient to explore only of the subtrees marked in yellow.

---

JE: That sounds like some pretty advanced Simics programming there (I would not call it scripting, it is way beyond that level of sophistication)!

BB: Thanks! Yes, Landslide is basically its own piece of software. It's about 4000 lines of code, written mostly in C.

JE: What types of kernel errors do you detect with Landslide?

BB: Well, in theory, the beauty of systematic testing is that it should be able to find any kind of race condition.

One example of a race that systematic testing can find that a data race detector couldn't is the everyday TOCTTOU: let's say the kernel validates a userspace buffer (with a lock held) then later accesses the buffer, but in the meantime another thread changed the permissions on that memory (with the same lock held), making the first thread's access fail. Because of the locking, it's not a data race, but Landslide can easily force an interleaving when the first thread drops the memory-validation lock (cf. mutex_unlock() being a good decision point, as I noted above).

Landslide itself is a little more limited than "in theory", of course. As implemented, the only source of nondeterminism it controls is the timer interrupts; so it controls thread scheduling, but not device input handling. As such, it's most useful for finding races in core or mostly-core parts of the kernel, such as the thread lifecycle implementation or the virtual memory system. Finding races in device drivers is beyond the scope of Landslide's current model (though potentially in the path of future work).

JE: Are you still working on the tool?

BB: Since I wrote my MS dissertation on it, I've put it on hold for a while. I've been doing some other concurrency-related projects in the meantime (which you could read about if you want on my website, linked above).

JE: What kinds of code have you tested Landslide on?

BB: I wrote Landslide targeting "Pebbles", which is a UNIX-like kernel architecture that students implement in six weeks in our undergrad operating systems class at CMU. Pebbles kernels have to be fully preemptible and support concurrent execution of standard thread-lifecycle events (fork, exec, exit, wait, yield, etc.). Using Pebbles as a case study was a good proving ground for potentially extending this work in the future to more mainstream kernels, such as Linux.

Students implement these kernels from the ground up (we only provide them enough starter code to boot up into an entrypoint function; none of the design or virtual memory or anything is done for them), so each kernel ends up with slightly different design and structure. Landslide had to be flexible enough to be able to attach to and control any of these kernels.

The real test was meeting with some of the students towards the end of the project to get them to try out Landslide, to see if it works "in the wild". Sometimes it was a struggle - the average time spent doing instrumentation was 100 minutes - but in the end, all four (out of five total) groups that completed the prerequisite instrumentation were able to find bugs in their kernels with Landslide.

JE: That is a very elegant twist to the project, I think. You both get to test your tool on real code that you know nothing about, and you teach the students that there are tools that can do almost magic things to help them check their code for errors.

JE: Do you have any fun examples from the work of bugs found?

BB: Sure. My favorite story is of one of the project's defining moments, a time when I just stopped in my tracks and thought "huh. I have made something smarter than me."

I had written a new unit test case, called double_wait, in which two threads in a parent process attempt to wait() on a single child process. I was testing it out on my own kernel (the one I wrote when I was an OS student), and I expected the search to complete with no bugs, each time having one parent thread succeed and the other return failure. Instead, after a bit of searching, Landslide made my kernel trip an assert in its condition variable linked-list logic.

This was a bug nobody ever knew about before - I'd written it three years before and didn't find it during stress-testing; and the TA who manually graded my kernel (by looking over it with a red pen) also overlooked it.

JE: That just goes to show how difficult reasoning about concurrency is for a human being.

JE: As you know, I previously interviewed the creator of SimTester. At a high level, these two tools built on Simics would seem to be similar, but I think that they are actually quite distinct. Could you expand a bit on that?

BB: Imagine my surprise when the SimTester interview came out just when I was writing my thesis's related work section. "Whoa, wait, another tool that runs in Simics and identifies key decision points and injects interrupts?"

So our projects are quite similar in those regards, but at a higher level my strategy is somewhat different from theirs. SimTester focuses on races involving device interrupt handling code, whereas Landslide is geared towards non-driver inter-thread races. Additionally, SimTester’s testing model involves forcing only one interrupt per test run, which is a lighter-weight approach than systematic exploration.

JE: Thank you for your time, this has been really interesting. Where can you go to find more information about Landslide?

BB: The thesis can be found at http://www.pdl.cmu.edu/PDL-FTP/associated/CMU-CS-12-118_abs.shtml (and the slides from the MSc defense at http://bblum.net/landslide-defence.pdf).

 

For additional information from Wind River, visit us on Facebook.

 

Crystal Forest on Simics

By Jakob Engblom

A few week ago, Intel announced the Crystal Forest platform, a new IA-based platform for networking and communications applications. The first Crystal Forest hardware is appearing now, but we have been working with Crystal Forest on Simics for more than a year already. 

While we cannot yet talk about the details of the hardware (and in any case, that is more Intel's story to tell, not ours), it is a pretty complex piece of silicon.  Crystal Forest is based around Sandy Bridge processors and features hardware acceleration for network and security tasks.  It features multicore 64-bit processors with virtualization and the latest iteration of the IA instruction set.  All of this is faithfully modeled in Simics.

A Crystal Forest system can be configured in a lot of different ways for a particular application.  The particular SKUs (chip variants, in Intel lingo) used can differ, and the number of processor cores go from a few to very many.  In Simics, it is easy to cover the configuration space with a very small effort. It is just a matter of setting simulation parameters to vary the hardware configuration, numbers of cores and accelerator blocks, and network setup.  The simulator makes a single deliverable potentially correspond to a large number of different configurations, unlike hardware boards which are by necessity limited to a single configuration.

Simics for Crystal Forest also enables system integrators to build their own custom platforms, including custom ASICs and rack configurations.

The first demo we have shown for Simics Crystal Forest emphasize networking.  Inside a single simulation, we put two Crystal Forest machines.  Each has a two-port i82599 network card, 8 GB of RAM, and a 4-core processor (several times more memory and processors than the host on which it is typically run). Two Ethernet networks connect the two machines, allowing experiments in routing and forwarding. To see the network traffic, we insert a "network instrument" on the two networks that count the Ethernet packets being sent and offers us an ability to inject network faults.

In action in Simics, the demo looks like shown below. Note the network instruments being shown in the "System Panel" view, and the machines and other components in the system listed on the left. The software stack on the target is a Wind River Linux demonstration setup that is identical to what is used with physical Crystal Forest machines - no adaptations were needed to run on Simics.

Unlike general x86 virtual machine solutions, Simics models the actual details of the Crystal Forest platform. This means that software stacks that require particular hardware support runs fine on Simics. In particular, the Intel Dataplane Development Kit (DPDK) has been proven to run on Simics. The DPDK is a middleware stack that provides software access to the acceleration and encryption functions of the Crystal Forest hardware, and it expects quite particular features from the hardware which is not found in most standard PCs.

Since Crystal Forest is a model of an actual chipset, it also boots actual real-world BIOSes.  Usually, the initial part of the boot of an x86 target in Simics and other simulators is done using simulator-specific boot code such as SeaBIOS (to save development time and simplify the model). However, the Crystal Forest model is detailed enough that actual production BIOSes (including UEFI, obviously) can run on it. This makes Simics an ideal platform for developing custom BIOS code for custom hardware extensions to the Crystal Forest platform and doing bringup on system solutions built on Crystal Forest.

Obviously, the Crystal Forest models have all standard Simics features, like reverse execution, determinism and repeatability, fault injection, checkpointing, and bug transportation. Thanks to Simics VMP, they can also run really fast on IA hosts.

Please note that the demonstration setup I used is just one potential configuration - Simics has been configured with more boards, more networks, and many more processor cores and target gigabytes of memory than this. It was just a suitable setup for the demo software used for this demonstration.

 

For additional information from Wind River, visit http://www.facebook.com/WindRiverSystems.  

Using Static Analysis to Improve Product Quality, Earlier and Cheaper

By Bill Graham

Fixing bugs is expensive. Fixing bugs is more expensive the later you leave them, in fact, its been shown to cost a magnitude higher with each major phase of development. The famous defect cost chart from Capers Jones shows the cost of a bug going from $25 at the coding phase to $16,000 in development.  Not only that, but most of the bugs are introduced into the system at this stage yet very little of them are found. Logic says we should fix them earlier to save money.

  (Source: Applied Software Measurement, Capers Jones, 1995)

 

Static analysis tools such as those offered by our partner Coverity Inc.  provide an easy, non-obtrusive way to detect defects in the source code – as it’s being coded. These tools can find all sorts of defects even before the code is submitted to version control or compiled into your integration builds. Significant and dangerous errors such as buffer and integer overflows, format string vulnerabilities and poor coding techniques can be found in your source code.  Static analysis tools are integrated into your build environment as part of your nightly integration builds, for example.  What is also extremely powerful is the integration of these tools into our Wind River Workbench. Developers can quickly and easily check for bugs in the code in the integrated development environment. A quick click and the current code file you are working on is checked for bugs.  Developers are understandably skeptical but success rate for static analysis tools is high and the technology has improved to where false positives (detected bugs that turn out to not be errors) are very low, typically less than 5%.

Using tools like Coverity’s Static Analysis in addition to automated test tools (Wind River Test Management, IPL’s Cantata++, for example) are great ways to improve the graph we see above.  Imagine the return on your investment – every bug you find and fix at the developer’s desktop saves you $1000 if you are lucky to catch it during integration or even $16,000 if it makes it into customer hands.

 

For additional information from Wind River, visit http://www.facebook.com/WindRiverSystems.  

Making a Faulty Serial Port

By Jakob Engblom 

As noted before, one common use case for Simics is doing fault injection.  Fault injection is typically added to an existing system without changing it, since the goal is to maintain a fundamentally fast simulator when not injecting faults.  There are a range of mechanisms available in Simics to implement fault injection. This blog post will take a look at two such mechanisms and how they are used in the context of injecting faults on a serial port.

We will look at two ways to induce bad behavior into a serial port in the system shown below:

First, we introduce faults in the interface between the serial port and the processor using it.  I.e., on the register bank of the serial port that the software on the system processor(s) use to send and receive data from the port.  This simulates a device with flaky interface circuitry, or a bus that would sometimes fail to correctly pass the data from the device to the processor.  It does not directly correspond to a faulty device, since the device operation is not affected (unless of course an error is injected on a transaction that targets the configuration or control registers of the device and have unintended side effects).

On the surface, this might seem trivial to implement: just pass all transaction from the main memory map through an intercept module, and have that module corrupt the transactions as it sees fit.  However, this would burden all transaction with a cost and remove the ability of the simulator to cache values and optimize accesses to memory and other devices which are not affected by the fault injection. Instead, in Simics, we can introduce a level of indirection and intercept only accesses to the serial device.

As shown in the picture below, we do this by introducing a private little memory map for the serial device. The mapping in the main memory map is set to point to this new memory map, and it in turn maps the serial device.  The new memory map has an intercept module attached to it which can see and modify both read and write transactions.

This target system modification can be done in a few lines of Simics scripting, even in the middle of a simulation. Simics allows arbitrary reconfiguration of the simulation system at runtime, and the changing of the memory map to point to something new and the introduction of a new memory map and fault injection module can both be done and undone at runtime.  Using a module instead of just a script for the injection work makes the fault injection more robust, and ensures that the injection of faults is part of checkpoints and allow for reverse execution.  Any Simics user can create such modules themselves, using the Simics Extension Builder product.

If we program the fault injector to replace some output characters with other characters, we get something like this:

Note how the output is suddenly corrupted with a bit of L337-style text.  This is achieved by changing the values written to the serial port transmission register, on their way from the processor to the device. The processor is not affected, only the data in flight.

A second fault injection pattern hooks into the back end of the serial device. The serial device has a front end that handles the registers facing the processor, and a back end that handles the transmission and reception of characters (transactions) on the serial line.  The front end and back end are tied together by the core device functionality.

The back end implementation has the serial device modeling talking to a model of a serial console (or other hardware attached to a serial port) using the Simics "serial interface".  Note that the serial device does not interact directly with a user, but rather it expects some other model in the simulation to provide this service.  In this example, it is a serial console, but it could just as well be another virtual machine. It can also be a fault injection module that puts itself into the path of the data, as shown below:

Unlike the previous case, here we put the fault injection module on the actual data path.  The fault injection module actually receives all data and passes it on to the other side of the connection. It is not selective in the same way as an intercept module. Just like with the intercept module, making this an explicit part of the simulation enables checkpointing and reverse execution to work during fault injection.

In the code, each output from the serial device calls a function in the serial interface of the fault injection module, which in turn calls the same function in the serial interface on the serial console. If data is just forwarded, neither the serial device nor the serial console sees anything different.  However, the fault injector can do anything with the data as part of passing it on, including changing data and dropping data. 

It is also possible to introduce spurious data transmissions, simulating random noise on the line that looks like actual data transmissions to the devices in the system.  Note that since Simics does not actually simulate the analog encoding on the serial line, a model of random works by generating entire characters - essentially simulating a case where noise is interpreted as valid data by the receiver circuitry.  The intercept module used in the previous example could also create transactions, but would have to make use of additional interfaces to do it (it would essentially perform its own independent memory accesses).

 

For additional information from Wind River, visit http://www.facebook.com/WindRiverSystems.  

Simics for Networked Systems: An Interview with RTI

By Jakob Engblom

Networked systems is one of the primary application areas for Simics. Simics can run many machines inside a single Simics simulation, and apply all Simics features to the system, not just to a single machine inside the system. This is something that Dan Poirot from Real-Time Innovations, RTI, has made quite extensive use of recently. It is a pretty cool use-case for Simics, so we arranged to interview Dan about his use of Simics.

Jakob Engblom: We start with the basics. Can you introduce yourself?

Daniel Poirot: Hello! My name is Daniel Poirot. I am the RTI Senior Field Applications Engineer covering the Southeast US. I came to RTI following 13 years selling development tools in the embedded industry. Prior to that I worked for 15 years as a developer in the NASA Johnson Space Center community.

JE: That's a very impressive bio, I only have about 14 years myself in the embedded space.

JE: So, what does RTI do?

DP: Realtime Innovations sells tools and services supporting and extending the Object Modeling Group (OMG) Data Distribution Service (DDS) standard. RTI DDS accelerates the integration of real-time, mobile and embedded applications with each other and with enterprise and legacy applications. RTI DDS provides the developer a data-centric, dynamically configured, fully anonymous, publish/subscribe system with many powerful Qualities of Service (QoS). Applications can be written needing very little knowledge of the deployed system ahead of time.

JE: How do you use Simics?

DP: I have used Simics at RTI as well as at positions with two earlier companies. In each case, Simics has provided a binary compatible representation of networks of different CPU types running different target operating systems! Simics is also a great way to prototype a system of systems or service oriented architecture. Additionally, Simics gives RTI a robust platform for testing our products during development.

 JE: So you simulate multiple target machines in a single Simics session, right?

DP: Yes indeed, including heterogeneous systems. I have used Wind River VxWorks and Wind River Linux as well as ‘kernel.org’ Linux on a number of simulated machines at the same time. I started with a single Power Architecture target and have progressed to different Power Architecture and Intel Architecture targets.

Simics offers the developer (using Windows or Linux on their PC) access to other chips (including Intel, Power, MIPS, ARM) running other operating systems (VxWorks, VxWorks 653, WR Linux, and others).

RTI offers a common API (OMG DDS) and interoperable network wire protocol (OMG RTPS) allowing applications written in different languages (C, C++, Java, .NET, Ada) to communicate seamlessly on these targets.

If we put this together, by using Simics, we can demonstrate and test these combinations on a single PC, no need to cart around a case of different development boards.

Here is an example screenshot (click to enlarge):

We see lots and lots of windows in a system where we are simulating multiple boards for use with Windriver Workbench.

‘Simics Control’ shows the configuration of two dual-core p2020 RDB boards, ADAM and BERTIL. Their serial consoles are on the left displaying typical VxWorks 6.8 operations.

Also included is CESAR, a p2020 DS board. It’s serial console is at the bottom left. It is showing WR Linux 4.0 commands.

The ‘Simics Command Line’ window is used to configure the running system. Here we enter commands such as ‘connect-real-network-host’. Errors are logged here as well results from as my successful configuration.

To the far right we see the ‘Advanced Device Development’ perspective of Windriver Workbench configured with Target Servers for the simulated boards.

The ‘Host Shell’s at the bottom of the screen are attached to the Target Servers for ADAM and BERTIL. Here we see similar but different VxWorks output.

In a typical development use case, RTI DDS may be loaded as a series of shared libraries and the target application.

Each of the VxWorks windows shows the RTI DDS libraries, libnddscore, libnddsc and libnddscpp, as well as an application, either ‘rtiddsping’ or ‘rtiddsspy’.These Ping and Spy applications are used initially to confirm proper network connectivity.

JE: Yes, using Simics like that can simplify both testing and demonstration of solutions. The final product obviously has to run a specific physical piece of hardware, but a large part of the development can be done in a virtual world with much more convenience, as you noticed.

JE: What kinds of network topologies are involved?

DP: RTI DDS supports a wide variety of standard networking protocols as well as shared memory and ‘disadvantaged or intermittent’ links. RTI DDS applications can communicate using shared memory, UDP, TCP, DTLS, TLS, IPv4 and IPv6 without having to program directly to those protocols. Using Simics we can create networks of subnets and show how applications communicate.

JE: So,you are not only using multiple machines, but also multiple network links inside of Simics?

DP: High speed, reliable communications across networks is fundamental to the RTI DDS product. It has be a challenge to setup a good demonstration of several networks. I always seem to need ‘n+1’ computers! With Simics we can virtualize the machines needed. The target operating systems are configured as they would on ‘real’ hardware. Testing and demonstration of a system of systems or service oriented architecture can actually be realized with different networks and different network IP numbers – just like in the ‘real’ world.

JE: Virtual  networks are convenient, I must say. The fact that  you can always add another machine without having to scavenge for lab hardware is something we have heard from many places as being a very valuable property of a virtual system compared to a physical one. 

JE: Was there anything else in Simics that helped you get your tests done?

DP: The high fidelity of the Simics targets allows us to use standard development tools such as Windriver Workbench. We are also able to use RTI networking tools such as Analyzer and Monitor to visualize DDS information. Open source tools such as Wireshark can even be used on the simulated networks.

JE: So you also connected the virtual machines to the outside physical network?

DP: Yes. I was pleased to see that Simics uses popular network simulation tools. We use OpenVPN at RTI so adding additional virtual interfaces using TAP-Win32 is very straightforward for me. The tools are all scripted so the details only need to be set once.

JE: Thanks for your time! This has been very interesting, seeing how Simics is applied to this kind of distributed networked system.

 

For additional information from Wind River, visit http://www.facebook.com/WindRiverSystems.

Stop, Think, and Tie Your Shoes Right

By Jakob Engblom

I was sent a link to a past TED talk about how to tie your shoes. It is worth watching, and from the discussions I have seen on Facebook around the video, it seems that most people do it the "wrong" way.  I have actually started to try to do it the "right" way, and the result is real improvement. 

Tying the "right" way does make the bands look better. However, it did require a little bit of practice and a conscious effort to change. 

Shoe tying is admittedly somewhat away from embedded systems development, but the principle is general.  Doing things in a better way requires three things: understanding that there is an issue with how things are done today (bad knots), knowledge of a better way of doing it (good knots), and the time and effort to change your habits (dedication).

In our world of embedded systems, you will always find issues with how things are done.  Too slow, too expensive, too many bugs...finding problems is the easy part.  What is harder to find are better ways of doing things, and what is really hard to find is the time and motivation to actually effect change.

A speaker at a Cadence event I attended last year pointed out that life is tough for hardware engineers.  In addition to doing their current job of designing hardware, they are supposed to start thinking about how to support software development.  How would you find time for that?  You are already too busy doing your assigned hardware design work to find time to care about the overall product design process.  To really make change happen, you would need room to stop, take a step back, and think about what you should do and how to do it.

We often encounter this when introducing Simics virtual platforms to new users.  Simics certainly seeems to provide many useful abilities and has the potential to change development processes for the better (browse through other Simics blog posts for more on what Simics does).  Turning that potential into actual process improvement requires both understanding how it work, and the time to implement and change habits.

In the end, it comes down to leadership and management.  Individual engineers have to be given the time and incentive to try new tools and new ways to use old tools.  Some things could be as simple to implement as a better way of tying your shoes -- while others are more like gathering 1000 people and convincing them all that they really should use shoes with bands rather than velcro-based sandals and to tie these newfangled shoe things in a particular way. There are also people who have not realized that shoes have laces, and the reason that they are constantly tripping and hurting themselves are those loose laces. The good thing about such cases is that tying the shoe ever which way is an improvement.

By the way, for some reason most of my swing dancing friends already knew how to tie shoes the right way. I guess shoe-tying matters more in some cases. If you want to look sharp, you have to invest the time to do things right!

 

Surfing the Web with Netscape 4

By Jakob Engblom

While looking into our old SPARC server targets for our new SPARC legacy offering, I  opened up a Simics demo checkpoint that we have had around for a long time (at least eight years). This demo is an UltraSPARC II running Solaris 9, and it shows how web browsers looked around 2001.  I used real-network to connect the machine to the Internet, and tried the default webbrowser - Netscape 4.78, built in June of 2001.

Since the Javascript implementation in the browser is pretty poor, and its CSS support terrible, the main Wind River home page would not basically not render in any useful form. Our blog page at http://blogs.windriver.com/wind_river_blog/simics/ does render - but maybe not in the expected way:

As a friend of mine commented on Facebook - is the problem that there are too many Jakobs there?

For all its shortcomings, we also have to admit just how revolutionary Netscape was. I still recall using version 1 to look at the few pages of the nascent world-wide-web when I was still a university undergraduate... And I am writing this blog using Firefox 6, which essentially is the latest in the line of development leading up from Netscape 4.

While it makes for some laughs to visit the past in this way, it can also be very useful.  If you have some old applications still living on an old machine like a Solaris 9 SPARC, what do you do?  Sometimes you can port the applications to more recent systems, but there are cases where it is just simpler, cheaper, and less risky to use a virtual machine to keep it running.  For example, if the source code is lost, the application is just a binary, the application depends on some device driver (which we can run on the virtual platform, since we run the complete OS stack), or even when there is source it has annoying word-length and endianness problems.

Note that if we would have liked to see a web site like it was back in 2002, we could have added a server to the virtual network to serve the pages like they looked back then. Preferably, running a contemporary web server on a contemporary OS to make it more "authentic."

 

 

How to Get Virtual

By Jakob Engblom

My blog posts about Simics tend to focus on how virtual platforms are used and what you can do with them. However, underlying all Simics use is the issue of getting a virtual platform in place. You need some kind of platform to run your software on in order to get the full benefit of Simics. Thus, creating virtual platforms is an important part of practical Simics use for many of our users.

We have been working with Simics for more than a decade, and we have learnt a lot about modeling along the way. The approach we have ended up with is a top-down methodology, where we start by investigating the target system as a whole from the perspective of the intended use cases, and then explore the details of each subsystem in order to determine how best to model it.  We then switch to bottom-up, applying an iterative modeling flow that delivers something useful as quickly as possible. During development, the goal is to always keep a useful (albeit probably incomplete) platform available at all times.

It is easy to get distracted by the details when creating a virtual platform.  If you start with a list of all the components of a system without considering the system context, you are likely to assume you need to model everything, as well as every function of everything. This is usually an overwhelming task, resulting in a protracted development project without clear focus or any chance of delivering something early. So, we need a better way. 

The method we typically use with Simics is to start by investigating how the system is going to be used, and which hardware units are involved in these use cases. In this way, we can often exclude parts of the target system from consideration. Certain subsystems or subsystem functions might not be needed at all, or needed only at a later stage of the project.  Even when modeling an SoC with the eventual goal of a complete model, it is possible to create a priority list and staging sequence that quickly gets something into the hands of users that applies to their initial use cases.

Once this analysis is complete, the existing library of Simics models might well cover a large part of what is needed. There could also be "near misses", where a small adjustment to an existing model is all that is needed. In this way, the ever-growing library of Simics models provides significant short-cuts towards a final system model.

Next, it is time to start building the new models needed to complete the target system. This process is run in an iterative, agile way. We strive to define a minimal subset of the target system that we start with, so that software can be run on the target as soon as possible. The outer loop of the modeling effort is the extension of the overall system model and the testing of software on it.  The inner loop is the creation and changing of individual device models, using both model-level unit tests and software tests to drive the development.

The net result is an flexible project where the virtual platform can be updated in reaction to changes to the target system design. Often, modeling projects are begun before the details of the eventual target hardware are completely known. As shown in the picture below, the result is a series of incremental model deliveries that represent the current design state of the hardware.

This modeling method applies regardless of whether the modeling being performed is pre-hardware (for a new chip that might not even be announced yet or a new board built on existing components), post-hardware for a board or system that exists right now, or post-obsolence for an old system that still needs to be maintenaned.

Modeling can be done by Wind River, Simics users themselves, or third-party services firms. All Simics users are equal in what they can model and their access to the Simics modeling tools and APIs.

To learn more about system modeling with Simics, please read our new white paper on creating virtual platforms with Simics.

Happy modeling! And remember - resistance is futile, you will be simulated!

 

 

Software at the Toddler Stage

By Jakob Engblom

Developing software is an adventure where you explore the unknown, one line of code at a time.  At some point, the program reaches the stage where you can actually run it and try it - and that's where the real fun begins.  The software will sometimes behave as expected, but more often than not it will not. It will do something else, or crash, or generally do the wrong thing.  It is very much like a toddler - you can rely on it to some extent, but you never know when it will decide to do something unexpected, funny, or just throw a fit over something that would have seemed inconsequential.

I have spent quite a bit of time recently with toddler stage of Simics 4.6 (prereleases and betas).  The end result is really good and qualifies as an adult specimen (as discussed previously on this blog), but there was a lot of entertaining (and frustrating) things that went by as Simics 4.6 was a toddler. All software developers have stories like these to tell, but I just wanted to share a few of the funnier ones and give a glimpse of just how much work it is to get something like Simics running.

My favorite issue was the strange memory allocation happening in Eclipse on 64-bit Windows host.   When I opened the Simics timeline view, the Eclipse process grew by exactly 4 GB plus 4 bytes.  When I closed it, the memory was deallocated, all nice and tidy.  It worked perfectly well on all other hosts.  It turns out that the reason was that our magnification slider (see picture) used a Windows widget that when running in 64-bit mode allocated exactly 4 bytes for each step of
the slider.  Since we wanted a slider that could handle a very large data set as the worst case, we had set it to allow 1 billion as maximum value.  Reducing this to a few hundred thousand allows
for sufficient resolution and causes no memory issues.  One still wonders why Microsoft made the widget behave that way.  I guess it just wasn't tested for anything but a volume control with 11 steps or something.  Once again, users violate the assumptions of software designers.

In another case, the debugger got a bit confused about endianness, and managed to display some variables as BE and others as LE. In the same view, for the same target program, at the same time.  Gave me a good laugh seeing it and diagnosing why some variables with the value "1" were shown as 16777216. I am a nerd at heart, I admit.  A bit like a child writing letters backwards as they learn to write.

The Simics toddler also was confused about directions sometimes. Pressing the "reverse" button instead made it go forward.  Pressing "stop" when it was stopped also made it run forward, which was a bit more surprising. All was easily sorted by wiring the right buttons to the right functions.

All software developers have many stories like these to tell, and it underscores the continued importance of creative and intensive testing.  Many issues do not show up until you start to put all the pieces together, and that aspect of integration testing is one of the main uses of Simics itself.

 

They Just Upgraded our Coffee Machine

By Jakob Engblom

They just upgraded our coffee machine in the office. When I came out into the kitchen to get another cup, a service technician was there with a laptop perched on the sink and a USB cable running into the innards of the coffee machine, into the back of its open front panel. When I asked what was going on, he said that he was updating the coffee machine. In particular, he was adding the feature of selecting the strength of the coffee being brewed.

A software update to add features to a coffee machine? It is a bit weird, but also a perfect illustration of how pervasive embedded software has become in all manners of systems.

It is not easy to explain to non-technical people just what an embedded system is and what software does when it is not running as an application on a PC (or smartphone or tablet or other familiar computing device). Our coffee machine upgrade does offer a nice illustration of what is going on, and how software and hardware combine to form cyberphysical systems.

Mechanically, the machine contains a grinder and a water heater and a water dispenser. Software controls these mechanical components to determine how much coffee to grind and how fine to grind it. The software controls the amount of water to use and how much to heat it. With these few variables, almost infinite variety is possible. As our machine also contains a milk dispenser, milk heater, and some other bells and whistles, it can easily provide quite a wide selection of beverages by combining various units and varying their operating parameters.

The importance of the software is clear to anyone who attempts to do the same tasks manually. If it is badly written or contains flawed control algorithms, the result can be almost arbitrarily awful. But it does make the system far more flexible. Decades ago, there would have been a few buttons on the front and each button would have been tied to a set of parameters for the mechanical units. The parameters would have been set by manual tuning of some adjustment screws on the circuit board. Now, the software can just set the parameters at any point in time, making it possible to adjust on the fly, and vary their value in order to vary the strength of the coffee.

The front panel of the machine demonstrates how embedded software can actually become visible to users. I understand the designer convenience of using a graphical display along with a numeric keyboard (!) to make selections. But it does make the machine feel more like a 1980's DOS adventure game than a user-friendly coffee machine. It takes a few keypresses (never less than three) to navigate the maze of menus and get something dispensed. It also allowed the programmers to include a service menu which definitely moves the machine from the realm of a simple appliance into being outed as a computer: