Making a Faulty Serial Port

By Jakob Engblom 

Engblom_lgAs noted before, one common use case for Simics is doing fault injection.  Fault injection is typically added to an existing system without changing it, since the goal is to maintain a fundamentally fast simulator when not injecting faults.  There are a range of mechanisms available in Simics to implement fault injection. This blog post will take a look at two such mechanisms and how they are used in the context of injecting faults on a serial port.

We will look at two ways to induce bad behavior into a serial port in the system shown below:

Initial system setup block diagram

First, we introduce faults in the interface between the serial port and the processor using it.  I.e., on the register bank of the serial port that the software on the system processor(s) use to send and receive data from the port.  This simulates a device with flaky interface circuitry, or a bus that would sometimes fail to correctly pass the data from the device to the processor.  It does not directly correspond to a faulty device, since the device operation is not affected (unless of course an error is injected on a transaction that targets the configuration or control registers of the device and have unintended side effects).

On the surface, this might seem trivial to implement: just pass all transaction from the main memory map through an intercept module, and have that module corrupt the transactions as it sees fit.  However, this would burden all transaction with a cost and remove the ability of the simulator to cache values and optimize accesses to memory and other devices which are not affected by the fault injection. Instead, in Simics, we can introduce a level of indirection and intercept only accesses to the serial device.

As shown in the picture below, we do this by introducing a private little memory map for the serial device. The mapping in the main memory map is set to point to this new memory map, and it in turn maps the serial device.  The new memory map has an intercept module attached to it which can see and modify both read and write transactions.

System modified to include a new memory map and an intercept module

This target system modification can be done in a few lines of Simics scripting, even in the middle of a simulation. Simics allows arbitrary reconfiguration of the simulation system at runtime, and the changing of the memory map to point to something new and the introduction of a new memory map and fault injection module can both be done and undone at runtime.  Using a module instead of just a script for the injection work makes the fault injection more robust, and ensures that the injection of faults is part of checkpoints and allow for reverse execution.  Any Simics user can create such modules themselves, using the Simics Extension Builder product.

If we program the fault injector to replace some output characters with other characters, we get something like this:

L33t fault in output on mpc8641

Note how the output is suddenly corrupted with a bit of L337-style text.  This is achieved by changing the values written to the serial port transmission register, on their way from the processor to the device. The processor is not affected, only the data in flight.

A second fault injection pattern hooks into the back end of the serial device. The serial device has a front end that handles the registers facing the processor, and a back end that handles the transmission and reception of characters (transactions) on the serial line.  The front end and back end are tied together by the core device functionality.

The back end implementation has the serial device modeling talking to a model of a serial console (or other hardware attached to a serial port) using the Simics "serial interface".  Note that the serial device does not interact directly with a user, but rather it expects some other model in the simulation to provide this service.  In this example, it is a serial console, but it could just as well be another virtual machine. It can also be a fault injection module that puts itself into the path of the data, as shown below:

System setup with an intercept device between serial port and serial console

Unlike the previous case, here we put the fault injection module on the actual data path.  The fault injection module actually receives all data and passes it on to the other side of the connection. It is not selective in the same way as an intercept module. Just like with the intercept module, making this an explicit part of the simulation enables checkpointing and reverse execution to work during fault injection.

In the code, each output from the serial device calls a function in the serial interface of the fault injection module, which in turn calls the same function in the serial interface on the serial console. If data is just forwarded, neither the serial device nor the serial console sees anything different.  However, the fault injector can do anything with the data as part of passing it on, including changing data and dropping data. 

It is also possible to introduce spurious data transmissions, simulating random noise on the line that looks like actual data transmissions to the devices in the system.  Note that since Simics does not actually simulate the analog encoding on the serial line, a model of random works by generating entire characters – essentially simulating a case where noise is interpreted as valid data by the receiver circuitry.  The intercept module used in the previous example could also create transactions, but would have to make use of additional interfaces to do it (it would essentially perform its own independent memory accesses).

 

For additional information from Wind River, visit http://www.facebook.com/WindRiverSystems.  

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>