By Jakob Engblom
In a recent Simics seminar, I was asked about repeatability, variability, determinism and Simics. This is a question that comes up almost every time I present about Simics in front of an audience with testing experience. The people asking the question intuitively think that determinism is a bad thing - since it sounds like it will limit the execution scenarios that will be explored in testing. For a tester, variation is a good thing. However, determinism is not in conflict with variation. And I think I found a perfect illustration of this in a setting that is a bit more accessible and easy to understand than computer simulators. In a computer game, Bad Piggies, from Rovio.
Bad Piggies is a game based on a physics simulation (not particularly realistic, but still reasonably consistent with everyday experience), where you put together strange contraptions to guide an endearing little pig pilot from the starting point to a goal. What struck me about the game is its incredible sensitivity to inputs, while still being 100% deterministic. To me, this is absolutely analogous to how Simics works (and I guess drawing that analogy indicates a pretty warped mind too).
There are two types on inputs in the game that matter: the (static) configuration of the vehicular contraption that you build, and the interactive inputs you provide during a level to turn on engines, pop balloons, or fire off boxes of TNT. An utterly minor change in the timing of an action can mean the difference between success and failure. On the static side, moving the center of gravity of a vehicle by putting the pilot pig somewhere else can have a huge impact on the behavior. There are levels where all you do is just move the center of gravity around to see which location finally hits the perfect balance.
Still, the game is clearly deterministic. This is particularly obvious in levels where no interactive input is needed. All you do is put the vehicle together and watch it go (roll down a hill, float away on a balloon, or throw the pig by exploding a box of TNT underneath it). In these cases, each game plays out the exact same way, as the physics engine is intentionally and precisely crafted to be deterministic. Whether I play an HD version on an Android tablet or a regular version on a iPod touch, the outcome is the same and the way you clear the levels identical. This is strong determinism.
Is not predetermined, however - when you put a new vehicle together and let it go, you really cannot predict what is going to happen. You might have an idea in mind, and you might hope it does the right thing. But before you play it through, you do not know. It most likely fails on the first attempt, and you go back and tweak the design (in computer programming, this is known as debugging). And try again. Each time, something different will most surely happen, since the setup input is different. There is huge variability, and if we see the game as a way to test the contraptions we put together, there is infinite room for variation on top of the fundamentally deterministic game engine.
Imagine how this would be if the game was really nondeterministic. It would be pretty much unplayable. It would turn into a die-rolling exercise, where you would create the exact same initial conditions, let it run, and hope that on some run, you would be lucky and get to the goal. Not my idea of fun, exactly. Determinism is clearly good and helpful for testing and development, as long as there is also variability and sensitivity to inputs. But if inputs do not changed, you do not want behavior to change. Normally, computer system are indeed random, and since this is a fact of life, many people make lemonade out of lemons and think of this randomness as a good tool for testing. But when you can do it, avoiding randomness is clearly superior as a way to approach systematic testing. And as a way to fly computer game pigs to the goal.
Simics works in the same way as the Bad Piggies game. For a static test case like a non-interactive boot of an OS, it will play out the same each time you run it (provided no configuration changes are made). Change the target OS image, and you will get something different. Change the number of processors or the size of memory, and something different will happen. Change the host machine (the machine on which Simics runs), but not the target, and you will get the same result.
If we add uncontrolled interactive inputs to the mix, we will get different results each time. We can make interactive inputs deterministic by scripting or recording them. And given deterministic inputs, the execution will be the same. This is how Simics achieves perfect repeatability. Not by limiting what executes and how, but by controlling inputs to a system so that any particular variation can be reproduced.
Taking this one step further, a simulator also allows you to systematically explore issues by programming input variations into scripts. At one extreme, this is fault injection, but even just automatically varying inputs to a program within legal bounds can be very interesting. For example, Simics has been used to provoke OS bugs, or test how a program runs as the number of processor cores in a system is varied.
Repeatability and determinism just mean that we can repeat any execution we happen to chance upon - not that we limit the space that can be explored in testing. For any tester who ever had to try to replicate and report a failed test case that only was seen once, repeatability is a wonderful thing indeed.
For additional information from Wind River, visit us on Facebook.
Subscribe
A good post, but Bad Piggies is not deterministic. I suspected this while playing the game, but I just confirmed it after reading this blog post.
I reran my solution for level 1-35 three times with zero input from me, I just let the cart roll down the hill and fall onto a box of TNT. All three times, the resulting explosion occurred in a different manner.
I think this is due to the way that the Android handles animation, you can see the overview here:
http://developer.android.com/guide/topics/graphics/prop-animation.html
If Bad Piggies ran its own internal timer it could be a deterministic game, but performance could suffer because when the apparent speed of the game would change as the phone's processing load grew or shrank. Instead the game relies on Android to tell it what the system time is when it is time to redraw the screen, and since the outside system load is non-deterministic, the time values at which the screen gets redrawn will be slightly different, and this will blow up into different behavior over time or as the pig cart receives impulse forces like explosions.
Posted by: Rich Dubielzig | 11/30/2012 at 11:16 AM