Mike Deliman

Quit Bugging Me: Revalidated

One day, I get this phone call.  "We're working with the system, we see the calls that update the exception handlers early on - connecting the clock routines, etc.  Then not very much farther on we see the system has run past where tasking should be running but it's not.  When we check, we're not getting any clock interrupts, and the clock exception handler isn't there any more.  How could the system do that?  Is there any way the OS could remove it's clock connections?"

The complete story was even more eyebrow raising.  "We're testing for a launch, this vehicle is a one-of-a-kind, there's no backups.  We're doing environmental limits testing.  We're still within spec, but operating at the low end of the electrical ratings.  We need to explain this anomalous behavior or scrub."

This was the start of a frantic week of work.  On PowerPC, the architecture stores it's exception vectors down in low ram.  A valid method of updating these vectors is to write new data to the vector address as data, then force the data cache to be pushed to the RAM, and finally tell the processor "any instructions you have cached for this area are invalid" (invalidate the I-cache).  This last step ensures that the CPU will fetch the newly-updated instructions the very next time they're called.

The hardware involved was fairly rare and expensive. We didn't have a lot of options available for hardware debug assist.  In fact, I had never even seen the hardware in person, all my work was theoretical and experiments were done with "similar" boards.

I used what I had to trace the execution sequence to the point in question, where we should have been seeing clock ticks come through. With all the hardware I had, there was no way I could recreate the problem.  In every case, I had clock ticks come through as expected, and with code to set the clock to an expected value earlier on, every single run showed the same exact results.  It was impossible for it to /not/ work.

The sequence, boiled down, that was under investigation, runs like this:

The runtime starts.  Vectors are written-to / populated with default values and routines.  Caches are invalidated and enabled. Board hardware is initialized.  A clock is set up, it's exception vector is populated via a data-write; the D-cache is "flushed" to RAM, the instruction cache is told "this area of RAM is updated" (invalidated), and we continue running.  1/60th of a second later we should get a clock tick, it should be processed through this exception vector and ....

The problem seemed to be that the exception vector was running the same old code - not the new code.  It could be checked that the new code had actually been pushed all the way into RAM - except we didn't have that capability at the time nor enough time to set it up.

In every other test setup, we saw the expected behavior - the clock tick delivered and tasking running exactly as expected.  In this particular setup, it was as if the clock tick never happened.  The CPU side of things clearly ran as if the code had never, ever been invalidated.

As a point of interest, we found the code executed was smaller than the instruction cache.  We'd never exhausted the cache.  We formed a theory, hard to prove, but interesting.  Running at the lower voltage end, the units under test were all the -same-, hardware wise.  With the lower voltage, was it possible that some anomaly of the hardware caused the instructions that had been invalidated to... somehow.. not be invalid?  The only way we could still be executing those instructions is if there were still in the I-cache, and we'd invalidated them.  Perhaps the hardware somehow revalidated the cache when run at low voltages?

Bringing power back up to spec showed normal execution.  Working with the hardware vendor we learned that there was a probability that the lab grade hardware might show anomalies not exhibited by deployment grade hardware, and indeed the lab grade hardware would fail tests that the deployment grade hardware passed without problems.

Testing was repeated on deployment grade hardware, which ran without incident.

Looking back, had I known then what I know now, it would have been expeditious to execute a field of no-ops the size of the I-cache immediately after installing the timer vector.  This would have caused the entire I-cache to be invalidated/populated with new entries, flushing the timer vector code completely.  The next execution of that vector would have forced the I-cache to re-read the whole vector, we would have seen the clock ticks where expected, and we could have explicitly tested the I-cache in this way.

Quit Bugging Me: Induction

Working on a customer problem once, we had an interesting phenomena.  Upgrading a system with a large VME cage and several boards, the customer replaced older processor boards with what were then "new" boards.  The old boards ran at (I think) 33 MHz, the new ones at more like 133MHz.

The overall system included motor control functions and sensor feedback, there were D-to-A, A-to-D, and other boards in the system.  The problem was, with a straight-and-simple rebuild of software for the new boards, the system was giving anomalous readings.  Checking the registers for values from the A-to-D boards, there were numbers showing up that were not possible.  Readings indicated the motors were on and moving their load when in fact the motors were not connected.  Switching back to the old computer boards eliminated the problem.  We added some hardware to examine VME bus activity, to see if the new CPUs were reading the bus wrong.  We had to put the test board between the CPUs and the sampling boards.

The sampling boards (A-to-D) were in the chassis right next to the CPUs.  They had a sampling rate not far off from the new CPU's clock rates.  Moving the sampling boards farther down the chassis reduced the severity of the anomalous data reads, but did not eliminate the problem.  The bus analiser showed the data on the VME bus was consistent with the data reported by the new CPUs.  This provided the final clues needed to pinpoint the problem and fix the system.

The new boards have clocks that run at a rate close to the sampling rate of the A-to-D boards.  The clocks were physically next to the A-to-D  boards in the VME chassis.  For whatever reason, the new boards oscillators were radiating enough energy at the right frequency to corrupt samples taken by the A-to-D boards.  By moving the boards farther apart we reduced the effect (inverse square law) but not by enough to eliminate it.  To test this theory we placed steel plates in a VME slot, as a separator between the rest of the boards in the bus and the A-to-D board.  At this point, both old and new CPUs read the same consistent readings from the A-to-D boards.  Removing the metal plates brought back anomalous readings with the new CPUs.

In this case it seemed like a new system of software and boards was improperly accessing a device.  Switching back to the old baseline eliminated all the new problems.  Analytical equipment verified the readings reported by the CPU were consistent with the data retrieved from the sampling hardware, so it wasn't a problem "within" the new software or hardware. The real problem was a completely unexpected interaction between legacy hardware and upgrade hardware.

Mitigating Risk

The phone rings.  A customer has an application destined for a high-risk environment, and somewhere in test they've found a new unanticipated condition.  A "Tiger Team" is formed - a group of experts who've "been here before", who understand the priority and the risks, and know how serious it is.  Many times a Tiger Team may give the go / no-go, the final answer that either saves a mission, or drops all that work into the dust-bin.  Such a team itself represents a *lot* of work, frustration, and time - investigations may run on days, weeks, or months, as long as they reach their conclusion on-time.  And there's the one facet that can't be changed by any engineering practice:  Time.

Developing space, mission- or life-critical applications is serious business.  The deployment environment may be harsh - and much about it may be unknown.  To make sure missions are successful, the known risks are eliminated as well as possible.  Some of these risks may include high radiation, occlusion by the Sun (Solar Conjunction), extreme differences in temperatures, harsh chemical environment, vibrations, or electrical noise.  Some of this you can plan for with hardware, some of this you can't.  You address the things you know about, that's all you can do.

With software engineering, you're also faced with unknowns.  One of the biggest impacts is the unknown date when you'll finally have the actual hardware the project is going to run on.  Using simulators can help out, a lot, enabling some degree of parallel development when hardware time is rare.  I encourage most of my customers to leverage the simulator wherever possible.  It's a good way to kick-start development, enable more hands on the project, and bring new engineers up to speed.

Once you have your hardware, there  may be some hardware-level debug issues.  Having a hardware debugging interface of some sort - an In Circuit Emulator (ICE), for example - can be invaluable.  As long as one has an LED to blink, one can write polled-output routines to display register settings (etc) to assist in debugging a hardware interaction, but nothing is as good as being able to dump desired address ranges (registers, etc) at will.  An ICE can give you that ability to see inside the hardware.

What do you do if the hardware is just not going to be available?  You can use similar boards, processors, etc, but.. it's not the *same* thing.  The minute you change the foundation, everything above it is going to have a new set of interactions with the foundation, and some of those may induce problems.  When updating hardware, timing, electrical, or even errata related to updated chips or even PAL equations may cause engineering delays - even if it's just a newer version of the same board.  Switching entire boards - or from engineering / test boards to "flight qualified" boards - can complicate the issue.

So.. what can you do when you don't have the real hardware on-hand, to help reduce risks associated with developing for that hardware?  In the past few years, hardware emulation tools have come a long way.  Hardware emulation is like a simulator - vxSim for example - except there's a layer of software that's designed to act *just like* the real hardware would, right down to devices and registers within those devices, etc. 

Leveraging known-good (mature) software designs is another way to limit risk.  Re-using application code that is well understood is one example.   Selecting a base of software that's been certified to adhere to accepted standards is another way.  It's always nice to know that software has been scrutinized by eyes other than the manufacturer's, and been found fit.  And there's no substitute for a rigorous test strategy, or following the oft-learned adage: test as you will fly, fly as you did test.

Safe design practices can't eliminate tiger teams completely - there will still be unknowns, still be conditions that aren't discovered until the last moment.  But by combining proper practices, mature systems, and the proper tools, many of the problems that lead to the need for tiger teams may be addressed, and may be discovered far enough in advance to prevent a project from running out of Time.