Mike Deliman

Remarkable

"Space... the final frontier" is how the pre-recorded message from NASA to the Crew of Discovery started this morning.

This afternoon while covering the final departure of Discovery from the ISS, the Newsman said "Remarkable - a remarkable bit of technology and a lasting mark on history" - or something to that effect. I thought "Remarkable" - that's a word Spock used to describe fantastic new technologies that had never been seen before. Things like Ion Drive in an autonomous space probe, or a life form that lives on something we'd considered impossible.

Some remarkable technologies demonstrated recently in the news include things like Northrop Gruman's X47B UCAS, and the comet flybys of Stardust NExT and DI Epoxi, and ongoing announcements of potentially hundreds of earth-like planets being found by Kepler Space Telescope.

One of the remarkable technologies just delivered to the ISS by Discovery is Robonaut R2. R2 is a telepresence operated robotic astronaut / assistant. It's the first robot designed to use the same tools as the astronauts, and to be able to assist them inside and outside on space missions.

Some other note worhty technology demonstrations/events coming up - later this month  MESSENGER will orbit Mercury.  In July  Dawn will go into orbit around asteroid Vesta.  Both of these orbital insertions represent "firsts" for mankind - the first time we've orbited and surveyed the sun-side of Mercury and the first time ever orbiting an asteroid.

All of these devices contain within them a bit of technology called VxWorks.  It's probably being used somewhere in the infrastructure that gets this posting to your monitor.  VxWorks has unparalled maturty and flexibility, going into everything from that internet modem in the closet to autonomous vehicles and robots.  I think that's also kind of remarkable...

Radioactive Telepresence

How timely!  A couple of months ago the discussion started - "will unmanned vehicles make a transition into civilian use".  I've been taking the stance that since we're talking vehicles - not just aircraft but all forms of non-stationary robot, that it is inevitable.  Even with aircraft I believe it is inevitable, though it may take a little longer for unmanned / automated aircraft to be certified for use in civilian airspace.

It would make sense that robots would be deployed for things that are either impossible for humans to do, or for things that are hazardous and dangerous.  On the impossible-for-humans side, quick return deep-dive missions in the ocean, and several-day long monitoring missions come to mind, as well as some interesting possibilities for telepresence tourism.  The hazardous side is easy to imagine - everything from maintenance of city infrastructures to handing toxic or radioactive substances would be fair game to use robots for, as well as underground mining.

Among some of the things I've thought about are robots with a divide-and-conquer design.  Some robots would be specialized worker robots, others might act as relay points to extend the reach of the overall system, and some functionality might overlap.

Just a couple of days ago, Bill posted about INTRA Groupe's Hazardous Materials handling robots.
This is an excellent example of a small herd of robots designed to perform complementary tasks. There's even a unit specially designed to act as a data relay, to extend the reach of the telepresence system.  Though these robots are currently remote-operated, as the need arises and becomes evident I would expect each system to be given some degree of autonomous control, in order to augment the abilities of their human controllers to achieve their ultimate goal.

Quit Bugging Me: Making Maps

A tool commonly used in embedded debugging is a linker map - a map of where all the symbols are in the runtime image.  These maps are useful as they turn raw addresses reported by some exception stubs (etc) into offsets into the data or text (program routines) in the computer's RAM.  They give you an idea of what may have been happening when the error occurred.

Producing a linker map is fairly easy.  Most linkers include command line options to produce a map.  This works fine and is very clear when used from a command line.  But things can get a little confusing from within an integrated gui environment.

The Workbench environment makes it very easy to create and use projects, and glue them together.  A problem that can occur, is since you build all projects by selecting "build project", you may not be sure of what tools are being used by the builder "under the hood".

A recent question we had was "How do I produce a linker map?  I provided the following flags to get the linker to make a map:  -Xlinker -map mapfile.  Instead of getting my map file, I got an error."  The error the customer was getting was "ldppc: cannot find -linker".

Thinking about the problem a bit I came to the realization that the customer was calling ldppc with the flags that tell ccppc "hand these options to the linker".  The problem is that the linker, ldppc, called with these flags directly, decided they meant something else.  The real problem is a misunderstanding that the project being modified uses the linker directly, instead of the compiler front-end.  I decided it would help to demonstrate.

To the compiler, "-Xlinker blah blah" tells the compiler "call the linker with the flags blah blah".

-Xlinker  means something completely different to the linker.  -X tells the linker "discard symbols".  -l<something> tells the linker "link in the library named <something>".  The kind of funny thing is that using -Xlinker, the linker was looking for a library named "inker", libinker.a .  Since the directive failed, the linker sort-of seemed to be complaining that it couldn't find itself!  Here's a demonstration of it: note that to complete the demonstration I needed to add a directive "look here for libraries" (-L.)

For the demonstration, I launched the Developer's Environment:

Q:\WIND\VX67WB31>ldppc --help | grep -i map
  -M, --print-map             Print map file on standard output
  -Map FILE                   Write a map file
Q:\WIND\VX67WB31>ldppc --help | grep X
  ...
  -X, --discard-locals        Discard temporary local symbols (default)
  ...
Q:\WIND\VX67WB31>touch libinker.a

Q:\WIND\VX67WB31>ldppc -L. -Xlinker
ldppc: warning: cannot find entry symbol _start; not setting start address

Q:\WIND\VX67WB31>rm libinker.a

Q:\WIND\VX67WB31>ldppc -L. -Xlinker
ldppc: cannot find -linker

More Bacon: increasing science return

In a recent blog I wrote, Bringing Back The Science Bacon,  one of the things I talk about is increasing the science returns of a mission by doing some data processing on the mission robot, before it sends back information.  If a computer has the ability, and the trade-offs gathering data, processing it into information, available computer throughput and bandwidth of the data channel are favorable for processing, data can be processed into information and the mission may return more relevant science.

Some great examples come from the Mars Exploration Rovers.  NASA/JPL has managed to operate the rovers for over seven years.  We've sent up a number of software updates to both rovers.  Updates include new capabilities, and new ways to look for science.

Here's another great example of increasing the science return by doing a little more processing on-board.  The rovers already have some autonomy, with abilities to detect and plot ways around obstacles.  By increasing autonomy, the AEGIS package gives the rovers the ability to detect targets of interest to science, and adjust their courses to investigate.  (Dear Editors of Aerospace-Technology.com - as orbiting entities, the Mars Rovers have very low orbits...)

This is exiting.  Among other things, AEGIS will enable the rovers to return science we would otherwise have missed.  In the old model, the rovers drive along, sometimes for extended periods, only taking and returning samples we tell them to.  With AEGIS, the rovers can now "sense" that a given object is scientifically interesting, decide to visit it, and return photos so the scientists can decide if it warrants a visit.  This is how NASA accelerates science.

And talk about a mission getting the bang for it's bucks?  This three month mission is 7 years into it's calendar, and is experimenting with ways to increase the science it currently returns, and science returns for future missions.  Very cool!

And The 2010 SpaceOps Award for Outstanding Achievement goes to...

Until today, I had never heard of The International Committee on Technical Interchange for Space Mission Operations.  Or the SpaceOps Awards.  An announcement arrived in my email this afternoon.  Guess who/what got the award?  Go on..  :-)  CONGRATULATIONS!

Quit Bugging Me: Revalidated

One day, I get this phone call.  "We're working with the system, we see the calls that update the exception handlers early on - connecting the clock routines, etc.  Then not very much farther on we see the system has run past where tasking should be running but it's not.  When we check, we're not getting any clock interrupts, and the clock exception handler isn't there any more.  How could the system do that?  Is there any way the OS could remove it's clock connections?"

The complete story was even more eyebrow raising.  "We're testing for a launch, this vehicle is a one-of-a-kind, there's no backups.  We're doing environmental limits testing.  We're still within spec, but operating at the low end of the electrical ratings.  We need to explain this anomalous behavior or scrub."

This was the start of a frantic week of work.  On PowerPC, the architecture stores it's exception vectors down in low ram.  A valid method of updating these vectors is to write new data to the vector address as data, then force the data cache to be pushed to the RAM, and finally tell the processor "any instructions you have cached for this area are invalid" (invalidate the I-cache).  This last step ensures that the CPU will fetch the newly-updated instructions the very next time they're called.

The hardware involved was fairly rare and expensive. We didn't have a lot of options available for hardware debug assist.  In fact, I had never even seen the hardware in person, all my work was theoretical and experiments were done with "similar" boards.

I used what I had to trace the execution sequence to the point in question, where we should have been seeing clock ticks come through. With all the hardware I had, there was no way I could recreate the problem.  In every case, I had clock ticks come through as expected, and with code to set the clock to an expected value earlier on, every single run showed the same exact results.  It was impossible for it to /not/ work.

The sequence, boiled down, that was under investigation, runs like this:

The runtime starts.  Vectors are written-to / populated with default values and routines.  Caches are invalidated and enabled. Board hardware is initialized.  A clock is set up, it's exception vector is populated via a data-write; the D-cache is "flushed" to RAM, the instruction cache is told "this area of RAM is updated" (invalidated), and we continue running.  1/60th of a second later we should get a clock tick, it should be processed through this exception vector and ....

The problem seemed to be that the exception vector was running the same old code - not the new code.  It could be checked that the new code had actually been pushed all the way into RAM - except we didn't have that capability at the time nor enough time to set it up.

In every other test setup, we saw the expected behavior - the clock tick delivered and tasking running exactly as expected.  In this particular setup, it was as if the clock tick never happened.  The CPU side of things clearly ran as if the code had never, ever been invalidated.

As a point of interest, we found the code executed was smaller than the instruction cache.  We'd never exhausted the cache.  We formed a theory, hard to prove, but interesting.  Running at the lower voltage end, the units under test were all the -same-, hardware wise.  With the lower voltage, was it possible that some anomaly of the hardware caused the instructions that had been invalidated to... somehow.. not be invalid?  The only way we could still be executing those instructions is if there were still in the I-cache, and we'd invalidated them.  Perhaps the hardware somehow revalidated the cache when run at low voltages?

Bringing power back up to spec showed normal execution.  Working with the hardware vendor we learned that there was a probability that the lab grade hardware might show anomalies not exhibited by deployment grade hardware, and indeed the lab grade hardware would fail tests that the deployment grade hardware passed without problems.

Testing was repeated on deployment grade hardware, which ran without incident.

Looking back, had I known then what I know now, it would have been expeditious to execute a field of no-ops the size of the I-cache immediately after installing the timer vector.  This would have caused the entire I-cache to be invalidated/populated with new entries, flushing the timer vector code completely.  The next execution of that vector would have forced the I-cache to re-read the whole vector, we would have seen the clock ticks where expected, and we could have explicitly tested the I-cache in this way.

Do you design flight software?

Normally I don't like this blog to sound like a marketing microphone, BUT... once in a while something comes along that's Worth It.  I'll keep it short and sweet.

Next month, Larry Kinnan will be running a web presentation about how VxWorks 653 OS is used in the Boeing 787 Dreamliner.   Click here for more information. 

I've learned a lot from Larry over the years.  I ask him the tough questions about ARINC and flight certification that customers ask me, he's one of the experts who back me up when I don't have the answers.

Sand Trap

Spirit Rover is heading to her sixth anniversary on Mars, but it looks like she'll have to sit the dance out.  Spirit has been trapped, basically high-centered on a hard spot surrounded with flour-like soil.  Some surprising test results show that one of the wheels we thought was jammed has some slight function, but the bad news is one of the other wheels no longer functions at all, the motor is burned-out.  Between the high-center, and the dead wheels, it doesn't look like Spirit is going to be able to dislodge herself.  On top of this bad news, Winter is coming.

The rovers are solar powered.  We've used tricks the past few winters to maximize that incoming power - mostly, parking the rovers so they are angled towards the sun.  It's a critical step to making them last so long.

In the warm summer days,  temperatures can reach a balmy 70F.  The long days generate plenty of power, and we're able to rove about and do science on the surface.  The nights get a bit cold, but there are heaters on board which can be run to keep the rovers' brains from freezing.  In Summer we have plenty of power to run those heater as needed, except perhaps during the occasional several-month-long dust storm.

In the Winter, temperatures can plummet to below -150F, colder than dry-ice.  Temperatures this cold can precipitate mechanical failures - for instance, there are glass lenses in the rovers which could literally break just from getting that cold. Metal fatigue becomes amplified.  Though there are heaters, with shorter, colder days, the heaters must run longer to maintain a safe temperature.

Spirit is trapped.  She can't reposition herself to angle the panels towards the sun.  With the increased needs for power over the Winter, this is not good news.

It's somewhat fitting that in the process of getting trapped, she made more important science discoveries about the surface of Mars.  It's sort of a last cry of triumph for the old girl.  She's almost six years on-planet, and six years in Rover-time is about 25 lifetimes in people years...   :)

Bit Rot Amnesia

Spirit Rover has been having a number of problems recently, anomalous behaviors, losing contact with Earth, even forgetting commands.  Amnesia has been one of the larger concerns.

At the heart of this issue is a bank of flash memory.  Flash pretty much works by saturating cells to "1"s, then writing the cells that need to be 0's.  "quantum wells" hold the charge for later recall.  Since this is all done with minute electrical charges, one would have to wonder how long it could last.

Over time, the charge in the "1's" cells is bound to leak.  High energy particles from the Sun and other sources may also strike parts of the flash, inducing charges, corrupting the contents of the flash.  Over time, some cells may flip from 0 to 1, etc, or become "indeterminate", reading 0 some times, and 1 others.  If such degradation was affecting any of the data structures or other rarely-written data, it could contribute to an apparent loss of data.  Even though the file system in use performs wear leveling (a process to "spread usage out" in a bank of flash memory), the data being preserved must be written for wear-leveling to come into play.  It is possible that some of the configuration and directory structure data may have been degraded over time.  This degradation is called "bit rot".

The only way to fight "bit rot" is to re-write the data with high confidence.  The best way to do this is to "erase" the entire bank of flash, and install a new file system.  The reformat command was issued to do this job.

Erasing and formatting a flash drive is among the most power intensive operations the rover has on Mars.  It is not surprising that it took 3 days for the results of the process.  Day one we format the drive, and get results of that.  Day two we do some test exercises and store the data in flash.  Day 3 we attempt to recover that data.

I guess now we have one solid metric for how long data can survive in a flash file system on Mars!

Congratulations to the MER team, on recovering saved telemetry from the flash drive.

Quit Bugging Me: ABI

What's in an ABI?  An Application Binary Interface is a lot like a API - Applications Programming Interface, except instead of just telling you how to make a call, what parameters to provide, and what returns and errnos to expect, an ABI also tells you something about how the interfaces work.  Understanding an ABI for a CPU means you understand how the registers are used, what they're for, and what they mean under different contexts.

Among the things ABIs give you is how your CPU formats memory (the stack) when it makes a call to a routine. An easy way to take a look at an ABI is to set a break point on a routine, and look at what the stack contains.  This is on a PowerPC target.  First I'll spawn a task, then extract the current information about the task - stack pointer, stack base, etc, using target-side debug routines.

The PowerPC ABI gives specifications for how and when routines use the stack to store values.  I'll use some knowledge of the ABI and values from the target-side debug routines to show you a simple ABI trick.

I've spawned:  printf("this is a test. %d %d %d %d %d %d \n", 1, 2, 3, 4, 5, 6) .

I've set break a break point on "fioFormatV" for this demonstration, so I can show you how printf was called.  When printf() calls fioFormatV() it has to store some of it's work on the stack.  It does this so fioFormatV() can do work without destroying the data printf is supposed to handle.  Let's look at some task information and the contents of the stack.  I've removed some output we're not concerned with for this example.

-> Break at 0x00053068: fioFormatV          Task: 0xfff9c20 (t1)

-> ti

  NAME         ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
----------  ------------ -------- --- ---------- -------- -------- ------- -----
t1          printf        fff9c20 100 STOP          53068  fff9b80       0     0

task stack: base 0xfff9c20   end 0xfff4e00   size 20000  high 160    margin 19840

r0         = 0x0002cae4   sp         = 0x0fff9b80   r2         = 0x00000000
r3         = 0x0fff4dbc   r4         = 0x0fff9ba8   r5         = 0x000545a4
r6         = 0x00000001   r7         = 0x00000004   r8         = 0x00000005
r9         = 0x00000000   r10        = 0x00000000   r11        = 0x0fff9b88
r12        = 0x0000000d   r13        = 0x00000000   r14        = 0x00000000
r15        = 0x00000000   r16        = 0x00000000   r17        = 0x00000000
r18        = 0x00000000   r19        = 0x00000000   r20        = 0x00000000
r21        = 0x00000000   r22        = 0x00000000   r23        = 0x00000000
r24        = 0x00000000   r25        = 0x00000000   r26        = 0x00000000
r27        = 0x00000000   r28        = 0x00000000   r29        = 0x00000000
r30        = 0x00054474   r31        = 0x00000000   msr        = 0x02029230
lr         = 0x000544d8   ctr        = 0x00054474   pc         = 0x00053068

-> d 0xfff9b80
0x0fff9b80:  0fff9bc0 eeeeeeee 0fff4dbc 00000001  *..........M.....*
0x0fff9b90:  00000002 00000003 00000004 00000005  *................*
0x0fff9ba0:  00000006 00000000 01000dee 0fff9bc8  *................*
0x0fff9bb0:  0fff9b88 eeeeeeee eeeeeeee eeeeeeee  *................*
0x0fff9bc0:  0fff9be0 0002cae4 00000000 00000000  *................*
0x0fff9bd0:  eeeeeeee eeeeeeee 00000000 00000000  *................*
0x0fff9be0:  00000000 00000000 eeeeeeee eeeeeeee  *................*
0x0fff9bf0:  eeeeeeee eeeeeeee 0fff4dbc 00000001  *..........M.....*
0x0fff9c00:  00000002 00000003 00000004 00000005  *................*
0x0fff9c10:  00000006 00000000 00000000 00000000  *................*
0x0fff9c20:  00000000 0fff9c20

-> tt t1
0x0002cae4 vxTaskEntry  +0x5c : printf (0xfff4dbc, 0x1)
0x000544d8 printf       +0x64 : fioFormatV ()

Okay... looking at the task trace, we see printf was called with some values.  The first value doesn't look like our string, it looks like an address.  This should be the address where our string is... that's what the ABI says:

-> d 0xfff4dbc
NOTE: memory values are displayed in hexadecimal.
0x0fff4db0:                             74686973  *            this*
0x0fff4dc0:  20697320 61207465 73742e20 20256420  * is a test.  %d *
0x0fff4dd0:  25642025 64202564 20256420 25640a00  *%d %d %d %d %d..*

Yes... that's the format string for printf.  From the task trace there's only one additional argument. But looking at values stored on the stack - starting at the address in the "stack pointer" address from the "ti" task information we see some things we recognize... all in a "row" there is the address of the string above, and the numbers 1 through 6 - this is the arguments passed in to printf.  At the very first address, there's an odd value - 0fff9bc0 - this address is between the stack pointer and stack base.  It's called a back-chain: it points back to the previous stack frame.  Following back-chains back, fff9bc0 leads to 0fff9be0, which contains 0.  But.. we're still not back to the base of the stack yet.  What gives?  This area - between the "0" valued back-chain and the base of the stack, is the initial stack frame which is used to preserve the arguments originally handed to taskSpawn().  The first stack frame is the part of the stack used by a routine referred to as "vxTaskEntry" in the "task trace" output.  This previous part is where taskSpawn() holds the original values for taskRestart().

From the PowerPC ABI, we can see that when one routine calls another, standard stack usage is to preserve the data the caller is going to need later on, by pushing it onto the stack.  printf is calling fioFormatV, printf is responsible for saving some registers, so it puts them on the stack (look up "caller-saves" and "callee-saves"). It's easy enough for us to identify the arguments we handed to taskSpawn for printf to use on the stack.

Looking back at the top stack frame - between fff9b80 and fff9bc0, we can change these values and affect what gets printed.  It should print "1 2 3 4 5 6". Let's change the 6 to a 9.  After changing the value, we'll "c" continue the execution and see what it prints.  In that top stack frame, "00000006" is stored at 0xfff9ba0, so we'll modify memory there:

-> m 0xfff9ba0
0x0fff9ba0:  0x0000-
0x0fff9ba2:  0x0006-0009
0x0fff9ba4:  0x0000-.

value = 0 = 0x0
-> c
value = 0 = 0x0
-> this is a test.  1 2 3 4 5 9

By changing the value printf() had saved on the stack according to the ABI, we changed the value printf actually printed out.

By using the ABI and the contents of memory, you can see if something has happened to corrupt the arguments given to a routine you're interested in testing.  You can also use this kind of method to force erroneous values into a routine for testing purposes.  Given that the ABI specifies a routine will store values to the stack when it calls other routines, you may need to find the first call-out performed by the routine you are interested in (as I did above using fioFormatV, which is called by printf).