Mike Deliman

And Then There Was One

NASA CONCLUDES ATTEMPTS TO CONTACT ROVER 'SPIRIT'

Last night the news of Spirit's mission coming to a close hit my mailbox.

For me, her journey started about eleven years ago.  A call from JPL came to my office phone, it was Glenn.  "Mike, we're putting the team back together, we're going back to Mars."

Those words were like music to my ears.  Completing the Mars Pathfinder project the thought occurred... "this, then, is the pinnacle of my career, the point  where I can mark as the top of my achievements."  This call changed all of that. There were other projects along the way, Gravity-Probe B, and others, but Pathfinder had been fantastic!

Over 3 long years we worked as a team, cogs in a bigger machine making something incredible.  We all put in long hours, working all day on our mission, dreaming at night of what we were working on (it's a human thing).  They began to take on personalities of their own, MER-A being feisty as the "older twin", MER-B cooperating and living the charmed life of the youngest child.  Then the
competition where the children gave our babies their names- Spirit and  Opportunity.  These seemed to be very fitting of our offspring, Spirit surely was full of fitfulness, where as Opportunity was always moving forward.

The only point where Oppy was contentious was Launch Day.  Though she herself was ready, her rocket kept having one problem or another.  We assembled several times at Jetty Point to watch her ride her candle into the cosmos, alas my 2 weeks were up, I had to leave, and Oppy was still on the pad.

It was decided in order to make the launch window we would "borrow" Spitzer's Delta rocket.  That gave us the go-power to get to Mars on time.  And we did.

"We".  after living with the rovers in mind day and night for so long, the team began to identify with our robots, their efforts were ours, a shared thing. Like the pride you feel watching your child graduate, first from grade school, then high school.  It was like we were sending them off to College.  Imagine sending your 3-year-olds off to Cal Tech or MIT.

Seven months later came the graduation, in the "6 minutes of Terror" landing. They were now on Mars, where they would unfold their wings (solar panels) and begin their flight into their professional lives.

Crawling across the alien world they were on, Spirit again showed her fitfulness on SOL18, and later, as she suffered various maladies and growth-pains.  Oppy on the other hand, landed in a bowl full of science, and fulfilled the  Mission Goals all at once.  Layered rocks and "blueberries" were found within meters of her landing spot, proof that Mars had once had oceans, had once been very much earth-like in the early years.

It's been 7 years now, into their 3-month mission.  They've completed their mission 28-times-over.  Both rovers found proof of the earth-like Mars, in rocks, salts, and formations.  Opportunity kept finding new places to explore, new wonderful things to share with us.  Spirit did as well, but suffered  arthritis (lost a wheel) and alzheimers (forgetting things writting to long-term storage).

Opportunity is currently rolling her way towards another distant crater, sporting improved software that can find interesting rocks for us to inspect, and image-identifying code that finds dust-devils in her landscape pictures. These small bits of improved software increase her science return.

With today's annoucement, Opportunity rides alone now.  Spirit has parked on a bad spot, we couldn't give her the extra energy she needed to winter over.

 

Spirit sleeps now forever in her spot on the Red Planet.

Rendezvous with a Rock

MESSENGER Spacecraft is approaching it's rendezvous with Mercury.  After flying for over 2400 days, in a little over 24 hours (from writing this)  MESSENGER will maneuver into orbit, the first time a satellite has orbited Mercury.  We hope to learn a lot about the composition of the surface and effects of solar weathering of the surface.  And there's the prospect of photographing the sun-side of the planet for the fist time!

In a few months Dawn Spacecraft will orbit Asteroid Vesta - the first time we've orbited an asteroid.  We hope to learn more about the composition of asteroids.   (We've landed on one, but technically it's harder to orbit than to land.  We've also successfully landed on a comet with Deep Imact...)  Dawn has not been in flight as long as MESSENGER, but it's orbital attempt is none the less impressive (Vesta is much smaller than Mercury).

Over the last ten years we've redefined our undertstanding of how Mars evolved to it's present day, learned an incredible amount about the Sun, Moon, and Jovian planets, and even changed our definition of what "habitable" is - of where and how we should look for signs of life.  All of this helps change our understanding of how the solar system works, how planets work, and how we fit in the picture.

It fascinates and amazes me that we can design computers and robots capable of performing these incredible feats.  There's a software logo.. "where do you want to go today?" .  Where do you want to go tomorrow, or in five to ten years?  I wonder what amazing things we will learn from the Sunlit side of Mercury.

VxWorks 5.3.1 for Rad6000 runs on both MESSENGER and Dawn

 

Remarkable

"Space... the final frontier" is how the pre-recorded message from NASA to the Crew of Discovery started this morning.

This afternoon while covering the final departure of Discovery from the ISS, the Newsman said "Remarkable - a remarkable bit of technology and a lasting mark on history" - or something to that effect. I thought "Remarkable" - that's a word Spock used to describe fantastic new technologies that had never been seen before. Things like Ion Drive in an autonomous space probe, or a life form that lives on something we'd considered impossible.

Some remarkable technologies demonstrated recently in the news include things like Northrop Gruman's X47B UCAS, and the comet flybys of Stardust NExT and DI Epoxi, and ongoing announcements of potentially hundreds of earth-like planets being found by Kepler Space Telescope.

One of the remarkable technologies just delivered to the ISS by Discovery is Robonaut R2. R2 is a telepresence operated robotic astronaut / assistant. It's the first robot designed to use the same tools as the astronauts, and to be able to assist them inside and outside on space missions.

Some other note worhty technology demonstrations/events coming up - later this month  MESSENGER will orbit Mercury.  In July  Dawn will go into orbit around asteroid Vesta.  Both of these orbital insertions represent "firsts" for mankind - the first time we've orbited and surveyed the sun-side of Mercury and the first time ever orbiting an asteroid.

All of these devices contain within them a bit of technology called VxWorks.  It's probably being used somewhere in the infrastructure that gets this posting to your monitor.  VxWorks has unparalled maturty and flexibility, going into everything from that internet modem in the closet to autonomous vehicles and robots.  I think that's also kind of remarkable...

Houston.... the Falcon has Flown!

Last week something remarkable happened. I'm not talking about the UFO sighted over Brisbane (...or am I?)  Last week Spacex successfully orbited the first successful commercial attempt to launch such a device. Wind River is proud to be a part of this historic endeavor.

It's hard to convey how exciting this is.  This represents a HUGE step for mankind, a transition of the technology necessary for space exploration now being handled by commercial entities.  Something like this may be our fledgling answer for "what to do about Manned exploration now that ARES is cut".

With permission, I'll reprint an entry from a space software engineer's blog entry as he watched the launch... (note: the "e#" marks are edit points as the engineer updated the blog in real-time.)

-----

http://www.spacex.com/webcast.php
has just come off a 15:00 hold....
[e1] it's been on hold for several hours.
[e2] 11:00 and counting...
This is a new commercial launch vehicle.
[e3]
Terminal count launch started.

T- 10:00
[e4] count continuing
changing camera angles, etc.
It's a cloudy day at the cape.

T- 9:00
[e5]
Standing by for firing....
T- 8:00
chatter about status

T- 7:00
[e6]
checking valves, etc, for propellant
more chatter. Still venting LOX.
T- 6:00
[e7]
Fully charged... on internal power.
transmitter working

T- 5:00
[e8]
Interrupters rotating arm. (?)
Count continuing.
T-4:00
[e9]
LOX venting has increased
there is venting at the bottom as well now
T-3:00
[E10]
T-2:30 LD GO FOR LAUNCH
[E11]
T-2:00
RCO verify range go - RANGE GO!
[E12]
(more people are on the stream, it's having problems now)
(T-0000)
IT'S UP!
[E13]
Missed the last minute of count - but I did get to see the actual launch. The Falcon 9 is on it's way up.
[E14] They seem to be having more feed problems (or I do). There - split screen now. They've passed MaxQ. The plume behind the rocket is feathered-out as expected in very thin atmosphere. (T+3:05)
[E15]
Nominal function at t+4:15 . Second stage is in flight. The main rocket motor is glowing hot.
Everything is running Nominally (except the error on the web site).
The rocket nozzle is glowing white hot in some places - this is expected. Everything is performing nominally.
[E16]
Telem acquired by New Hampshire. There are apparently grass fires near the hangars. The camera on the rocket appears to be functioning well, the rocket is in a slow roll. So far it's looking all good.
Missed 2nd stage separation and ignition.
Right now there's a feed error...
[E17]
The launch has been a success. More info, photos, and video will be posted to spacex as it becomes available.

------

Note: there are video highlights available at the SpaceX web site, as well as more information about Falcon, Dragon, and Spacex.  CONGRATULATIONS SPACEX!!

Radioactive Telepresence

How timely!  A couple of months ago the discussion started - "will unmanned vehicles make a transition into civilian use".  I've been taking the stance that since we're talking vehicles - not just aircraft but all forms of non-stationary robot, that it is inevitable.  Even with aircraft I believe it is inevitable, though it may take a little longer for unmanned / automated aircraft to be certified for use in civilian airspace.

It would make sense that robots would be deployed for things that are either impossible for humans to do, or for things that are hazardous and dangerous.  On the impossible-for-humans side, quick return deep-dive missions in the ocean, and several-day long monitoring missions come to mind, as well as some interesting possibilities for telepresence tourism.  The hazardous side is easy to imagine - everything from maintenance of city infrastructures to handing toxic or radioactive substances would be fair game to use robots for, as well as underground mining.

Among some of the things I've thought about are robots with a divide-and-conquer design.  Some robots would be specialized worker robots, others might act as relay points to extend the reach of the overall system, and some functionality might overlap.

Just a couple of days ago, Bill posted about INTRA Groupe's Hazardous Materials handling robots.
This is an excellent example of a small herd of robots designed to perform complementary tasks. There's even a unit specially designed to act as a data relay, to extend the reach of the telepresence system.  Though these robots are currently remote-operated, as the need arises and becomes evident I would expect each system to be given some degree of autonomous control, in order to augment the abilities of their human controllers to achieve their ultimate goal.

Quit Bugging Me: Making Maps

A tool commonly used in embedded debugging is a linker map - a map of where all the symbols are in the runtime image.  These maps are useful as they turn raw addresses reported by some exception stubs (etc) into offsets into the data or text (program routines) in the computer's RAM.  They give you an idea of what may have been happening when the error occurred.

Producing a linker map is fairly easy.  Most linkers include command line options to produce a map.  This works fine and is very clear when used from a command line.  But things can get a little confusing from within an integrated gui environment.

The Workbench environment makes it very easy to create and use projects, and glue them together.  A problem that can occur, is since you build all projects by selecting "build project", you may not be sure of what tools are being used by the builder "under the hood".

A recent question we had was "How do I produce a linker map?  I provided the following flags to get the linker to make a map:  -Xlinker -map mapfile.  Instead of getting my map file, I got an error."  The error the customer was getting was "ldppc: cannot find -linker".

Thinking about the problem a bit I came to the realization that the customer was calling ldppc with the flags that tell ccppc "hand these options to the linker".  The problem is that the linker, ldppc, called with these flags directly, decided they meant something else.  The real problem is a misunderstanding that the project being modified uses the linker directly, instead of the compiler front-end.  I decided it would help to demonstrate.

To the compiler, "-Xlinker blah blah" tells the compiler "call the linker with the flags blah blah".

-Xlinker  means something completely different to the linker.  -X tells the linker "discard symbols".  -l<something> tells the linker "link in the library named <something>".  The kind of funny thing is that using -Xlinker, the linker was looking for a library named "inker", libinker.a .  Since the directive failed, the linker sort-of seemed to be complaining that it couldn't find itself!  Here's a demonstration of it: note that to complete the demonstration I needed to add a directive "look here for libraries" (-L.)

For the demonstration, I launched the Developer's Environment:

Q:\WIND\VX67WB31>ldppc --help | grep -i map
  -M, --print-map             Print map file on standard output
  -Map FILE                   Write a map file
Q:\WIND\VX67WB31>ldppc --help | grep X
  ...
  -X, --discard-locals        Discard temporary local symbols (default)
  ...
Q:\WIND\VX67WB31>touch libinker.a

Q:\WIND\VX67WB31>ldppc -L. -Xlinker
ldppc: warning: cannot find entry symbol _start; not setting start address

Q:\WIND\VX67WB31>rm libinker.a

Q:\WIND\VX67WB31>ldppc -L. -Xlinker
ldppc: cannot find -linker

More Bacon: increasing science return

In a recent blog I wrote, Bringing Back The Science Bacon,  one of the things I talk about is increasing the science returns of a mission by doing some data processing on the mission robot, before it sends back information.  If a computer has the ability, and the trade-offs gathering data, processing it into information, available computer throughput and bandwidth of the data channel are favorable for processing, data can be processed into information and the mission may return more relevant science.

Some great examples come from the Mars Exploration Rovers.  NASA/JPL has managed to operate the rovers for over seven years.  We've sent up a number of software updates to both rovers.  Updates include new capabilities, and new ways to look for science.

Here's another great example of increasing the science return by doing a little more processing on-board.  The rovers already have some autonomy, with abilities to detect and plot ways around obstacles.  By increasing autonomy, the AEGIS package gives the rovers the ability to detect targets of interest to science, and adjust their courses to investigate.  (Dear Editors of Aerospace-Technology.com - as orbiting entities, the Mars Rovers have very low orbits...)

This is exiting.  Among other things, AEGIS will enable the rovers to return science we would otherwise have missed.  In the old model, the rovers drive along, sometimes for extended periods, only taking and returning samples we tell them to.  With AEGIS, the rovers can now "sense" that a given object is scientifically interesting, decide to visit it, and return photos so the scientists can decide if it warrants a visit.  This is how NASA accelerates science.

And talk about a mission getting the bang for it's bucks?  This three month mission is 7 years into it's calendar, and is experimenting with ways to increase the science it currently returns, and science returns for future missions.  Very cool!

And The 2010 SpaceOps Award for Outstanding Achievement goes to...

Until today, I had never heard of The International Committee on Technical Interchange for Space Mission Operations.  Or the SpaceOps Awards.  An announcement arrived in my email this afternoon.  Guess who/what got the award?  Go on..  :-)  CONGRATULATIONS!

Quit Bugging Me: Another Surprise NaN!

An Earlier QBM, "Surprise NAN" covered how floating point computations may become corrupted from unexpected sources.  Today's Quit Bugging Me is about... another Surprise! NAN!

An application runs with several tasks.  One task, which computes a set of floating point values, periodically comes up with bad values, and sometimes the name becomes corrupted.  

Here is a short example...  done with the solaris simulator.  Kernel options do -not- include any hardening, nor any guard pages / stack protection.  In fact, only the options related to "MMU_BASIC" are included.  I have a breakpoint set just before the exit so I can catch overflows.  surpriseNAN recurses on it's argument, easily overflowing the default stack size.  I call logMsg first thing, it's a cheap way to get the task ID printed out.  I'm going to provide edited kernel shell output to show the points.

Symptoms:  the task names are valid on entry but suddenly go away, there's a crash that involves "reschedule", the values contained in the FPregs are... corrupted badly:

-> task spawned: id = 0x40b4820, name = t2

task spawned: id = 0x40bb968, name = t3

task spawned: id = 0x40ad6d8, name = t4

task spawned: id = 0x40a6590, name = t5

task spawned: id = 0x409f448, name = t6

task spawned: id = 0x4098300, name = t7

0x40bb968 (t3): alive.

0x40ad6d8 (t4): alive.

0x40a6590 (t5): alive.

sometime later the BP is hit.  Several of the tasks are "missing", though none completed.

> i

  NAME      ENTRY     TID    PRI   STATUS      PC       SP     ERRNO  DELAY

-------- ---------- -------- --- ---------- -------- -------- ------- -----

tExcTask   excTask   40d8250   0 PEND          a89cc  40d8058       0     0

tLogTask   logTask   40d37b8   0 PEND          a89cc  40d35c0       0     0

tWdbTask   5eea4     40ce308   3 PEND          664c8  40cdfd8       0     0

!       surpriseNAN  40bb968 100 READY         6a80c  40b4cb8       0     0

!       surpriseNAN  40ad6d8 100 READY         6a80c  40a6a28       0     0

       surpriseNAN   40a6590 100 SUSPEND       6a810  409f950       0     0

value = 0 = 0x0

Notice - the task t5 hit the breakpoint and was suspended.  The other two tasks are "ready" and never hit the break point.  They are no longer running though.  And what happened to the names?  Let's look at one of the tasks...

-> ti 0x40bb968

NAME       ENTRY      TID    PRI   STATUS      PC       SP     ERRNO  DELAY

-------- ----------- ------- --- ---------- -------- -------- ------- -----

        surpriseNAN  40bb968 100 READY         6a80c  40b4cb8       0     0

stack: base 0x40bb968  end 0x40b4c08  size 27856  high 27856  margin 0    

options: 0x1d

VX_SUPERVISOR_MODE  VX_DEALLOC_STACK    VX_FP_TASK          VX_STDIO            

%pc    =    6a80c   %npc   =    6a810   %psr   =        0   %mask  =        0

%tbr   =        0   %y     =        0                                        

%g0    =        0   %g1    =       5f   %g2    =        0   %g3    =  40ad6d8

%g4    =        0   %g5    =        0   %g6    =        0   %g7    =        0

%i0    =        1   %i1    =    db800   %i2    =        0   %i3    =        0

%i4    =        0   %i5    =        0   %fp    =  40b4d30   %i7    =  40d9d60

%l0    =  40bb968   %l1    =        0   %l2    =        0   %l3    =        0

%l4    =        0   %l5    =        0   %l6    =        0   %l7    =        0

%o0    =        0   %o1    =    db800   %o2    =        0   %o3    =        0

%o4    =        0   %o5    =        0   %sp    =  40b4cb8   %o7    =    6a804

%fsr   = eeeeeeee
%f00   = 0.793335   %f02   =        4   %f04   =        8   %f06   =        8
%f08   =  7.87933   %f10   = 0.00819755   %f12   = 0.223435   %f14   = -0.668483
%f16   = -0.902314   %f18   =        0   %f20   =        0   %f22   =      NaN
%f24   =      NaN   %f26   =      NaN   %f28   =      NaN   %f30   =      NaN
value = 67819224 = 0x40ad6d8

What happened here?  The FP-REGS are trashed?  and the name is trashed... something must have corrupted the system.  Is it a stack over-run?

-> checkStack

  NAME        ENTRY        TID     SIZE   CUR  HIGH  MARGIN

------------ ------------ -------- ----- ----- ----- ------

tExcTask     excTask      40d8250  15984   504  3016  12968 

tLogTask     logTask      40d37b8  15984   504  2104  13880 

tWdbTask     0x000005eea4 40ce308  16040   816 11376   4664 

!           surpriseNAN  40bb968  27856 27824 27856      0 OVERFLOW

!           surpriseNAN  40ad6d8  27856 27824 27856      0 OVERFLOW

            surpriseNAN  40a6590  27856 27712 27856      0 OVERFLOW

So in fact there is not only one stack overflow, but 3.  What has happened here?

Looking back at the TaskIDs returned from taskSpawn, we can see these were all spawned with their OS data structures right in a row.  With no memory protection (no memory protections included!) the task stacks are not protected.

When the surpriseNAN recursively writes over the top of it's own stack, it corrupts the area where the floating point data are saved.  Some of the values written work-out to be valid floating point numbers, others are not numbers.  This is near the area where the task name is stored.  With a sufficient overflow, the task name is corrupted as well as the FP data.

When the task overflows it's stack far enough, it will eventually over-write other data.  In this case the other data is the register set of the task immediately adjacent (in lower addresses) in RAM.  

Something to notice in the corrupted FP-REGS - the FSR is set to a stack-fill value (so the FSR screams "eeeeeeee!").   This is one more tell-tale that the context for this task has been corrupted.

100 Trillion

"Billions and Billions" was a phrase, fantastic at the time, that made Carl Sagan famous.  At the time he was talking about the stars in the Milky Way, how many there were.

Next week, Mars Recon Orbiter, will be marking it's 4th anniversary orbiting Mars.  It also has another milestone to celebrate:  MRO has brought back more data to Earth from Mars... than *all* deep space missions combined.  Not just missions to Mars, but all deep-space missions!

The ability to return data is crucial to the success of a mission - as is the quality of the data returned.  A mass of data with no information to it is useless.  In any case, MRO is able to relay data from other sources as well as it's own data.

To date, MRO has returned over 100 terabits so far, 100 trillion bits of data, from the Red Planet.  Enough data to provide us a pretty through education about the wet past of Mars, it's cyclic nature, and where to keep looking for more interesting stuff on Mars.

Enough data to make a 35 hour long HD video! That's a lot of data...

Happy Anniversary to MRO and the entire team!  Congratulations on reaching 100 Trillion Bits.