Mike Deliman

Houston.... the Falcon has Flown!

Last week something remarkable happened. I'm not talking about the UFO sighted over Brisbane (...or am I?)  Last week Spacex successfully orbited the first successful commercial attempt to launch such a device. Wind River is proud to be a part of this historic endeavor.

It's hard to convey how exciting this is.  This represents a HUGE step for mankind, a transition of the technology necessary for space exploration now being handled by commercial entities.  Something like this may be our fledgling answer for "what to do about Manned exploration now that ARES is cut".

With permission, I'll reprint an entry from a space software engineer's blog entry as he watched the launch... (note: the "e#" marks are edit points as the engineer updated the blog in real-time.)

-----

http://www.spacex.com/webcast.php
has just come off a 15:00 hold....
[e1] it's been on hold for several hours.
[e2] 11:00 and counting...
This is a new commercial launch vehicle.
[e3]
Terminal count launch started.

T- 10:00
[e4] count continuing
changing camera angles, etc.
It's a cloudy day at the cape.

T- 9:00
[e5]
Standing by for firing....
T- 8:00
chatter about status

T- 7:00
[e6]
checking valves, etc, for propellant
more chatter. Still venting LOX.
T- 6:00
[e7]
Fully charged... on internal power.
transmitter working

T- 5:00
[e8]
Interrupters rotating arm. (?)
Count continuing.
T-4:00
[e9]
LOX venting has increased
there is venting at the bottom as well now
T-3:00
[E10]
T-2:30 LD GO FOR LAUNCH
[E11]
T-2:00
RCO verify range go - RANGE GO!
[E12]
(more people are on the stream, it's having problems now)
(T-0000)
IT'S UP!
[E13]
Missed the last minute of count - but I did get to see the actual launch. The Falcon 9 is on it's way up.
[E14] They seem to be having more feed problems (or I do). There - split screen now. They've passed MaxQ. The plume behind the rocket is feathered-out as expected in very thin atmosphere. (T+3:05)
[E15]
Nominal function at t+4:15 . Second stage is in flight. The main rocket motor is glowing hot.
Everything is running Nominally (except the error on the web site).
The rocket nozzle is glowing white hot in some places - this is expected. Everything is performing nominally.
[E16]
Telem acquired by New Hampshire. There are apparently grass fires near the hangars. The camera on the rocket appears to be functioning well, the rocket is in a slow roll. So far it's looking all good.
Missed 2nd stage separation and ignition.
Right now there's a feed error...
[E17]
The launch has been a success. More info, photos, and video will be posted to spacex as it becomes available.

------

Note: there are video highlights available at the SpaceX web site, as well as more information about Falcon, Dragon, and Spacex.  CONGRATULATIONS SPACEX!!

Radioactive Telepresence

How timely!  A couple of months ago the discussion started - "will unmanned vehicles make a transition into civilian use".  I've been taking the stance that since we're talking vehicles - not just aircraft but all forms of non-stationary robot, that it is inevitable.  Even with aircraft I believe it is inevitable, though it may take a little longer for unmanned / automated aircraft to be certified for use in civilian airspace.

It would make sense that robots would be deployed for things that are either impossible for humans to do, or for things that are hazardous and dangerous.  On the impossible-for-humans side, quick return deep-dive missions in the ocean, and several-day long monitoring missions come to mind, as well as some interesting possibilities for telepresence tourism.  The hazardous side is easy to imagine - everything from maintenance of city infrastructures to handing toxic or radioactive substances would be fair game to use robots for, as well as underground mining.

Among some of the things I've thought about are robots with a divide-and-conquer design.  Some robots would be specialized worker robots, others might act as relay points to extend the reach of the overall system, and some functionality might overlap.

Just a couple of days ago, Bill posted about INTRA Groupe's Hazardous Materials handling robots.
This is an excellent example of a small herd of robots designed to perform complementary tasks. There's even a unit specially designed to act as a data relay, to extend the reach of the telepresence system.  Though these robots are currently remote-operated, as the need arises and becomes evident I would expect each system to be given some degree of autonomous control, in order to augment the abilities of their human controllers to achieve their ultimate goal.

Quit Bugging Me: Making Maps

A tool commonly used in embedded debugging is a linker map - a map of where all the symbols are in the runtime image.  These maps are useful as they turn raw addresses reported by some exception stubs (etc) into offsets into the data or text (program routines) in the computer's RAM.  They give you an idea of what may have been happening when the error occurred.

Producing a linker map is fairly easy.  Most linkers include command line options to produce a map.  This works fine and is very clear when used from a command line.  But things can get a little confusing from within an integrated gui environment.

The Workbench environment makes it very easy to create and use projects, and glue them together.  A problem that can occur, is since you build all projects by selecting "build project", you may not be sure of what tools are being used by the builder "under the hood".

A recent question we had was "How do I produce a linker map?  I provided the following flags to get the linker to make a map:  -Xlinker -map mapfile.  Instead of getting my map file, I got an error."  The error the customer was getting was "ldppc: cannot find -linker".

Thinking about the problem a bit I came to the realization that the customer was calling ldppc with the flags that tell ccppc "hand these options to the linker".  The problem is that the linker, ldppc, called with these flags directly, decided they meant something else.  The real problem is a misunderstanding that the project being modified uses the linker directly, instead of the compiler front-end.  I decided it would help to demonstrate.

To the compiler, "-Xlinker blah blah" tells the compiler "call the linker with the flags blah blah".

-Xlinker  means something completely different to the linker.  -X tells the linker "discard symbols".  -l<something> tells the linker "link in the library named <something>".  The kind of funny thing is that using -Xlinker, the linker was looking for a library named "inker", libinker.a .  Since the directive failed, the linker sort-of seemed to be complaining that it couldn't find itself!  Here's a demonstration of it: note that to complete the demonstration I needed to add a directive "look here for libraries" (-L.)

For the demonstration, I launched the Developer's Environment:

Q:\WIND\VX67WB31>ldppc --help | grep -i map
  -M, --print-map             Print map file on standard output
  -Map FILE                   Write a map file
Q:\WIND\VX67WB31>ldppc --help | grep X
  ...
  -X, --discard-locals        Discard temporary local symbols (default)
  ...
Q:\WIND\VX67WB31>touch libinker.a

Q:\WIND\VX67WB31>ldppc -L. -Xlinker
ldppc: warning: cannot find entry symbol _start; not setting start address

Q:\WIND\VX67WB31>rm libinker.a

Q:\WIND\VX67WB31>ldppc -L. -Xlinker
ldppc: cannot find -linker

More Autonomy In The Science Sector

In a continuing thread of discussions on LinkedIn, we've been talking about robotic craft moving from military use into more common civilian use.  Though many have taken the discussion to be focused on flight, I've deliberately taken a much wider view, defining the word "vehicle" to mean "anything that moves".  By that definition, the Roomba Vacuum is an unmanned autonomous vehicle.

Some would say I take liberties with my definition.  I do.  :)  I'm an engineer who's worked with customers who create rather sophisticated robotic systems, so I feel perhaps I've earned a little liberty there.  But for those who prefer the stronger definition, here's a story for them: NASA'S Global Hawk Completes First Science Flight. There we go, an autonomous UAV moving directly from the Military World right into the heart of Scientific Data Collection.

I find this pretty exciting.  There are numerous technologies that can be demonstrated, leveraged, tuned and perfected with this craft.  Aside from it's direct use as an observing platform, because of it's altitude it has huge line-of-sight coverage, and could possibly be used to collect data from remote sporadically-connected sensing stations. Devices like the new self-powered underwater buoys which only call-back when they've surfaced would be a great example. (Google for "Disruption Tolerant Networking" and "InterPlanetary Internet" for more about sporadically connected devices.)

It's also exciting in that it's use of a sword turned plow-share, an example of robots moving from military duty into civilian operation.  We could use more autonomous and semi-autonomous robots for things like long deep-water projects, working in mines, or hazardous environments.  This is an exciting time, as the field of autonomous robotics is still in it's infancy, and there is so much potential in what such robots can do for mankind.

Autonomous Ocean-Powered vehicle

NASA Demonstrates Novel Ocean-Powered Underwater Vehicle

PASADENA, Calif. – NASA, U.S. Navy and university researchers have successfully demonstrated the first robotic underwater vehicle to be powered entirely by natural, renewable, ocean thermal energy.

The Sounding Oceanographic Lagrangrian Observer Thermal RECharging (SOLO-TREC) autonomous underwater vehicle uses a novel thermal recharging engine powered by the natural temperature differences found at different ocean depths. Scalable for use on most robotic oceanographic vehicles, this technology breakthrough could usher in a new generation of autonomous underwater vehicles capable of virtually indefinite ocean monitoring for climate and marine animal studies, exploration and surveillance.

Read More

I find this news release both very interesting and timely.  Recently on Linked-In there have been discussions about whether or not autonomous robots would find deployment in the commercial and private sectors (as opposed to military use).   Much of the discussion has been around autonomous flying vehicles,  this is a perfect example of why I feel the discussion should be more broad in scope.

This is an interesting application for autonomy, and even more, a very novel approach to the "perpetual motion" machine.  It's current incarnation is a NASA/NAVY co-operative venture, but this could as easily be a project undertaken by a private  undersea resource exploration / consultation company.

Wouldn't you love to take a look under the hood?  The power system must be fascinating in and of itself, but there must also be a range of low-power sensors, radios, and I'd love to see what tricks they employ to get the most out of the computers.   With these just coming out, one can imagine that someday navigating undersea will be a lot like guiding a submarine through a gigantic Galileo Thermometer.

The whole world of unmanned and autonomous vehicles is in a fledgling state.  There will eventually be more unmanned  robotic and autonomous vehicles than their manned counterparts.  (note: the lines are already blurred - there are manned semi-/autonomous vehicles like automated monorail transport systems in airports that only need human interaction for maintenance and emergencies, and aircraft using Autoland, and unmanned remote operated/piloted vehicles like Tiburon or Predator.)

More Bacon: increasing science return

In a recent blog I wrote, Bringing Back The Science Bacon,  one of the things I talk about is increasing the science returns of a mission by doing some data processing on the mission robot, before it sends back information.  If a computer has the ability, and the trade-offs gathering data, processing it into information, available computer throughput and bandwidth of the data channel are favorable for processing, data can be processed into information and the mission may return more relevant science.

Some great examples come from the Mars Exploration Rovers.  NASA/JPL has managed to operate the rovers for over seven years.  We've sent up a number of software updates to both rovers.  Updates include new capabilities, and new ways to look for science.

Here's another great example of increasing the science return by doing a little more processing on-board.  The rovers already have some autonomy, with abilities to detect and plot ways around obstacles.  By increasing autonomy, the AEGIS package gives the rovers the ability to detect targets of interest to science, and adjust their courses to investigate.  (Dear Editors of Aerospace-Technology.com - as orbiting entities, the Mars Rovers have very low orbits...)

This is exiting.  Among other things, AEGIS will enable the rovers to return science we would otherwise have missed.  In the old model, the rovers drive along, sometimes for extended periods, only taking and returning samples we tell them to.  With AEGIS, the rovers can now "sense" that a given object is scientifically interesting, decide to visit it, and return photos so the scientists can decide if it warrants a visit.  This is how NASA accelerates science.

And talk about a mission getting the bang for it's bucks?  This three month mission is 7 years into it's calendar, and is experimenting with ways to increase the science it currently returns, and science returns for future missions.  Very cool!

And The 2010 SpaceOps Award for Outstanding Achievement goes to...

Until today, I had never heard of The International Committee on Technical Interchange for Space Mission Operations.  Or the SpaceOps Awards.  An announcement arrived in my email this afternoon.  Guess who/what got the award?  Go on..  :-)  CONGRATULATIONS!

Quit Bugging Me: Another Surprise NaN!

An Earlier QBM, "Surprise NAN" covered how floating point computations may become corrupted from unexpected sources.  Today's Quit Bugging Me is about... another Surprise! NAN!

An application runs with several tasks.  One task, which computes a set of floating point values, periodically comes up with bad values, and sometimes the name becomes corrupted.  

Here is a short example...  done with the solaris simulator.  Kernel options do -not- include any hardening, nor any guard pages / stack protection.  In fact, only the options related to "MMU_BASIC" are included.  I have a breakpoint set just before the exit so I can catch overflows.  surpriseNAN recurses on it's argument, easily overflowing the default stack size.  I call logMsg first thing, it's a cheap way to get the task ID printed out.  I'm going to provide edited kernel shell output to show the points.

Symptoms:  the task names are valid on entry but suddenly go away, there's a crash that involves "reschedule", the values contained in the FPregs are... corrupted badly:

-> task spawned: id = 0x40b4820, name = t2

task spawned: id = 0x40bb968, name = t3

task spawned: id = 0x40ad6d8, name = t4

task spawned: id = 0x40a6590, name = t5

task spawned: id = 0x409f448, name = t6

task spawned: id = 0x4098300, name = t7

0x40bb968 (t3): alive.

0x40ad6d8 (t4): alive.

0x40a6590 (t5): alive.

sometime later the BP is hit.  Several of the tasks are "missing", though none completed.

> i

  NAME      ENTRY     TID    PRI   STATUS      PC       SP     ERRNO  DELAY

-------- ---------- -------- --- ---------- -------- -------- ------- -----

tExcTask   excTask   40d8250   0 PEND          a89cc  40d8058       0     0

tLogTask   logTask   40d37b8   0 PEND          a89cc  40d35c0       0     0

tWdbTask   5eea4     40ce308   3 PEND          664c8  40cdfd8       0     0

!       surpriseNAN  40bb968 100 READY         6a80c  40b4cb8       0     0

!       surpriseNAN  40ad6d8 100 READY         6a80c  40a6a28       0     0

       surpriseNAN   40a6590 100 SUSPEND       6a810  409f950       0     0

value = 0 = 0x0

Notice - the task t5 hit the breakpoint and was suspended.  The other two tasks are "ready" and never hit the break point.  They are no longer running though.  And what happened to the names?  Let's look at one of the tasks...

-> ti 0x40bb968

NAME       ENTRY      TID    PRI   STATUS      PC       SP     ERRNO  DELAY

-------- ----------- ------- --- ---------- -------- -------- ------- -----

        surpriseNAN  40bb968 100 READY         6a80c  40b4cb8       0     0

stack: base 0x40bb968  end 0x40b4c08  size 27856  high 27856  margin 0    

options: 0x1d

VX_SUPERVISOR_MODE  VX_DEALLOC_STACK    VX_FP_TASK          VX_STDIO            

%pc    =    6a80c   %npc   =    6a810   %psr   =        0   %mask  =        0

%tbr   =        0   %y     =        0                                        

%g0    =        0   %g1    =       5f   %g2    =        0   %g3    =  40ad6d8

%g4    =        0   %g5    =        0   %g6    =        0   %g7    =        0

%i0    =        1   %i1    =    db800   %i2    =        0   %i3    =        0

%i4    =        0   %i5    =        0   %fp    =  40b4d30   %i7    =  40d9d60

%l0    =  40bb968   %l1    =        0   %l2    =        0   %l3    =        0

%l4    =        0   %l5    =        0   %l6    =        0   %l7    =        0

%o0    =        0   %o1    =    db800   %o2    =        0   %o3    =        0

%o4    =        0   %o5    =        0   %sp    =  40b4cb8   %o7    =    6a804

%fsr   = eeeeeeee
%f00   = 0.793335   %f02   =        4   %f04   =        8   %f06   =        8
%f08   =  7.87933   %f10   = 0.00819755   %f12   = 0.223435   %f14   = -0.668483
%f16   = -0.902314   %f18   =        0   %f20   =        0   %f22   =      NaN
%f24   =      NaN   %f26   =      NaN   %f28   =      NaN   %f30   =      NaN
value = 67819224 = 0x40ad6d8

What happened here?  The FP-REGS are trashed?  and the name is trashed... something must have corrupted the system.  Is it a stack over-run?

-> checkStack

  NAME        ENTRY        TID     SIZE   CUR  HIGH  MARGIN

------------ ------------ -------- ----- ----- ----- ------

tExcTask     excTask      40d8250  15984   504  3016  12968 

tLogTask     logTask      40d37b8  15984   504  2104  13880 

tWdbTask     0x000005eea4 40ce308  16040   816 11376   4664 

!           surpriseNAN  40bb968  27856 27824 27856      0 OVERFLOW

!           surpriseNAN  40ad6d8  27856 27824 27856      0 OVERFLOW

            surpriseNAN  40a6590  27856 27712 27856      0 OVERFLOW

So in fact there is not only one stack overflow, but 3.  What has happened here?

Looking back at the TaskIDs returned from taskSpawn, we can see these were all spawned with their OS data structures right in a row.  With no memory protection (no memory protections included!) the task stacks are not protected.

When the surpriseNAN recursively writes over the top of it's own stack, it corrupts the area where the floating point data are saved.  Some of the values written work-out to be valid floating point numbers, others are not numbers.  This is near the area where the task name is stored.  With a sufficient overflow, the task name is corrupted as well as the FP data.

When the task overflows it's stack far enough, it will eventually over-write other data.  In this case the other data is the register set of the task immediately adjacent (in lower addresses) in RAM.  

Something to notice in the corrupted FP-REGS - the FSR is set to a stack-fill value (so the FSR screams "eeeeeeee!").   This is one more tell-tale that the context for this task has been corrupted.

Domesticated Robotics?

A discussion about robotics was started up in an online discussion forum.  A question was posed to the group:

Will unmanned vehicles eventually see duty in civilian applications?

Technology for unmanned military vehicles may eventually trickle down to commercial applications on Main Street. In the near future, autonomous vehicles will be used to deliver packages, collect garbage and fill potholes. http://bit.ly/bSrqfx

The way I read the question, it asks about unmanned "vehicles" in general.  Without specification I would regard this to include all autonomous robots, all sizes, all forms of motivation (wheels, legs, wings, whatever).

Part of my initial response was...

Given the increasing needs for communication points, bandwidth, and the increasing desire for remote data sensing (think: telerobotic tourism as an eventual end), it is an inevitability that unmanned robotic vehicles will play a significant role in the commercial and private sectors.

Expanding on this some, we are already seeing an expansion in domestic robotics.  Googling for "domestic robot" gets you nearly 58,000 hits!  There are devices covering from vacuums to "butlers" to sophisticated electronic teddy bears appearing in this fledgling market segment.

In the industrial sector, there are mining robots, deep sea robots, welders, archivists, pipe inspectors, and harmful-environments service robots.  Like the home PC market had in 1975, this market has some potential for growth. 

My instincts tell me the real question isn't "will there be unmanned vehicles [autonomous robotics] in civilian duty", but how will we best make use of the explosion of these devices, maximize their safety, provide some level of individual security for the owners/users, and ... make it all work on limited power budgets.

Getting back to the need for connections, your local municipality may begin to employ remote service machines.  These machines may be used (as in the referenced article's question) to maintain the infrastructure, and perform communal duties of domestic maintenance.  These collections of robots will need data channels to convey back the results of their duties, nominal status, and report noted exceptions.  One could foresee a day when semi-intelligent autonomous systems might be used to augment domestic and emergency services personnel.

Could you imagine if your area had some natural disaster, and the local water provider were able to send robotic pipe inspectors through the system to listen for people trapped inside buildings?  Or perhaps your vacuum cleaner reporting that your apartment is on fire while you're on vacation, and which room is on fire?  Though either of these sound a little fantastic, both are easily feasible with today's technologies.  It could easily be done with a mix of a hypervisor and use of existing software on a low-power multicore platform, the rest of it is just batteries, motors, radios, and a fancy box.

The security side of it can come from a mix of approaches and disciplines, depending on what the intended goals of the platform are.  For instance, you don't want intercepted pipe inspection data used wrongly (for industrial espionage, for instance), so the sensor feedback channels could require some form of encryption or pre-processing.

Larger systems / devices might require some form of review by certification agencies.

I don't think it's a question - will there be more unmanned gizmos working for us?  It's a certainty I think, just a mater of time.  I'm wondering how long until they want counted in the census...

Bringing Home the Science Bacon

MRO recently passed a milestone of bringing back more data than all previous deep space missions combined.  To do this MRO was equipped with a large dish antenna and powerful radio, and is running a more powerful computer than it's companions on and orbiting Mars.  It gets a fair share of antenna time from the DSN antennae on the ground.

Science means collecting data, processing it for it's information, and analyzing the information.  All of these activities can be greatly facilitated by some degree of computer assistance. When it comes to remote-operated scientists (rovers, gliders, orbiters, diggers, swimmers, crawlers, hover-ers..) the computer has to do all the work.

Where it comes to an autonomous robot, or even partially autonomous, a balance must be struck between the amount of data that can be brought back (telemetry bandwidth), the amount of data collected by the science instruments, the amount of processing it takes to turn the data into information, and the processing ability of the computer(s) in the robot.  If the largest gating factor is the bandwidth, and you can gain 10x the science return by processing the data first, you may limit how much data you collect in favor of processing it into information.  If your biggest limitation is processing power, you may just concentrate on getting data off of science instruments and ready for transmit back.  The typical deep space mission has both challenges, and more.

There are terrestrial applications that have similar challenges.  Remote sensing networks, deep sea ROVs and robots, and some industrial applications, as well as military applications, may need to operate independently for periods of time, storing data for upload at the "next" connection.  There could be limitations on storage, processing ability, power, connection bandwidth. 

Some of the solution could be to run multicore.  You may be able to run a multicore chip at a reduced clock rate to save power and gain more processing power than a similar single-core chip at full clock speed.  By increasing the processing power, and using mixed methods (SMP,AMP) as in Mark's "Sea Of Cores...Now What?" discussion, you may be able to handle both steering data and processing it into information. This could increase the science return of the mission.  But with multicore, how do you balance the trade-offs and where do you focus your increase in processing power?

By simulating the whole affair you may be able to gauge how fully loaded your system is.  If you have processing power left, and also important, data steering capacity left, you may be able to share your telemetry stream with another project.  This is part of what NASA has done with the satellites orbiting Mars and the MER and Phoenix robots. The robots collect their data and do what processing they can, storing the results.  They are designed to send data directly to Earth, and do.  When possible, they can also transfer data to the satellites, borrowing some of their bandwidth.  By using a little slack-time in the orbiter's schedules, they're increasing the science return of the robots on the surface.

Increasing the processing power of one key system can increase the data-product returns of an entire group of systems. It can increase the returns both in terms of the quantity of data and quality of information returned.  With careful planning, the "more remote" systems can be coordinated to take advantage of data-stream offload, perhaps doing some data processing, or just collecting more raw data.  Prudent use of a virtualized / multicore system could be one great way of doing this.

None of the current Mars robots are empowered with multicore - when they were designed multicore wasn't as prevalent as today, and there are still very few multicore chips designed for the rigors of space.  There are plenty of off-the-shelf chips with multiple cores available.  Terrestrial based applications can start benefiting from SMP/AMP systems today.