Mike Deliman

Space Junk: collision imminent

It was bound to happen eventually.   On February 10th  two large satellites collided in orbit - Kosmos 2251 crashed into Iridium 33 [spaceweather] [bbc]. 

Even with sophisticated tracking and modeling mechanisms, two large satellites managed to crash into each-other. On the one hand it's surprising - even the experts all say "unexpected" and "extremely unlikely".  On the other hand, there's an awful lot of junk up there.  Tools like JTRACK  can give you an idea of just how much stuff there is up there.  Even just looking at the working satellites alone, it's an awful lot of stuff to keep track of.  Add-in the space junk and dead satellites, and amount of stuff to track becomes pretty overwhelming.

As the number of items grows, the problem of simulating the whole thing becomes an issue of computational bandwidth.  Tracking alone won't tell you when to look out for collisions, but it will give you data to use in simulations which may give you some advance warning.   Tracking that many objects becomes a problem of it's own.  But I don't think that either is the real solution.

Ultimately what will need done is at least combination of things.
* international agreements will need to be made on a protocol for space-use.  For instance, defunct satellites need to be decommissioned and removed from orbit somehow. 
* safe ways of removing the space-junk that's already up there needs to be devised.  Some sort of agreements / property rights would probably have to be worked-out for defunct satellites.  (For instance, I've heard the Smithsonian has first-rights to all space-stuff returned by the US / found on US soil.
* some sort of agreements or body will need to be in place to handle problems such as this, where a 'defunct' satellite destroys a working one.

It may even come to be that we will need some sort of active machinery to prevent collisions, perhaps even something akin to current military high-energy laser systems.  Such systems would require sophisticated controls, real-time software, and probably certification as a life-and-mission critical system.

I'll have to assume something like this will be built in the future.  No-one has contacted me about my low-tech idea of a "giant fly paper strip" that could be attached to the tail of the ISS...

Quit Bugging Me: Induction

Working on a customer problem once, we had an interesting phenomena.  Upgrading a system with a large VME cage and several boards, the customer replaced older processor boards with what were then "new" boards.  The old boards ran at (I think) 33 MHz, the new ones at more like 133MHz.

The overall system included motor control functions and sensor feedback, there were D-to-A, A-to-D, and other boards in the system.  The problem was, with a straight-and-simple rebuild of software for the new boards, the system was giving anomalous readings.  Checking the registers for values from the A-to-D boards, there were numbers showing up that were not possible.  Readings indicated the motors were on and moving their load when in fact the motors were not connected.  Switching back to the old computer boards eliminated the problem.  We added some hardware to examine VME bus activity, to see if the new CPUs were reading the bus wrong.  We had to put the test board between the CPUs and the sampling boards.

The sampling boards (A-to-D) were in the chassis right next to the CPUs.  They had a sampling rate not far off from the new CPU's clock rates.  Moving the sampling boards farther down the chassis reduced the severity of the anomalous data reads, but did not eliminate the problem.  The bus analiser showed the data on the VME bus was consistent with the data reported by the new CPUs.  This provided the final clues needed to pinpoint the problem and fix the system.

The new boards have clocks that run at a rate close to the sampling rate of the A-to-D boards.  The clocks were physically next to the A-to-D  boards in the VME chassis.  For whatever reason, the new boards oscillators were radiating enough energy at the right frequency to corrupt samples taken by the A-to-D boards.  By moving the boards farther apart we reduced the effect (inverse square law) but not by enough to eliminate it.  To test this theory we placed steel plates in a VME slot, as a separator between the rest of the boards in the bus and the A-to-D board.  At this point, both old and new CPUs read the same consistent readings from the A-to-D boards.  Removing the metal plates brought back anomalous readings with the new CPUs.

In this case it seemed like a new system of software and boards was improperly accessing a device.  Switching back to the old baseline eliminated all the new problems.  Analytical equipment verified the readings reported by the CPU were consistent with the data retrieved from the sampling hardware, so it wasn't a problem "within" the new software or hardware. The real problem was a completely unexpected interaction between legacy hardware and upgrade hardware.

Serial Intent

Howdy out there,

I realize this is a blog that's supposed to be about real-time programming issues, and mostly I've posted about planetary and space based projects, with a few announcements about technology and news items.  Though these combine subjects near and dear to my heart (space, and VxWorks), these were mostly "interest stories", not solid real-time issues.  (It must be kind of like tuning into a financial news station and seeing stories about some new exotic fluffy breed of dog.)

In line with the Real-Time issues theme, I'm going to try to post more often about debugging systems and common issues I've seen over the past 20+ years of working with VxWorks.  This is not because of feedback from the readership, it's something I just decided needs to be done.  *smile* please don't be confused by the subject lines...

Starting tomorrow, I'll try to post at least one story a month under the title: Quit Bugging Me.

Kind regards,
  mike

First Light, First Flight

Kepler Space Telescope recently had it's First Light images released!  Just how many stars do you think Kepler is watching every day and night?

This star in the circle is TrES2, a star known to have a planet around it.  It's in a cluster of stars known as NGC6791.   Zooming out more,  here's an inverted-image of Kepler's entire field of view. How much of the field of view do you think was in that second picture?

Here is a Star Map of Kepler's field of view.  There's a bunch of squares in the middle of the picture.  Down the bottom edge there are 3 squares.  If you click on the right end one, it actually contains two rectangles.  You'll see a little circle in there with "N6791".  That's where the cluster of stars NGC6791 is.

How many stars does Kepler watch?  In this image zoom in on the fuzzy black dot below and left of the "Kepler" in the bottom right corner.  That blob of stars that makes a fuzzy blob - is the area labeled "N6791" in the other picture - is what you zoom in on in this image.   The answer: Kepler is interested in over one hundred thousand stars, and is watching them all at once.  It will be interesting to see what bread crumbs Kepler finds for SIM and TPF to follow (SIM and TPF will be making a more complete survey of distances to stars and looking for more earth-like planets).  I'm proud that Kepler's flight software runs on VxWorks.

Speaking of flight software...  it looks like The Boeing 787 may have it's "first flight" fairly soon.  :)

Virtually... Not Yours

So with all this virtualization going on, one computer can run multiple copies of an OS (or multiple OSs), all acting like they own the whole computer. With only some fancy shim layer keeping them from corrupting themselves and everything else, what are the implications for security?  Security is an issue that has been raised more frequently as computers have become more powerful and the network more pervasive.  I mean, viruses spread fast enough when the World Wide Web was mostly single computers with modems dialing up to servers.  In this day of broadband everywhere and constant connections,  doesn't virtualization present a special problem? Doesn't a machine running 10 virtual servers just mean that a virus gets to invade 10 servers for the price of entering one machine? Let the SPAM flow!  NOT!

The problems mentioned are part of an ongoing concern for both the desktop world and the embedded world.  There are standards, and evaluation labs, and  certifications that can be obtained.  They mean "This computing platform can replace several physical computers with one physical computer providing several virtual computing environments, and keep them sufficiently separate so as to protect each one from all the others."  Most of these - such as ARINC 653 and MILS, set forth specifications for systems to satisfy in order to obtain a certification.  These certifications can be issued according to the importance of the system being certified; for example to something like "DO-178B level A" or "EAL 6+" for a system of high criticality (life-critical functions, "if it fails, people die").  Though these standards may not use the word "virtualization" outright, the services they require can be provided by a robust virtualized system.

Among the things these standards can do for us:  provide for protections that would prevent a problem in any one virtual environment from affecting any other.  A virus could corrupt and crash a virtual server, the supervising software could shut down and destroy, and rebuild that virtual environment, starting with a clean slate.  "No Virus, Bad Virus...  NOT YOURS!"  This oversimplifies the whole idea, but you get the picture.  If one virtual machine crashes, it's not going to affect the entire system, the other virtual machines keep running.  You might lose that powerpoint presentation you were "researching", but the feedback from the UAV being displayed, the real-time map updates, and the status of your deployed troops will still continue unaffected by the loss of that document or the restart of the OS that was affected by it.

Today we're announcing  VxWorks MILS, currently under  evaluation to EAL6+.  Now then... what about multicore?  :-)

Virtually Yours...

This morning we've made a couple of announcements that will hopefully surprise and delight the Real Time computing community.  Wind River is releasing two new products that have their base in  virtualization: Wind River's Hypervisor, and Wind River's MILS 2.0.  Both of these products introduce platform virtualization, used in different ways.   Wind River's implementations allow for a "floatilla" of OS's" to be run amongst a "sea of Cores" - it can be run as a one-to-many, many-to-many, or many-to one configuration (that is with SMP, one OS can be run by many cores, the Hypervisor is capable of running / scheduling many OS's among many cores, and both the Hypervisor and MILS 2.0 are capable of running many OS's over one shared core).

Both of these products are very high-powered and sophisticated.  Both products are designed with an eye to the future.  The Hypervisor itself can be used in SMP, AMP, Supervised-AMP and cooperative multicore configurations, and can also be configured to paravirtualize several Operating Systems and run them all (time-share style) on a single core.  The MILS 2.0 product is designed specifically to paravirtualize OS's and run them on a single shared-core with a high degree of separation maintained by the MILS 2.0 separation kernel.

Many of our customers have built multiprocessor systems, sometimes because there wasn't enough horsepower on one board to handle the needs of their system, sometimes because of a need to keep data processing nodes separate (security).  With today's processors, you may be able to replace a multi-board system with either a single multicore board running Hypervisor or a single board running a modern (single) core with our MILS 2.0 separation kernel.  Both products allow a number of instances of VxWorks to run on paravirtualized platforms, or run without an OS at all (bare-metal or minimal executive, etc), and the Hypervisor also supports Wind River's Linux on paravirtualized platforms.

So what is virtualization?  I think of virtualization as a star-trek like copy machine, where you start off with a computer, and make "virtual" copies of it.  The copy can have anything that's already on the real machine, or a subset of those features.  The Hypervisor schedules those copies to run on the real machine, making them believe they are completely in control.  This can allow you to replace several older machines with one virtualized machine and get the same work done.

Some popular examples include VMware, Parallels, and QEMU.  These are all available for desktop machines, allowing users to create polymorphic environments.  These are mostly "type-2" hypervisors, meaning the hypervisor is a guest running under a main operating system and is subject to the whims of that OS.  Wind River's implmentations are "Type 1", meaning our Hypervusor runs as the only supervisory access code on the system and has absolute control over the system.

I'm pretty excited about this.  It brings supercomputing into the embedded realm!

Tolerating Delays

This may sound a bit funny, but in the space industry, we're constantly playing catch-up.  We're either looking at or for things that happened millions or hundreds of millions of years ago, sending rockets off to get to where something will have been just in time to take a picture of or bore a hole into it, or designing new rockets for flight 5 years from now with computer bits that would have been considered top-of-the-line 5 or 10 years ago.  When we're recovering data and sending commands from and to deep space probes, we point our antennae to where the probe is supposed to be 30 minutes from now and start sending our data now; the idea is by the time the data actually gets there the craft will be where it was expected and receive the commands, and send it's data back to us.

This process I just described - of anticipating where a craft is, transmitting before it's there - and it transmitting back - is pretty much how the Deep Space network is currently used.  It takes huge amounts of planning, all done in advance, to set up the multiple sessions that allow one successful exchange like that to work out.  People consult tables of times when craft will be "visible", consult tables of one-way light distances to find transmission times.  When they think they know when they need time, and how much data they expect to exchange (how much  time they need),  they contact the folks who run the big antennas.  if the time slot is available, arrangements are made, and that one set of transmissions can take place.  The folks who run the Deep Space Network and their customers do this sort of thing all the time - it's how they try to make the most efficient use of their giant antennae.

Thinking about this whole process, it would seem that you should be able to automate much of it, if not all of it.  There may be some circumstances where manual intervention is necessary, but if much of it were automated, automation might be able to help bringing data back indirectly.  For a given satellite or probe a table could be maintained with information about the DSN and other contact points it maintains regular connections with.  Other probes and satellites could maintain similar tables.  This would enable other deep space probes, satellites, orbiters, and craft for planet exploration (landers, crawlers, flying / roving automata) to plan and create connections as-needed to trade data with each-other.  On top of these communication connections you could layer protocols and methods, and from this, you could create a sort of network.

That is exactly what this article titled Dot Mars is about.  Delay Tolerant Networking, or DTN, is an experimental protocol to enable the internet to reach out off of the Earth.  JPL's Interplanetary Overlay Network (ION) implementation was tested last summer between the Deep Impact / EPOXI satellite and the Deep Space Network.  The test results were: success.

DTN does more than just create another way to move data back from space.  And it's potential reaches into terrestrial applications as well as to oter planets.  The gains that could be leveraged by spacecraft could also be leveraged here.

In space DTN could be a key factor in project design.  If a team knew for sure that the DTN internet was available, and would offload data at regular intervals, they could take advantage of that fact.  If you knew you could send back a set amount of data per day every day, and you'd only ever have to store perhaps a week's worth of data backlog, you could leverage this in your craft's design.  You may be able to design-in less long-term storage, making the craft lighter.  You may have more bandwidth available, so you may decide to use sensors with more sensitivity or take more samples than you may otherwise have planned on.  Returning more data and possibly better data should mean a higher likelihood of a successful mission.  Overall, encorporating DTN into a space project could mean reduced costs and better returns.

Bringing the paradigm back down to earth, DTN could be deployed in a large number of terrestrial applications.  For example, remote sensors currently wait to be connected to and commanded to offload data, or they attempt to send back their data at regular intervals.  If they miss a specific connection it could mean lost data.  If they were able to organize and tap into a DTN style of network, they could possibly offload their data to some alternative trusted node and not have to incur any data loss. 

It could even affect your home life.  Sensors, for example in your car, could store event related data indicating you need the spark plugs replaced.  Right now if you disconnect the battery before a mechanic reads your OBDII data, you'd lose that information.  With a DTN-enabled system, the car's onboard computer could store that data, and tag it as important.  The next time it's near a trusted-node - at home, at work, or perhaps even hile filling-up at a gas station, it could email that data to the owner perhaps allerting them before a breakdown occurs.  Having your car break down at the just wrong time could be an intolerable delay.

Security In An Insecure World

Diebold has apparently gaffe'd their software again.  Their ATM systems run software based on Windows.  This gives me a perfect opportunity.  Recently some of my friends heard me talking about our MILS software starting under it's evaluation path.  I heard a question being asked more and more about government programs and practices, "okay, but what good does this do for me?"  In the article mentioned above we have an excellent example of where MILS could benefit the average person every day:

But Yerrapragada told SCMagazineUS.com that machines need run-time control software to ensure nothing can tamper with authorized applications.  "You could have firewalls and hardening, but...if those things are not patched, you are out of luck," he said.

It's right there in the quote above - machines need run-time control software to ensure nothing can tamper with authorized applications - and MILS-enabled platforms are a perfect example of such run-time control software.  MILS enables engineers to carve-up the computer being run on into partitions and set up access rights between those partitions. MILS, for instance, can prevent an unauthroized access from writing into application software, or prevent data from exiting a machine via unauthroized channels.

"Fences build better neighbors" is a saying I've heard.  In the old days there were problems between various ranchers until barbed-wire and land access rights were created, deployed and enforced.  The barbed-wired laid out exact boundaries, and the access rights detailed who could go where, when, and for what reasons.

With a proper MILS package, a secure software application should be divisible into discrete parts.  It should enable separate engineering teams to craft thier parts of the application, working with the information they need for that partition to do it's work. The System Integrator should be able to "glue" the parts together with logic that provides the barbed-wire and access-rights between the partitions.  The separation kernel and configuration logic should be able to provide enough separation that any individual partition could be updated without affecting any other partition.  And it should provide the security to prevent any kind of tampering from taking place.  In the language of an every-day ATM, a properly designed and configured MILS system would be able to prevent accesses via the customer card reader and keypad from reprogramming any part of the ATM or accessing any data not normally visible by the current account holder.

What MILS can buy the average tax-paying consumer is security in systems we use every day and do not regard as threats.

Evolution of a Planet's Evolution

NASA Science News for March 26, 2009

Back in 2004, most experts would have said this story is impossible. No rover could possibly survive long enough on Mars for a five-year update. Yet here it is. Mission scientists reveal what Spirit and Opportunity are up to on the Red Planet today--and what their prospects are for the future.

FULL STORY at http://science.nasa.gov/headlines/y2009/26mar_marsroverupdate.htm?list102569
---------

Twenty years ago, experts would have said the thought of Oceans on Mars would be impossible!

When I was a kid, what we knew about Mars was not much. We knew it was red, and sometimes had "snow" on the poles. We knew it had a longer orbit than Earth. And that there were patterns and blotches visible in earth-bound telescopes. Other than that, the prevailing science was: Mars is dry, cold, and dead, and other than a long-ago period of wetness, it had no potential for life.

We (earthlings) sent various probes past, and to Mars. Viking sent us back "proof"¹ that Mars was dead . Aside from the temperatures, and lack of water, between the chemistry and the radiation at the surface, it was unlikely for life to ever form.

When I was in my young thirties, there was this rock that was found, a meteorite from Mars, they said. The electron microscope photographs showed small blobs that were in patterns which resembled the way certain bacteria form here on earth. The age-old debate suddenly opened up - perhaps Mars wasn't always dead? With the debate re-fueled, the timing couldn't be better. Mars Pathfinder was being readied for Launch that December, to head for a July 4th, 1997 landing. At this point, though, the science community was still mostly under the impression that Mars was, and always had been dead.

MPF, also known as Sagan Memorial Station and Sojourner Rover, gave us a few new lessons. One was that we could get very highly capable robots to the surface of Mars to do real science, and the other was that we had to be able to bring the science packages to the science targets if we really wanted to glean everything we could from our visit. The findings from MPF and MGS were compelling enough to warrant more missions. Mars is a Harsh Mistress, though; it would be nearly a decade before we would manage to put another science robot successfully on the surface of Mars.

The evidence started building up after MPF landed. MGS and later Mars Odyssey returned evidence of huge amounts of water ice under the "permafrost" covering Mars. Later, evidence of Methane was seen in the upper atmosphere - on earth methane usually comes from living things. The evidence of water and the results and lessons-learned from Mars Pathfinder (and follow-on attempts) were compelling enough reasons to go back to Mars.

Our Lucky Girl, Opportunity, landed right in a bowl of scientific evidence. From her first day photographing rocks on Mars she had the evidence in hand that Mars had not only been wet, but wet and warm for an extended period. There is a need to change our models of Mars to include oceans of water.

Results from Phoenix give us more mechanisms for water accretion and more evidence. The weather laser detected water ice crystals forming and precipitating in the upper atmosphere. The science oven did not find anything chemistry-wise to rule-out life, and confirmed water-ice near the surface. These are remarkable findings. The water-ice confirms the readings from the orbiting robots - that there is water ice on Mars.

Last summer science teams from Keck Telescope announced they'd found plumes of methane being released from underground on Mars. Weather this is from a heated mineral soup reaction or from a biological process is uncertain, but the methane plumes were unexpected. Methane is a short-lived gas, so the gas was probably created recently.

Over the last 20 years using remote scientific probes, NASA has managed to rewrite the History of Mars, from a planet where life could not exist, to one where where traces of life may still exist. Mars Science Lab might send us back the answer to that question.

Wind River is proud to have been a part of each of the projects named, except MGS and Viking.

¹: [excerpt] "Failure of the gas chromatograph-mass spectrometer to detect organic compounds was devastating for those who believed that life on Mars was possible."

Happy Ada Lovelace Day!

with Ada.Text_IO; use Ada.Text_IO;
procedure Happy_Ada_Lovelace_Day is
begin
  puts("Happy Ada Lovelace Day!");
end Happy_Ada_Lovelace_Day;

Yep - it's Ada Lovelace Day. She invented software engineering, you know. So yes, she's one of my Heroes. (Ada program stolen from a friend... thanks Chris.)