Mike Deliman

Twenty

In 1989 I got my first "real" job, left an auto shop I'd been working at in Santa Cruz, and started working for a pre-tech-bubble company.  Integrated made VME racks, boxes from the size of a large toaster all the way up to racks larger than your kitchen refrigerator.  We also ported, maintained and sold versions of BSD Unix and something I'd never heard of before that we called Uniworks.

Uniworks was this shrink-wrapped version of someone else's product - this tiny company up in Emeryville that was churning out this incredible tiny unix-like system.  You could embed this software in the bootroms of your computer, and it would come up with networking and everything else on-line, and you could build applications fairly easily.  We assisted in recovering some debris off of Florida that year, using this product.  It was a.. Challenging job.

Uniworks was based on VxWorks 4.something, and though it was somewhat clunky compared to today's packages, it was amazing that it was fairly easy to configure and build, and it worked.  At the time - the first day I saw it - I thought... "wow... that stuff has the potential to change the world."  At that time - the company I worked for had over 200 employees, and Wind had something like 50.  I never imagined that the tables would turn, nor how fast.

1989 - that was twenty years ago.  Two years later I started working at Wind River.  VxWorks has been utilized in everything from MP3 players to telerobotic surgeons to autonomous space robots since then.  It has been used to map the human genome, unravel the story of Mars' evolution, and helped find planets around distant stars.

Wednesday the Mars Rovers team sent up commands to reformat Spirit's flash drive.  Yesterday Spirit managed to write data to the flash drive, which had been failing to retain data overnight.  This morning we retrieved saved telemetry from Spirit.  Once again, we've mitigated a problem from 30,000,000 miles away, using simple commands in VxWorks.  If we can get Spirit Free now, she may have another 2000 days roving Mars (for our 90 day mission).

I wonder, what will we find over the next 20 years?  :-)

Virtually... Not Yours

So with all this virtualization going on, one computer can run multiple copies of an OS (or multiple OSs), all acting like they own the whole computer. With only some fancy shim layer keeping them from corrupting themselves and everything else, what are the implications for security?  Security is an issue that has been raised more frequently as computers have become more powerful and the network more pervasive.  I mean, viruses spread fast enough when the World Wide Web was mostly single computers with modems dialing up to servers.  In this day of broadband everywhere and constant connections,  doesn't virtualization present a special problem? Doesn't a machine running 10 virtual servers just mean that a virus gets to invade 10 servers for the price of entering one machine? Let the SPAM flow!  NOT!

The problems mentioned are part of an ongoing concern for both the desktop world and the embedded world.  There are standards, and evaluation labs, and  certifications that can be obtained.  They mean "This computing platform can replace several physical computers with one physical computer providing several virtual computing environments, and keep them sufficiently separate so as to protect each one from all the others."  Most of these - such as ARINC 653 and MILS, set forth specifications for systems to satisfy in order to obtain a certification.  These certifications can be issued according to the importance of the system being certified; for example to something like "DO-178B level A" or "EAL 6+" for a system of high criticality (life-critical functions, "if it fails, people die").  Though these standards may not use the word "virtualization" outright, the services they require can be provided by a robust virtualized system.

Among the things these standards can do for us:  provide for protections that would prevent a problem in any one virtual environment from affecting any other.  A virus could corrupt and crash a virtual server, the supervising software could shut down and destroy, and rebuild that virtual environment, starting with a clean slate.  "No Virus, Bad Virus...  NOT YOURS!"  This oversimplifies the whole idea, but you get the picture.  If one virtual machine crashes, it's not going to affect the entire system, the other virtual machines keep running.  You might lose that powerpoint presentation you were "researching", but the feedback from the UAV being displayed, the real-time map updates, and the status of your deployed troops will still continue unaffected by the loss of that document or the restart of the OS that was affected by it.

Today we're announcing  VxWorks MILS, currently under  evaluation to EAL6+.  Now then... what about multicore?  :-)

Tolerating Delays

This may sound a bit funny, but in the space industry, we're constantly playing catch-up.  We're either looking at or for things that happened millions or hundreds of millions of years ago, sending rockets off to get to where something will have been just in time to take a picture of or bore a hole into it, or designing new rockets for flight 5 years from now with computer bits that would have been considered top-of-the-line 5 or 10 years ago.  When we're recovering data and sending commands from and to deep space probes, we point our antennae to where the probe is supposed to be 30 minutes from now and start sending our data now; the idea is by the time the data actually gets there the craft will be where it was expected and receive the commands, and send it's data back to us.

This process I just described - of anticipating where a craft is, transmitting before it's there - and it transmitting back - is pretty much how the Deep Space network is currently used.  It takes huge amounts of planning, all done in advance, to set up the multiple sessions that allow one successful exchange like that to work out.  People consult tables of times when craft will be "visible", consult tables of one-way light distances to find transmission times.  When they think they know when they need time, and how much data they expect to exchange (how much  time they need),  they contact the folks who run the big antennas.  if the time slot is available, arrangements are made, and that one set of transmissions can take place.  The folks who run the Deep Space Network and their customers do this sort of thing all the time - it's how they try to make the most efficient use of their giant antennae.

Thinking about this whole process, it would seem that you should be able to automate much of it, if not all of it.  There may be some circumstances where manual intervention is necessary, but if much of it were automated, automation might be able to help bringing data back indirectly.  For a given satellite or probe a table could be maintained with information about the DSN and other contact points it maintains regular connections with.  Other probes and satellites could maintain similar tables.  This would enable other deep space probes, satellites, orbiters, and craft for planet exploration (landers, crawlers, flying / roving automata) to plan and create connections as-needed to trade data with each-other.  On top of these communication connections you could layer protocols and methods, and from this, you could create a sort of network.

That is exactly what this article titled Dot Mars is about.  Delay Tolerant Networking, or DTN, is an experimental protocol to enable the internet to reach out off of the Earth.  JPL's Interplanetary Overlay Network (ION) implementation was tested last summer between the Deep Impact / EPOXI satellite and the Deep Space Network.  The test results were: success.

DTN does more than just create another way to move data back from space.  And it's potential reaches into terrestrial applications as well as to oter planets.  The gains that could be leveraged by spacecraft could also be leveraged here.

In space DTN could be a key factor in project design.  If a team knew for sure that the DTN internet was available, and would offload data at regular intervals, they could take advantage of that fact.  If you knew you could send back a set amount of data per day every day, and you'd only ever have to store perhaps a week's worth of data backlog, you could leverage this in your craft's design.  You may be able to design-in less long-term storage, making the craft lighter.  You may have more bandwidth available, so you may decide to use sensors with more sensitivity or take more samples than you may otherwise have planned on.  Returning more data and possibly better data should mean a higher likelihood of a successful mission.  Overall, encorporating DTN into a space project could mean reduced costs and better returns.

Bringing the paradigm back down to earth, DTN could be deployed in a large number of terrestrial applications.  For example, remote sensors currently wait to be connected to and commanded to offload data, or they attempt to send back their data at regular intervals.  If they miss a specific connection it could mean lost data.  If they were able to organize and tap into a DTN style of network, they could possibly offload their data to some alternative trusted node and not have to incur any data loss. 

It could even affect your home life.  Sensors, for example in your car, could store event related data indicating you need the spark plugs replaced.  Right now if you disconnect the battery before a mechanic reads your OBDII data, you'd lose that information.  With a DTN-enabled system, the car's onboard computer could store that data, and tag it as important.  The next time it's near a trusted-node - at home, at work, or perhaps even hile filling-up at a gas station, it could email that data to the owner perhaps allerting them before a breakdown occurs.  Having your car break down at the just wrong time could be an intolerable delay.

Security In An Insecure World

Diebold has apparently gaffe'd their software again.  Their ATM systems run software based on Windows.  This gives me a perfect opportunity.  Recently some of my friends heard me talking about our MILS software starting under it's evaluation path.  I heard a question being asked more and more about government programs and practices, "okay, but what good does this do for me?"  In the article mentioned above we have an excellent example of where MILS could benefit the average person every day:

But Yerrapragada told SCMagazineUS.com that machines need run-time control software to ensure nothing can tamper with authorized applications.  "You could have firewalls and hardening, but...if those things are not patched, you are out of luck," he said.

It's right there in the quote above - machines need run-time control software to ensure nothing can tamper with authorized applications - and MILS-enabled platforms are a perfect example of such run-time control software.  MILS enables engineers to carve-up the computer being run on into partitions and set up access rights between those partitions. MILS, for instance, can prevent an unauthroized access from writing into application software, or prevent data from exiting a machine via unauthroized channels.

"Fences build better neighbors" is a saying I've heard.  In the old days there were problems between various ranchers until barbed-wire and land access rights were created, deployed and enforced.  The barbed-wired laid out exact boundaries, and the access rights detailed who could go where, when, and for what reasons.

With a proper MILS package, a secure software application should be divisible into discrete parts.  It should enable separate engineering teams to craft thier parts of the application, working with the information they need for that partition to do it's work. The System Integrator should be able to "glue" the parts together with logic that provides the barbed-wire and access-rights between the partitions.  The separation kernel and configuration logic should be able to provide enough separation that any individual partition could be updated without affecting any other partition.  And it should provide the security to prevent any kind of tampering from taking place.  In the language of an every-day ATM, a properly designed and configured MILS system would be able to prevent accesses via the customer card reader and keypad from reprogramming any part of the ATM or accessing any data not normally visible by the current account holder.

What MILS can buy the average tax-paying consumer is security in systems we use every day and do not regard as threats.

Wind River Webinar: Q and A

Hello All,

last week as some of you know I was the featured presenter / presentation for a Webinar.   (you may have problems watching that with firefox...) .  During the course of the webinar, we were asked a number of questions, and we ran out of time...

Here are some of the questions and answers we couldn't get to.

--
Q: What is the biggest advantage of Vxwork over other real-time operation system?
A:  VxWorks is the most well-deployed and well-used commercial realtime OS in the world.  Because of this, the OS is more well-tested than any other commercial RTOS, for this reason I would say the maturity of VxWorks is perhaps it's greatest advantage.
--
Q:have you ever patched the OS or only the application sw?
A: In most cases, only application code is updated in space, though it is possible to patch  or replace even the bootrom code in many of the space robots.
--
Q:Can VXWorks 'replace' itself in case of malfunction?
A: I'm not certain what "replace" means in this context; some of our customers have invented ways to detect bad RAM locations, map around those locations, and load vxWorks to the remaining RAM. 
--
Q:How do you ensure that the RTOS does not crash? Even due to a simple NULL-pointer access? Do you have any built-in Crash Recovery mechanisms in WindRiver for Space systems?
A: While debugging with newer versions of VxWorks it is possible to use the MMU to trap accesses to, for instance, the 0-page, to catch accesses to uninitialized pointers, etc.  Once code is sufficiently  debugged, the MMU may be disabled if desired, or the product may be deployed with the protection enabled.  As far as crash-recovery systems, most of the systems in-flight have created health maintenance and monitoring systems, and event logging systems.  Wind River has learned from this, and our newer OS releases have support for configurable event logs and health monitoring.
--
Q:Do you customize Vx Works for individual customers ?
A: We have services experts available to help with everything from the initial installation, to implementation of the entire product, including modifying vxWorks for a particular project.
--
Q:Which the CPU platforms are apted for VxWorks? How to get more info about VXWorks and its using?
A: VxWorks is available for many PowerPC, MIPS, ARM, Xscale, and other CPU types.  Wind River has offices world wide.  Please check at WWW.Windriver.com for office locations.
--
Q: I know you have a trial version of vxworks, but for a person who want to learn it, it expires quick. do you have a slim version with no expiration?
A: We do not have a "slim" version.
--
Q:  Is there a Webinar that spotlights VxWorks? A Demo?
A:  Demos are available of all of our products, pleas contact your local sales office.
--
Q: how do you update your software on space robots?
A: This is highly dependent on the hardware and hardware capabilities used to implement the robots, so unfortunately our customers must usually invent the right methods.
--
Q: AI and VX Works.. Any efforts to embed supporting AI as a native support on VX Works?
A: Though AI systems and some AI capabilities have been implemented using VxWorks, I am not aware of any efforts to embed more than rudimentary AI functionality along with VxWorks; Stardust, DS1, and the Mars Exploration Rovers are the best examples I can think of that incorporated any degree of AI.  Wind River is not planning on adding any AI capabilities to VxWorks (or Linux) at this time.
--
Q: how do you know when to time out communication to a mars rover when there is so much delay and interference?
A: That is left to the folks who implement the radio protocols used by the Deep Space Network to communicate with all the probes/robots/satellites in deep space.  They are experts with communications in deep space.  I expect sometime in the near future that this will all change with Delay Tolerant Networking and the implementation of the InterPlanetary Internet.
--
Q: Is VxWorks migrating from support of ASIC hardware to FPGA's?
A: VxWorks runs on a variety of platforms including COTS boards, ASIC, and FPGA based designs.
--
Q: All application using embedded controller or PCs?
A: Many manufacturers of COTS computer boards for VME, PCI, or cPCI supply VxWorks BSPs for their boards, and Wind River Systems supports BSPs for several COTS boards directly.  I hope this answers the question.
--
Q: Do you see an increase in the percentage of embedded applications that use some form of Linux vs others like pure VxWorks (i.e. not including VxWorks Linux)?
A: To be clear: VxWorks is NOT Linux, the two are not even remotely related to each-other; vxWorks pre-dates Linux by... years.  Likewise, Linux did not evolve from VxWorks.  They are similar in some respects (Posix, networking support, etc), but they are not even "kissing cousins".  Wind River does have our own versions of Linux available along with VxWorks.
Though I have seen an increased presence of Linux in the embedded arena, and an increased presence in the development and testing phases of even software for space applications, I have not seen Linux promoted to controlling a mission (e.g. the primary flight computer) yet.
--
Q: What problems are unique and interesting to  underwater implementations of VxWorks like those faced by MBARI?
A: I wish I had a contact at MBARI for you to ask!  As far as I know, most issues unique to the situation would be shared by all submersible vehicles, and MBARI has excellent experience in dealing with submersibles and software for underwater applications. Once the mechanical issues of sealing out the environment are taken care of, weather it's space or deep-sea, the rest becomes implementing software to control your devices, debugging, and deployment.
--
Q: [referring to an earlier question] Additional details on my earlier question on RTOS decision making ... The specific application is for an RTOS decision to be made for new instrumentation used in human spaceflight ... that is where do I start?
A: I would imagine you would need to start with the specifications for your deliverables: what kinds of certifications you will need, if any, and what kind of constraints the computer needs to operate under.  For instance, will this be a deep-spacee project requiring rad-hard hardware, or will rad-tolerant hardware suffice?  Will you need FAA or military certification, or none at all?  Wind River is happy to discuss the software packages we have and how they may be applied to your project.
--
Q: which vxworks versions have flown in the past and how customized were they ? (components)
A: VxWorks 5.2, 5.3.1, 5.3.1 MER Edition, 5.5.1, and I think 6.2 have all flown in space.  Other versions have flown in military aircraft, etc.  The most customized component of VxWorks I believe would be the DosFS file system on the MER rovers.  The folks at JPL made some great improvements after the SOL18 issue. 

For the most part, we strive to keep the releases "as close as possible" to the standard releases in order to facilitate technical support and software maintainability.  Newer radiation hardened chips are available that are very similar to commercial parts, like standard PowerPC or Sparc chips, and these newer chips run the same (current) versions of vxWorks as everyone else does.  This allows the customer to use our standard technical support for many issues, making experts more available for all issues.
--
Q: There were flash managment issues on MER and the Polar Lander. Could you explain the issue?
A: The flash management issues on MER "A" Spirit was actually more a RAM management  issue combined with a debug feature, precipitated from a "feature" of the DOS file system.  I am not intimate with any problems experienced on Mars Phoenix Lander, but the last I heard MPL's experts have identified a possible application problem.  Given when I'd heard this, I'd expect they may already have tested and sent-up a fix.
--
Q:Is VxWorks already been ported to RAD750 or Leon3FT ? Do you support academic R&D with easy to access software or anything other than that ? Thanks!
A: Yes - Rad750 BSP is supported by BAE Systems, from Wind River's perspective it's pretty much a "generic" PowerPC 750 and uses standard software and tools.  BSPs are available for VxWorks versions 5.x and 6.x, I believe 6.4 is available and a BSP for 6.6 will be available soon.

Wind River does not directly support VxWorks on Leon, but the manufacturer does have a solution with VxWorks 6.x available (I find this very fascinating and would love to "play" with it sometime).  :)

Wind River does have a University Program, contact your local Wind River sales office for details.
--
Q: How many versions/levels are there of VxWorks, and, how do you choose a version of VxWorks for example a Mars rover?
A: VxWorks has been around for a while, well over 20 years.  When I first saw it, the version was 4.0, and there was one version of vxWorks that ran on top of other companies kernels.  Now there are various versions of VxWorks for specific markets, and platforms available to help tailor VxWorks for specific usage.  I would always recommend using the latest version available for your hardware platform that satisfies the needs of your program.

This was the first webinar I've ever presented.  I hope you enjoyed it as much as I did. 

Processing Paradigm: it's all about capacity

When I was a kid anything that was a computer, or had one in it, was pretty obvious. Computers weren't "just everywhere".  Now we've got multiple CPU chips on a board, multiple CPU cores on a chip, different kinds of cores on a single chip, multiple computer boards in a single chassis, ... it goes on.    These things are embedded in everything around us, all this hardware glued together to achieve something.
This brings us to software.  Software has to evolve to keep up with hardware, and with the needs of users.  With all these cores and chips and boards and systems running around, it gets a bit confusing.   Software is what enables everything from multi-processing to "poly-processing".  There's a lot of buzzwords about it - SMP, AMP, POS, VOS, Real time kernel, Separation Kernels... but what does it all mean?  And.. what's it for?

Eventually these things will all become even more blurry. As the realization that the "system" is a composite of the hardware and software all together spreads beyond the aerospace and defense industries, the various specifications will have to cover a fusion of SMP, AMP, and partitioned systems.

For now, though... what I'm talking about includes:
SMP - Symmetric  Multi-Processing - two or more cores running out of the same RAM, each processing parts of a shared task.   The cores may or may not run the same exact OS, but often do.
AMP - Asymmetric Multi-Processing - this is where several separate computers process data as parts of an overall larger system.  They may briefly share data, or have a limited amount of shared memory, but other than this each computer is a separate processing node.
SMP is comparatively  new, but AMP has been available through packages like vxMP for a long time.
SMP and AMP are multi-processor applications, where multiple computers are connected together to achieve a larger goal.  They both bring up issues of concurrency, and offer their own challenges for the implementer.

Way on the other end of the scale are what might be called "poly-processing computers" partitioned systems, where a single computer is virtually carved up and treated as if it's really several computers.   Each of those virtual computers can run a completely different OS - one for it's rich applications and ease of install, one for it's sophisticated communications abilities and firewall, perhaps even a third or forth to handle other kinds of processing. or devices.  You can already share a PC between Windows and Linux (for example) using virtual machine software technologies.  But did you know that similar things are being done with flight systems - using a single high-powered computer to replaces dozens of of single-purpose computers, to reduce weight, cost, and complexity? 

SMP and AMP are for systems where one CPU can not possibly handle the work load before it.   If the work can all be done in one "space" -simultaneously acquire data, process data in place, store data - SMP is a good fit, as all the processors run out of one pool of RAM.  One processor can be tasked with managing / storing the pools of data, another can perform the processing-in-place.

If the work should be handled in separate spaces - say the processing can't be done in-place for whatever reason - AMP would be a better fit. 

In the case of poly-processing (via vitalization, etc), this paradigm fits more where you want to take advantage of the extra processing power of a modern computer to have it replace a number of older computers (or other devices), to save on weight and expense of a newer system with similar function.  A Good examples of this could come from the shipbuilding and aircraft industries, where you have multiple separate mechanical and computational systems that may easily be replaced with smaller, lighter, more capable modern electronics.  Each of these separate systems might still need a small general purpose computer to control them, but that can be costly and add weight.   A single, modern, high-powered computer may be able to replace dozens of smaller, less capable systems.

With any of these paradigms it all comes down to: what is your application, how does it need to be accomplished, and does your system have enough processing capacity to achieve your goals.

Prize: Lunar X

You've heard about it, haven't you?  The Google Lunar X prize?

The X-Prize foundation seeks to encourage radical breakthroughs for the benefit of humanity.
In case you don't know or don't remember the Ansari X prize, it was a contest for a private (non-government) entity too engineer and launch a re-usable space entry vehicle twice within a short time (to prove it was re-usable).  Burt Rutan and his crew at Scaled Composites created The White Knight and Space Ship One, and won the Ten Million Dollar prize.  Even though Paul Allen backed the project with a lot more than $10mil in funding, you have to admit the results are pretty fantastiic: not  only did they achieve the goal, but Virgin Atlantic is going to create a service based on this.  We may yet see commercial flights into space affordable for folks who aren't billionaires.

Back to the Lunar X prize.  The idea here is similar, but the challenge is for a private team to put a rover on the Moon, the prize is Thirty Million Dollars.

Ignoring the question as to weather private industry can build a rover that can withstand space, radiation, the G-forces of liftoff and landing, temperature extremes, vibration of rocket propelled thrust, and to be sure it's understood - Radiation,  the basic problem boils down to this: the expertise and resources needed to launch a rocket capable of carrying everything necessary to get a rover TO the moon, orbiting, and landed in good operating condition, and the rocket itself.

Looking at just the landing part - remember, the moon has no atmosphere, so airbags would be a bit tough to work out, and parachutes are out of the question.  This has to be a retro-rocket landing.  Which means: some sort of airframe, controllable rockets (unless you're really lucky), rocket fuel, etc.  Translation: weight.   

The rover itself doesn't have to do much, but what it does have to do is pretty big for a private group: survive the radiation and temperature extremes of space long enough to land on the surface and send back proof of operation.   To protect the computers from radiation you can shield them - usually with metal.  For the temperature extremes you can make a thermal box with some sort of heating and cooling provisions. Those will need powered somehow,  and probably will need some kind of battery backups at least for parts of the cruise and landing.  That translates to...: weight.

And you need a transmitter (at least), capable of being received on the earth from the Moon, licensed, and all that, and it'll need an antenna to direct the transmission.  That's  more weight.

Can private industry build a rover capable of going to the Moon, surviving landing, and able to run in the extremes of heat, cold, and radiation in space?  I have no doubt at all that it can be done.   I don't know if it would be as capable, elegant, and demure as our Wonder Twins on Mars, but it can be done.
The hard part is going to be strapping it all on top of a rocket capable of lifting all the weight to the Moon, and renting the facility to do it with.  Even if you had the help of Space Ship One to get you into sub-orbital space, you still have to have enough to break earth orbit, at the right time, get you to the moon, and slow you down enough to drop you to the surface... and the rest of it.  I'm pretty sure that for a rocket that big you're going to need a facility like KSC, Arianespace, or Baikonur, I doubt a Space Ship One style launch could manage a craft big enough for the rest of that, mostly due to the weight factor.

Is it impossible?  Many thought the Ansari-X prize was impossible. 

I'm taken back to when I was a litte kid, listening to a 78rpm record we had of a famous speech.  That speech was about hopes, and dreams, and challenges, and difficulties, and the strength of human resolution, and eventually about setting and achieving nearly impossible goals.  And I remember later watching on T.V. as we fulfilled those goals.

Do I think the Lunar-X prize is impossible?  I think it'll only take time.

Mitigating Risk

The phone rings.  A customer has an application destined for a high-risk environment, and somewhere in test they've found a new unanticipated condition.  A "Tiger Team" is formed - a group of experts who've "been here before", who understand the priority and the risks, and know how serious it is.  Many times a Tiger Team may give the go / no-go, the final answer that either saves a mission, or drops all that work into the dust-bin.  Such a team itself represents a *lot* of work, frustration, and time - investigations may run on days, weeks, or months, as long as they reach their conclusion on-time.  And there's the one facet that can't be changed by any engineering practice:  Time.

Developing space, mission- or life-critical applications is serious business.  The deployment environment may be harsh - and much about it may be unknown.  To make sure missions are successful, the known risks are eliminated as well as possible.  Some of these risks may include high radiation, occlusion by the Sun (Solar Conjunction), extreme differences in temperatures, harsh chemical environment, vibrations, or electrical noise.  Some of this you can plan for with hardware, some of this you can't.  You address the things you know about, that's all you can do.

With software engineering, you're also faced with unknowns.  One of the biggest impacts is the unknown date when you'll finally have the actual hardware the project is going to run on.  Using simulators can help out, a lot, enabling some degree of parallel development when hardware time is rare.  I encourage most of my customers to leverage the simulator wherever possible.  It's a good way to kick-start development, enable more hands on the project, and bring new engineers up to speed.

Once you have your hardware, there  may be some hardware-level debug issues.  Having a hardware debugging interface of some sort - an In Circuit Emulator (ICE), for example - can be invaluable.  As long as one has an LED to blink, one can write polled-output routines to display register settings (etc) to assist in debugging a hardware interaction, but nothing is as good as being able to dump desired address ranges (registers, etc) at will.  An ICE can give you that ability to see inside the hardware.

What do you do if the hardware is just not going to be available?  You can use similar boards, processors, etc, but.. it's not the *same* thing.  The minute you change the foundation, everything above it is going to have a new set of interactions with the foundation, and some of those may induce problems.  When updating hardware, timing, electrical, or even errata related to updated chips or even PAL equations may cause engineering delays - even if it's just a newer version of the same board.  Switching entire boards - or from engineering / test boards to "flight qualified" boards - can complicate the issue.

So.. what can you do when you don't have the real hardware on-hand, to help reduce risks associated with developing for that hardware?  In the past few years, hardware emulation tools have come a long way.  Hardware emulation is like a simulator - vxSim for example - except there's a layer of software that's designed to act *just like* the real hardware would, right down to devices and registers within those devices, etc. 

Leveraging known-good (mature) software designs is another way to limit risk.  Re-using application code that is well understood is one example.   Selecting a base of software that's been certified to adhere to accepted standards is another way.  It's always nice to know that software has been scrutinized by eyes other than the manufacturer's, and been found fit.  And there's no substitute for a rigorous test strategy, or following the oft-learned adage: test as you will fly, fly as you did test.

Safe design practices can't eliminate tiger teams completely - there will still be unknowns, still be conditions that aren't discovered until the last moment.  But by combining proper practices, mature systems, and the proper tools, many of the problems that lead to the need for tiger teams may be addressed, and may be discovered far enough in advance to prevent a project from running out of Time.

MS taking some pain out of WinCE?

Microsoft has released WinCE 6.0 - this time as a "shared source" release, with various enhancements.  Some of the big claims-to-fame:  It's the "first commercial hard real-time OS released as a shared-source product".

I'm not sure exactly what this means.  Perhaps they're using their source licensing model - "shared sourcing" - as a qualifier?  Since it's the only RTOS made by Microsoft with sources shared through their license, that makes the "shared" part true.  From that perspective, there are licensing options for source code from most proprietary OS companies, you just can't post it all to the web for anyone to download.

As far as "open" goes, MS is capitalizing on the fear of the GPL - no-one has challenged it far enough yet to know if using a GPL OS would make it so you have to release your *applications* under a GPL umbrella. But ignoring this bit - there are a number of realtime or at least nearly-deterministic kernels available via GPL and Open Source communities (a quick online search yields over a thousand pages with the exact phrase "Open Source RTOS").

Determinism: putting the "Hard" in Real-time.

There are hundreds of open-source OS's - some are even deterministic enough to be "hard realtime".  RealTime is basically "fast enough to keep up" - but hard realtime means "measurably deterministic" - not just fast enough to keep up, but measurably and predictably able to keep up with a given load. WinCE 6 makes the claim that it's hard realtime  But... is it really a hard-realtime system?  Just claiming you're "RealTime" isn't enough - as This Stanford Comparison  shows, some systems claiming to be "Real Time" don't quite show determinism, especially when running as a "loaded" system.  Part of determinism is being able to give "deterministic" execution - that is, you can be guaranteed that the OS itself won't take longer than a given window of time to react to the world.  Execution times for critical system activities that vary over an order of magnitude is not a shining example of "determinism" (see page 4!).

I don't know if you've done any research into this part of the topic - but if you do, you'll notice that many comparisons for determinism messier things like context-switch time and Interrupt Latency (how long it takes the OS to handle an interrupt or change contexts, usually measured under idle and highly-loaded conditions).  I've never seen such a document for WinCE - and most of what I've seen was either from MS themselves or from companies hired by MS to run their tests.  Though their method is scientific, it requires specific hardware and tools, and their graphs don't give you the complete story for context switching or interrupt latency.  Since the method isn't shown against any other RTOS, there's no comparison possible.

WinCE 6 has extensive use of virtual memory spaces, including supporting up to 32,000 processes each with a 2GB addressable virtual address space.  This requires at least changing memory maps, and possibly "swapping" chunks of text and data  between RAM and some fixed storage.  The act of swapping - pushing data to/from physical media - induces indeterminism.

Other than this - there are some interesting developments.

It's being bundled with Visual Studio with an integrated Platform Builder plug-in.  Microsoft says ""Under one roof, you have the entire development chain from device to application".  Eventually it might have some sort of data visualization tools, perhaps some kind of system event display, maybe even.. hardware-assisted debug... could it be that Microsoft may be joining the DSO trend?

Does it have a motor?

Back in the 80s there was a car commercial, where a small crowd of doubters are looking over a neighbor's new car.  Amongst the comments and questions about the options that make a car comfortable comes the quip "Does it have a motor?"

It's true that a car can be a car without a motor, or any of the comfort options, but let's face it - with no motor, it's not worth much.  Same thing for a car in snow country without a heater, or a car in the Mojave without working A/C.  Some of the "comfort options" become necessities depending on the intended use.

The same thing can be said for Operating Systems.  In this blog entry on Applications for Linux, DMarti brings up a very big issue facing Linux today:  What exactly is in your pot of soup?  The big applications producers are starting to take notice of what it means to support Linux.

Linux is a lot like soup - how it turns out is a combination of the ingredients and how you cook them.  It seems like most providers are grabbing whatever freezes they "need", applying a specific set of patches, and some proprietary patches or enhancements, rolling up a release, and dropping it out on FTP with one or two "demo" platforms.  They may offer some open-source applications they've ported to their release, or some of their own packages.  But as DMarti points out - the bigger players are going to want a standardized platform to base their support on, not just the results of some "compliance" tests, and not just one released platform.  And that means a known starting point, free of proprietary changes that may inhibit future growth or portability.

Although software manufacturers like Adobe may want to start off with a big name like Red Hat as a platform, they would be wise to take a conservative approach.  In order to run on a wider audience than just the "preferred platform", the implementation itself should be based on a locked and well-known "version" of Linux, without dependencies on any proprietary extensions.  I think as time goes on, this will be a strategy more and more software manufacturers follow.

Wind River Systems has our "own" release of Linux.  But ours is a little different than others.  We've taken a bit of an idealistic approach to our release.  We start with a known kernel from kernel.org, and a known set of patches and libraries - these are all publicly available.  We apply the patches in a step-by-step method, and provide instructions on exactly what we did.  We provide BSPs to support Linux on specific hardware.  We don't have proprietary extensions built-in to the base offering, we don't have patches or extensions that may or may not ever make it into any other Linux release, we don't have software included that would "gate" a user's ability to comply with a standard or upgrade to future patches from kernel.org (etc).

Any big player would understand how dangerous it is to tie one's product or future to a proprietary standard or system.  History is full of lessons where superior technologies lost out to open standards because the open standards made for increased usability and lower price-points (Betamax, anyone?).   I think there will be a chosen Linux platform upon which the likes of Adobe standardize, but I also think the lessons of the past will guide a choice that can be accessed by anyone, not just a single proprietary release.