Alex Wilson

Software Certification

I wonder why it is always the case that the period of time just before year end is suddenly very hectic! There must be a universal law about that, something like Murphy's Law...

Anyway, I have just returned from 2 weeks of travel around Italy, Sweden and the US visiting customers and presenting our solutions. I always enjoy meeting folks and seeing what they can do with our software as I love to see the latest and greatest devices; and so I have been negligent in posting to my blog for that time and noticed over the last few weeks some noticeable events have taken place.

Firstly we had the final certification of the Airbus A380, a long awaited event, especially after the reported delays [See blogs by John Bruggerman and Paul Parkinson] . I noticed an interesting point in the press release on this. Namely that the certification had begun with system tests in 2001. This shows you the amount of time and effort it takes to take a complex aircraft such as the A380 through certification.  I was amused by the lack of software mentioned in the press release, I guess most readers don't want to  hear the aircraft is flown by software these days?

Secondly, we had the first flight of the long awaited F-35 Lightning II. The UK government has also finally signed up to the next phase of the project which means we can expect to see some UK business for F-35 over the next few years. It is kind of interesting seeing these arguments over technology transfers in a company who has embraced Open Source in a big way!

Both of these aircraft are using RTCA DO-178B for software certification, so I think I'll take a look at that standard. I wonder if the UK F-35 will be accepted with DO-178B or will they need our own DEF STAN 00-56?

Which brings me to the point of my blog, which is why are there so many software safety standards out there? It seems that this area of software development that does need to consider a little standardising of it's own, or even using a standard of standards (I guess you can do that if (as Paul Tingey discusses) you can have a "consortia of consortia"?)

It seems each time a new aircraft (certainly military), or unusual project begins, committees form and new standards for software safety emerge, even though the FAA and RTCA have been using DO-178B for a number of years, and very successfully too.

Some of the customers I met with do have another issue on this; that is they are creating devices that could put lives at risk in systems that do not have an applicable safety authority, such as land vehicles or submersibles (which reminds me - I must find out if the road safety authorities look at software safety).

The time spent on creating these "new" safety standards would be better spent utilising what already exists - if all of these efforts had been used to enhance best practices around the use of DO-178B wouldn't that make the device software world a whole lot safer? After all isn't that one of the key tenets of DSO?

Compliance vs Conformance?

What is the difference between compliance and conformance and what does that mean to you and me? Or more importantly what does this mean to your project?

Well, compliance is an informal industry term generally accepted to mean the system provides support for some of a given standard. Technically, an OS that provides only one API of a POSIX standard is "compliant," though not very useful. Vendors of compliant systems are generally expected to offer documentation describing which parts of the standard are supported, and which are not. Wind River for instance, has such a statement to show "compliance" levels of its VxWorks Operating System against the various POSIX standards. This reads like 100% conformance to SCA 2.2.1 (another POSIX Subset for Software Defined Radio) etc.

Conformance on the other hand is a recognition of  formal testing, that prove that an operating system provides 100% support for a given standard. Certification of conformance is awarded to an operating system after an official Certification Authority has reviewed not only the results of formal testing, but formal conformance documentation as well. In the case of POSIX this is well covered on the IEEE and The Open Group sites. Products that are POSIX certified can claim conformance, and in doing so also publish a list of which of the optional components they support to exactly define what support they provide. This statement also lists the configuration of the system, and the test environment required to re-produce the tests.

To do this the company involved must purchase an official test suite  from a certification authority , run the tests and  submit the results for certification for conformance. As of November 2003, The IEEE and The Open Group extended POSIX Certification to the 2003 Edition of IEEE Std 1003.1. This has subsequently been extended to include IEEE Std 1003.13-2003 (what we are interested in),  for  both  the PSE54 Multipurpose Profile and  the  PSE52 Real time Controller Profile. The test results are supplied to the Certification Authority  such as the IEEE or The Open Group, in order to audit the results and confirm they have all passed. This along with the Conformance report then give a company the right to say they are conformant to the standard tested against.

Certified conformance is obviously a much better indication as to a product ' s suitability, if POSIX is a requirement for your system, as you don't have to rely on marketing data sheets showing "compliance" but know the product has been audited and tested to the standard involved. You as an application builder can then tick the box and get on with developing a product just using the chosen standard.

As John Jones pointed out on my last blog only one company is listed as having run these tests and that for 1003.1, the complete POSIX standard, this in itself is interesting as the older POSIX standard 1003.1-1998 had many companies listed against it (from memory as I could not find the old listing anymore!

I wonder why this is? 

One reason for the lack of testing at the Real Time OS level is that the PSE52 and PSE54 tests are  relatively new, not only to the POSIX community, but also to the DSO world and you have to adapt these tests to run in a DSO environment (such as porting the test suite from a UNIX based self-hosted environment to run in a client-server mode, as a device target doesn't always have the system resources to support running the large test framework)

Another reason could be the usual product development issue of scheduling in these additional workloads into an already heavily oversubscribed feature development schedule along with demands for more "exiting" technologies such as Mobile-IP  [Webopedia] or Remote Diagnostics [Wind River Management Suite].

Or maybe the demand to see these POSIX certified products just is not there yet?