Paul Parkinson

Astute optronic mast case study

I don't often have the opportunity to discuss how Wind River's Aerospace and Defence customers are using our technologies in their applications due to security restrictions and commercial confidentiality.

So, I am very grateful to Thales for allowing us to announce that they have used VxWorks for their latest generation optronic mast which is being used on the UK Royal Navy's Astute class submarines.

If you're interested in the challenges Thales faced in migrating their Ada application from a custom Digital Signal Processor (DSP) architecture to COTS multi-processor PowerPC architecture running VxWorks, and how they overcame them, the case study (PDF) has recently been published on the A&D Customers page.

Rhapsody && Coverage Analysis

I've recently been working with a customer to integrate Wind River Coverage Analysis (formerly known as CoverageScope) with Telelogic Rhapsody within their development tools framework for a mission-critical system.

At first glance, it might seem strange to integrate a code coverage testing tool with a UML design tool, as these aren't even adjacent phases in the software development life cycle (implementation being between them). However, the advent of Eclipse in recent years has helped to break down the silos which used to confine tools individual development life cycle stages and improved development work flow.

The integration enables an application to autocode-generated from Rhapsody and built with Coverage Analysis instrumentation and downloaded and run on the target processor, with coverage results being continuously streamed back to the host and displayed in the Coverage Analysis GUI. This provides immediate feedback of of the level of coverage, and the coverage trend against time (useful for confirming if the test suite tests more and more code over time, or whether it is testing the same code over and over again).

The aspect of the integration which left the biggest impression on me though, was how it enabled the customer to confirm that the right code is executed in response to an event being injected at the UML model level. This is an important aspect of system testing, and it is often difficult to demonstrate the traceability from design to implementation. This was achieved by building the application with both Rhapsody animation and Coverage Analysis instrumentation, and then run or animated on VxWorks with events being injected from Rhapsody and the executed code was shown in Coverage Analysis.

If you're interested in seeing this for yourself, the integration steps are now documented in AppNote-344 on Wind River Online Support website (login required).

nEUROn UCAV programme

In case you missed the news, the article 'COTS for unmanned flights' (Electronic Product Design, 22nd April 2008) describes how the nEUROn UCAV programme (Airforce Technology) has standardized on the VxWorks 653 RTOS.

nEUROn is a technology demonstrator programme involving five European companies, which in itself is not unusual, but it does have a significant objective - to demonstrate the maturity of technologies by producing modular safety-critical avionics systems running on COTS-based on-board computers. The development of avionics systems for a UCAV are in some ways even more challenging than for a military fast jet, given that it will provide a similar level of capability, but yet has less Space, Weight and Power (SWaP) available. This is one of the driving factors behind the ARINC 653 software architecture, enabling multiple applications to be hosted on a common computing platform (see 'ARINC 653 software weighs less' for background).

However, this advance results in a further challenge, that of being able to support modular and incremental certification of the IMA platform. This is necessary because of the very significant costs of safety certification under RTCA/DO-178B and EUROCAE ED-12B, especially at the higher software integrity levels (SIL). If an avionics platform comprising, say ten applications and several million source lines of code (SLOC) has been developed, and one of the applications undergoes an upgrade, it would be very time-consuming and prohibitively expensive to re-certify the whole platform. Alex Wilson covers these issues in more depth in his webcast 'Towards Incremental Certification with ARINC653'.

Fortunately, the experiences gained on recent IMA programmes in conjunction with the evolution of ARINC 653 standard has resulted in the mature VxWorks 653 implementation which fulfills these requirements and will provide a critical foundation for the objectives of the nEUROn programme.

I'll look forward to the nEUROn taking off in 2010.

ARINC 653 partition jitter & scalability

Last week, I was looking at a customer's avionics software architecture to determine whether it might be possible to migrate this to an ARINC 653-based Integrated Modular Avionics (IMA) environment. This raised some interesting questions in terms of performance and scalability.

If you consider the implementation of an avionics system using an ARINC 653 software architecture, the RTOS enforces temporal partitioning of the safety critical applications running in the individual partitions through the use of ARINC 653 time slot scheduling.  (ARINC 653 defines a schedule consisting of major frame comprised of a sequence of minor frames which specify the execution time of an individual partition for a fixed duration; at the end of the major frame the schedule repeats). However, the ARINC 653 scheduling implementation needs to be deterministic, i.e. the next partition should start execution at the exact time predefined in the ARINC 653 schedule, or very close to it. The delta between the defined time and the actual time is known as partition jitter.

So, what approaches can be used to minimize partition jitter?

Let's consider the ARINC 653 architecture - it actually performs two types of scheduling: (1) time slot scheduling of the partitions; and (2) priority-based preemptive scheduling within the partitions. Let's assume that the RTOS is running in the processor's privileged supervisor mode, and the applications are running in the processor's user mode and in different memory contexts (in order to provide spatial partitioning - another key requirement of Integrated Modular Avionics).

Now, if the time slot scheduler and the priority preemptive scheduler are both implemented in the RTOS kernel, this could create a significant overhead for the RTOS kernel in handling a significant number of system calls associated with each processes context switch within a partition. This could lead to performance degradation as the number of partitions in the system increases (as the ratio of the total number of processes and partitions divided by the major frame duration is likely to increase also).

However, an alternative approach would be to only implement the time slot scheduler in the RTOS kernel, and implement the priority preemptive scheduler within the partition in user mode. The advantage of this approach is that the priority preemptive scheduler can perform schedule application processes in user mode without having to perform a system call into the RTOS kernel and the associated overhead of user/supervisor context switch. This approach not only helps to minimize partition jitter, but also make the architecture very scalable, which is why it is used in VxWorks 653. (I mentioned this scalability in an earlier blog, where VxWorks 653 was running with sixty-five ARINC 653 partitions at Avionics 2007 Show in Amsterdam).

Of course, it's also important to be able to detect at runtime whether partition jitter ever exceeds a predefined threshold, and VxWorks 653 implements this through the Temporal Violation Detection  mechanism, which would raise an alarm to the Health Monitoring System.

By the way, the phrase Temporal Violation Detection sounds more like a concept from Einstein's Special Theory of Relativity (wikipedia) than software engineering, but I don't think we're going to have to consider the impact of Time Dilation (wikipedia) in ARINC 653 systems just yet, as we're still a long way from traveling close to the speed of light...

ARINC653 software weighs less

I'm about to head off to the airport for Avionics 2007 in Amsterdam. I'm really looking forward to hearing the latest developments in civil and military avionics at conference, especially in relation to one of my favourite subjects - Unmanned Air Vehicles (UAVs).

I am also looking forward to showing a VxWorks 653 system which was created by a Wind River wizard (codename: Uncle Larry). This has not just a few applications running in different ARINC 653 partitions, but sixty-five.

Why is this relevant? Well, because aircraft manufacturers are under increasing pressure to add more and more functionality to avionics systems; and at the same time they need to reduce weight and volume of these systems. In recent years, the approach of migrating a number of  federated applications onto a common Integrated Modular Avionics (IMA) platform has become increasingly prevalent, and this is being widely adopted in conjunction with the ARINC 653 software model. Of course, an IMA platform needs to have sufficient computing power to host many applications, but the sixty-five partition system shows just how scalable this model can be.

So, software can weigh less.

RTLinux acquisition from FSMLabs

I was really pleased to see the news about Wind River's acquisition of RTLinux hard real-time Linux (ZDnet) from FSMLabs.

Linux is already starting to be used in certain types of Aerospace & Defence applications due to its maturity (which I discussed in the previous blogs: 'Linux in Aerospace & Defence' and 'Linux fit for Defence'), but these are mainly non real-time classes of applications where Linux has provided a good technical fit to the requirements. However, RTLinux technology extends the reach of Linux into real-time, because it achieves hard real-time performance where as some other real-time Linux implementations have only achieved soft real-time and have also forked the Linux kernel with proprietary extensions which could be a maintenance nightmare on A&D programmes with long in-service lifetimes.

You may be thinking that Linux as provides high performance on modern microprocessors, so does the distinction between soft and hard real-time really matter? In my view, it matters a great deal. 

• A real-time system is one which needs to complete a number of operations correctly by a specific deadline.
  
• In a hard real-time system, completion of the operations after the deadline is unacceptable and may even result in system failure.
• In a soft real-time system, deadline misses can be tolerated with degraded system performance.

Back in the days when I wrote host platform device drivers and libraries for high-performance multiprocessor DSP VMEbus (VITA) systems, I needed to develop stress tests to assess how the device drivers would perform under the severe loads which might be encountered in a deployed application. One test involved sending VME interrupts from a DSP (wikipedia) slave board to a host platform as fast as possible.  Of course, the DSP was able to generate interrupts at a very high sustained rate, and I was able to accurately measure with a VMEbus analyser the interrupt response times of the host platform and its operating system (which was either a multi-user OS or an RTOS). 

What was interesting when comparing the performance of the various operating systems, was not just the typical interrupt response times, but the variation between best case, worst case and average response times for each of the operating systems, especially in relation to increasing system load. This revealed that although some of the soft real-time platforms could achieve fairly respectable interrupt response times most of the time (especially under fairly light load), under severe load, the interrupt response times could vary dramatically. The hard real-time platforms (VxWorks in particular)produced much less variation.

The outcome was that the suitability of a particular host platform to the performance requirements of a specific application wasn't determined by its best-case response time, but on its worst-case response time. For me, this lesson has a direct bearing on the suitability of Linux for hard real-time applications; as RTLinux's hard real-time performance not only provides faster response, but also much better determinism and predictability, this will enable the adoption of Linux in a new class of applications. In A&D, I expect that Linux will now be able support certain types of radar and communications applications, etc.; but of course there will always be applications with such as EW (electronic warfare) systems and other sensors which even RTLinux will not address but they will instead will exploit the extreme hard real-time performance of VxWorks for a tactical advantage (for example, sub 2 microsecond interrupt response time on a 1GHz PowerPC), but these are at the two ends of the hard real-time spectrum.

So, now it's Linux - fit for hard real-time defence.

VxWorks 6.4 POSIX PSE52 Conformance Certification

In the post 'Great News for POSIX', Alex Wilson commented on the announcement that VxWorks has becomes the first operating system to achieve certified conformance to POSIX PSE52 Realtime Controller 1003.13 standard. Alex has also discussed the different POSIX profiles here.

I was thinking about what this announcement means for developers? I have to confess that it's been a while since I spent all my time writing code, but I can clearly remember when I used to write device drivers and interface libraries under SunOS, Solaris, OS-9, LynxOS and VxWorks (wikipedia) to interface enterprise UNIX platforms and RTOS platforms to high-performance digital signal processor (DSP) VME boards.

The internals of each OS were of course quite different, which had an impact on the design of the device drivers, especially where multithreading (wikipedia) and MT-safety were concerned. The implementation of MT support provided by the OS platforms varied to a certain extent, so careful design was required to achieve a consistent application library API across platforms to aid portability and to ensure that the implementations behaved in a consistent manner too.

The most logical way of doing this was to use POSIX, as this was often at least partially supported on each of the OS platforms. (The Solaris implementation of threads had some differences, but this was well covered by Bil Lewis and Daniel J. Berg in 'Threads Primer - A Guide to Multithreaded Programming' (p117, according to the edition which is still on my bookshelf). By using an abstraction layer, it was possible to avoid exposing any OS dependencies to the user in the application libraries API.

Back to the present, now that VxWorks 6.4 has achieved 100% PSE 52 conformance, I can see how this will make it easier to implement portable libraries (and applications for that matter) on different real-time platforms. In A&D, I can imagine that this will be useful for programmes undergoing mid-life update, who want to migrate off legacy platforms; and for new programmes, who want to develop portable code to enable future portability.

Is safety-critical ready for multi-core?

You may feel that you're experiencing déjà vu reading this blog, but the title 'Is safety-critical ready for multi-core?' is different to an earlier blog, which was 'Is multi-core ready for safety-critical?'.

In the earlier blog, I discussed some recent developments in multi-core processors in the context of their suitability for use in safety-critical systems. In this blog, I want to discuss how a particular type of safety-critical system could exploit the capabilities of multi-core devices. However, before I dive into the details, I should really outline the different types of safety-critical systems (at least in the simple categories which I like to use). These are broadly as follows:

• Type 1: Single application, safety critical
• Type 2: Multiple applications, single level of safety-criticality
• Type 3: Multiple applications, multiple levels of safety-criticality

Type 1 is very common in industrial automation, transportation, defence, and of course aerospace (where it is known as safety-critical federated avionics). This is readily supported by a safety-critical COTS RTOS implementation, such as VxWorks/Cert (subset of 5.4) or the forthcoming safety-critical implementation of VxWorks 6.x.

Type 2 is really a variation on Type 1, but the certification requirements are the same, everything is safety-certified to the same level. In aerospace, this is one instance of Integrated Modular Avionics (IMA).

Type 3 is where things become more interesting, and can apply to a range of different types of applications. In aerospace, this is another instance of IMA, but the different levels of safety criticality (e.g. RTCA DO-178B Level A and Level D) either require that the whole systems is certified to the level of the most critical application (which can be very expensive), or by implementing robust separation of applications.

The ARINC 653 software model provides robust separation through temporal and spatial partitioning, which is implemented in the VxWorks 653 RTOS running on a single processor core.  This architecture is well suited to systems which exhibit cyclic/periodic behaviour and/or perform synchronous I/O, e.g. flight control systems, but there are other types of applications which do not exhibit cyclic/periodic behaviour and/or perform a lot of asynchronous I/O, and may not fit well with the ARINC 653 timeslot scheduling model. These applications sensors such as radar warning receivers and electronic countermeasures (wikipedia) on military fast jets.

The NATO ASAAC IMA model might provide an alternative approach for these these types of application, but this does not have widespread commercial support yet. However, the advent of multi-core introduces another possibility - a system comprising two applications with different-levels of safety-criticality, but running one on a separate instance of a priority-based pre-emptive scheduling kernel on each of the processor cores within a dual-core device - known as  Asymmetric Multi-Processing or AMP (Wind River).

This could potentially be implemented by running the safety-critical subset of VxWorks 6.x on each core in AMP mode.  The VxWorks 6.x multi-core Board Support Package (BSP) would of course need to set up the memory map and I/O devices correctly to avoid coupling (FAA, MS-Word), and any inter-core communication mechanism would need to be robust (i.e. faulty messages from Level D processor core could not impact the operation of the Level A processor core). 

This approach would provide the immediate response required by the sensor applications, and also robust partitioning. In short, the best of both worlds. You could also view this as shrinking the form factor of a traditional dual-processor VME or CompactPCI system onto a single piece of silicon.

Now, when can I get hold of a Freescale PowerPC MPC8641D or Intel Core Duo board to try this out?

Is multi-core ready for safety-critical?

I recently attended the Military & Aerospace Electronics conference in the UK, at which a number of technical presentations were given on new technologies relevant to this sector, including an excellent one by Gordon Hunt of RTI on the Data Distribution Standard (which probably deserves a blog entry in its own right). However, the presentation which I want to comment on here is 'COTS VME processing: 4-way SMP Intel vs. Quad Distributed PowerPC' given by Richard Jaenicke, Director of Technology Integration at Mercury Computer Systems.

Richard gave a considered overview of the different types of multiprocessor architectures and their suitability to different types of application. He then went on to discuss the relative merits of a number of multi-core devices, and their suitability for deployment in the various multiprocessing architectures. This included insights into the performance speed-up of the multi-core Intel Core Duo over single-core Pentium architectures, and why the Dual-Core Intel Xeon is more suitable for 4-way symmetric multiprocessing (SMP) architectures. He also went on to discuss the Freescale PowerPC MPC8641D dual-core processor (Freescale: architecture diagram), and how this can be used in a number of different SMP and distributed shared memory architectures. I was struck by two things: firstly, that the most suitable processor choice really is dependent on the overall system requirements (i.e. it's a question of different horses for different courses); secondly, that Richard had given a very interesting and informative presentation without even talking about Mercury's products (note to self: must make technical seminar presentations more interesting in future).

This was a very useful and timely backgrounder, as I've noticed an increase in interest in using  multi-core devices for A&D applications recently. The benefits of multi-core are becoming well-known: increased performance, higher processing densities, reduced physical footprint (things which have been key factors in the development Integrated Modular Avionics architectures, coincidentally), so I am interested in how multi-core could be exploited in a diverse range of A&D applications, not just avionics.

Since the event, I've been thinking about how software platforms might be implemented on top of these multi-core and multiprocessor architectures. Tomas Evensen recently discussed Intel multi-core support, and Maarten Koning has also considered fault-propagation in multi-core devices, but what really interests me is the suitability of multi-core devices for safety-critical systems. The type of software platform which can be supported seems to be dependent on the level of flexibility of the multi-core architecture, and whether the cores can be configured to operate in an independent manner, or whether there is coupling and/or dependencies between the cores as there are obvious ramifications in a safety-critical environment.

The next question which springs to my mind is: What types of safety-critical software platforms could be supported on multi-core architectures?