Emeka Nwafor

Kill Bugs: Volume 1

The Bride: [Japanese] I need Japanese steel. 

Hattori Hanzo: [Japanese] Why do you need Japanese steel? 

The Bride: [Japanese] I have vermin to kill. 

Hattori Hanzo: [English] You must have big rats if you need Hattori Hanzo's steel. 

The Bride: [English] ... Huge. 

(from Kill Bill: Vol. 1)

I will probably need the help of a therapist to assist me in decoding why I think of that passage from Tarantino's great movie whenever I think of our portfolio of multicore development tools. When I ponder the personality of our development tools (i.e. the fearless Hattori Hanzo and his sword) and the risks presented to our customers in getting their multicore software to be correct (i.e. vermin - in reference to the challenges and bugs, not our customers) the exchange works for me. 

I digress.

We just announced a new release that extends our capabilities for multicore and virtualization for embedded applications. As always, I'm very proud of our development tools engineering team. A special shout-out to our teams in Canton and Vannes for their efforts in delivering this release.

Last week, my colleague - Bill "The Truth" Graham - blogged about the importance of development tools in reducing the inherent complexity of the task at hand. This latest release - actually, two product releases - is special because we have extended our hypervisor support to support Intel Xeon processors AND we have extended our development tools capabilities to now include "hypervisor-aware" debugging with our JTAG debuggers.

Why do I think this is important?

My mind goes back to a conversation with one of our early adopters of our hypervisor technology. The folks we met with were responsible for developing and stabilizing the software platform. The hypervisor technology is a critical component of their project as they are looking to consolidate multiple processors onto a single multicore device. They depend on our technology for memory protection to ensure the reliability of their system. One of the clearest memories I have from that day was when they asked "how do I debug my system running on the hypervisor"? The question was not asked out of curiousity - in their situation having assurances that we could address their needs in this area was critical for success.

So, what have done to help Kill Bugs when using virtualization?

The latest release of our Wind River On-Chip Debugging solutions enables debugging of a system running on the hypervisor without the need to add a target resident agent (see Figure 1). This new capability helps developers with tools to develop and debug hardware initialization code, virtual board bring-up, kernels, and drivers. When developing software for virtualized systems, these solutions give developers the confidence to know that they can set breakpoint, watch memory, or analyze stack frames.

Figure 1 - Overview of Hypervisor-aware debugging using Wind River On-Chip Debugging [click to enlarge]

This capability borrows from the concepts applied when debugging Linux processes using a JTAG debugger, but extends them to include the concepts of virtual boards running on the hypervisor. One of the special attributes of our solution is that we go beyond the MMU support provided by the processor to provide visibility and control over all of the virtual boards running on the system - not just the instance that is currently executing.

There's much more that can be discussed on this topic. More information and videos can be found here and there on our site. More importantly, we would like to hear your views on how to Kill Bugs when developing virtualized systems.

2009 Autumnal Equinox

September 21st marks the last full day of the memorable summer of 2009, a summer that was memorable on several fronts - including personal and professional. My summer of 2009 includes memories of rainfalls of near biblical proportions, personal triumphs and losses, the passing of several social icons, and the evolution of the embedded software market.

There is some small debate as to when summer officially ends and when we must stop wearing white to work. If the sun could talk, I would guess that it would suggest that summer officially ends and fall begins at the moment of the autumnal equinox. This is the moment in the September when the sun is vertical to the equator. This year's autumnal equinox will occur on September 22nd at 17:18 ET (21:18 UTC) - just about five hours and eight minutes after the opening keynote at the 2009 Intel Developer Forum being held in San Francisco, California. This is also about nineteen and half hours before the day 2 keynote titled "Developing for the Continuum of Intel Platforms".

The team has been working intensely to prepare for IDF and the Day 2 keynote. We have contributed a short segment that we call "A Day in the Life of an Embedded Multicore Developer... in under 5 minutes". This portion of the keynote is intended to convey how Wind River's development tools and run-times work together cohesively to simplify dealing with the challenges of adapting existing uniprocessor code for multicore. The demo will showcase the Tilcon Graphics suite, Wind River Workbench, and a preview of soon to be available JTAG debugging solutions for Core 2 Duo and Xeon processors. (This is an extension of the Atom configuration of the solution that was made generally available this past July) I am very appreciative of the effort the team put in to create the demo systems.

These types of demonstrations are going to be important as we continue to evolve in the multicore era. A few months ago, I participated in a multicore development tools panel with Dave Stewart of Critical Blue. Dave made a comment to the effect that embedded developers "don't know what they don't know about multicore development" [editor's note: I am paraphrasing] and stressed the importance of education on the different technologies and techniques involved with multicore software. This comment resonates with me and I feel that it puts onus on vendors to support our "techno-speak" with demonstrable proof-points.

If you have an area of interest as it relates to our development tools support for multicore, drop me a comment and I'll put together a posting or video that helps to show the solution in action.

[You can follow Emeka's feed on Twitter - http://twitter.com/enwafor ]

What's the 4-1-1 on 3.1.1?

We have just announced that we are taking orders for Wind River Workbench On-Chip Debugging version 3.1.1. Here's some infomation (i.e. "the 4-1-1") on this release. Version 3.1.1 is a significant update to the software and firmware that powers our JTAG debugger units - the Wind River ICE 2 and the Wind River Probe. A couple of the highlights include our support for RMI Corporation's XLR and XLS processor lines, a new capability that strengthens our support for multicore and multithreaded processors. This component of the release was the result of some hard work and great teaming between the folks at RMI, our professional services organization, and our engineering team in Canton. The other thing that has our team excited is that version 3.1.1 introduces support for Intel Architecture starting with support for the Intel Atom product family. Intel Architecture support in our on-chip debugging solutions extends our existing coverage for ARM, MIPS, and PowerPC. These different architectures are supported using firmware updates to the same physical hardware - this is something that our clients have told me that they appreciate since the same debug solution can be used across multiple projects in there heterogeneous environments. I tend to feel a certain amount of pride everytime we execute a new release, but I'm particularly proud and nostalgic about this release. I'm proud because our clients are jazzed that we are expanding our optimized-for-multicore JTAG debugging solution to the Intel Architecture and are pleased that we have a non-intrusive debug solution for Intel that helps to abstract the debugging of operating systems and the stuff running on them - including operating systems like VxWorks and Wind River Linux.

So why the feelings of nostalgia? It is because the 80x86 instruction set was the first architecture I worked with.

Coming out of university, I started my professional and embedded software career with a company in Montreal that developed point-to-point TDMA-based wireless networks. The company had been primarily a hardware company, but because this "software thing" seemed important, they were ramping up their software engineering team. When I was brought in, I was asked to study if-and-how C++ could be used in this environment. We were also asked to investigate techniques to improve the quality of our software design processes by using formal requirments management techniques, OO-techniques, and state machines to improve the quality of our design activities.

In those first few weeks of my employment, my mornings would consist of researching new software engineering techniques and my afternoons were spent in the lab learning how to debug the distributed system. At this time, most of the system was running on an Intel 80C186 processor and the software consisted mostly of undocumented assembly language that had been written by a former employee who had left to pursue other interests because he had become independently wealthy. (This was the polite explanation that was given to me)

The afternoons were - let's just say - "difficult". My mentor "V" was a brilliant engineer, but not much of a talker. I would later describe "V" as a man who was put on earth with a limited number of words that he could speak for his entire life, and he was using each and every one of these words sparingly. So my afternoons would be spent watching "V" debug the system from a UNIX terminal that was connected to one or more HP 64700 emulators attached to either a central station or a terminal station. "V" was also a six-sigma UNIX blackbelt and a smoker. So, we would sit together at this terminal - with smoke billowing all around the screen - and I would watch as "V" would type a flurry of shell and emulator commands that would grab code from SCCS, compile it in the background, load it into the emulator, and then display assembly code in the debugger window. I would struggle to keep pace as "V" entered a flurry of commands - vi, grep, sed, make, get, sccsdiff, "yy", "set trigger on read/write 0F0B83E00H", "set trigger on I/O output to 040H"  - while navigating all over the filesystem. It was intense - so intense that it didn't feel appropriate to ask questions while all this activity was going on.

I was totally and completely lost.

Usually after an hour or so of analyzing the 80x86 assembly code and traces, "V" would speak saying something like:

"Ah-ha! Can't you see it?! The blah-blah hasn't received a message from the blah-blah, and the connection memory updates are not getting to the blah-blah. This is why we don't have dial-tone."

There would then be another flurry of typng activity culminating in a "make" command. And then he would walk away. I usually used this time to review the history of the commands "V" had typed in the shell. In hindsight, it really felt like trying to understand the hieroglyphics left behind my another civilization.

This went on for weeks (it seemed longer) and then due to some scheduling screw-up, all of the senior software engineers who knew the system were on vacation at the same time. At that time our engineering director - who looked like a Hellenic version of the Malboro Man - was a real "getting things done" kind of guy. There was an ASIC for our next generation terminal station that had to get done and they needed some software guys to test it out. The director walked into the pit where the new engineers were researching software engineering techniques and I remember the discussion going something like this:

Malboro Man: "Sonny, what are you doing for a living?"

Me: "I'm reading about C++ for our next generation software development efforts..."

Malboro Man: "Sonny, does this look like an [expletive deleted, gerund form] library?! 'M' needs to debug the ASIC in the lab before he can go on vacation. Go help him."

We walked to the lab together and I'll never forget the look on "M's" face when he saw that some new grad was between the debugging of his $100K ASIC and a much needed vacation with his family. The next week and a bit was challenging, but proved to be rewarding. We connected the HP 64700 emulator to the new board.This activity in itself was noteworthy just based on the size and cost of these emulators (see figure below). As a new grad, I always felt nervous about dropping one of these emulators that cost about the same amount as my annual salary at the time. Because they were so big, they were almost always mounted in some dangerous configuration.

With the emulator mounted to the board with the new ASIC, I sat down in front of the UNIX terminal, brought out my notebook of commands that I captured, and together with "M" we somehow (read: miraculously) verified the ASIC and "M" went on vacation with his family. For me the bigger accomplishment was learning from the experience of debugging the system at the HW-SW interface level. I quickly developed an appreciation for the visibility that on-chip debugging provides into the system. This knowledge of the system helped  me in later phases of my career as we designed both the hardware and the software for our next generation systems. Reading the specs is one thing, but using an on-chip debugger to gain insights into the execution of the software proved to be essential capability required to help verify and optimize our software.

This year's TechInsights Embedded Market Study by TechInsights confirms that "Testing and Debugging" activities continue to take up the most time in our embedded software development products. Embedded developers need quality debugging tools to help address these challenges and meet their schedules. What's also encouraging is that when asked the questions "Which of the following are your favorite/most important software/hardware tools", "debuggers" rated just a hair behind "compilers" and "JTAG/BDM" debuggers finished in the top-five. These on-chip debugging solutions are essential for embedded and device software development.

This summer's release brings this on-chip debugging capability to an avalance of [new uniprocessor and multicore processors from Freescale, Intel, Broadcom, and Texas Instruments] and puts Wind River in a great position to help engineers tackle their debugging challenges so that they can enjoy more of their summers.

E

P.S. If you have an on-chip debugging story that you want to share, I would be glad to hear it.

[You can follow me on Twitter - enwafor]