Emeka Nwafor

Kill Bugs: Volume 1

The Bride: [Japanese] I need Japanese steel. 

Hattori Hanzo: [Japanese] Why do you need Japanese steel? 

The Bride: [Japanese] I have vermin to kill. 

Hattori Hanzo: [English] You must have big rats if you need Hattori Hanzo's steel. 

The Bride: [English] ... Huge. 

(from Kill Bill: Vol. 1)

I will probably need the help of a therapist to assist me in decoding why I think of that passage from Tarantino's great movie whenever I think of our portfolio of multicore development tools. When I ponder the personality of our development tools (i.e. the fearless Hattori Hanzo and his sword) and the risks presented to our customers in getting their multicore software to be correct (i.e. vermin - in reference to the challenges and bugs, not our customers) the exchange works for me. 

I digress.

We just announced a new release that extends our capabilities for multicore and virtualization for embedded applications. As always, I'm very proud of our development tools engineering team. A special shout-out to our teams in Canton and Vannes for their efforts in delivering this release.

Last week, my colleague - Bill "The Truth" Graham - blogged about the importance of development tools in reducing the inherent complexity of the task at hand. This latest release - actually, two product releases - is special because we have extended our hypervisor support to support Intel Xeon processors AND we have extended our development tools capabilities to now include "hypervisor-aware" debugging with our JTAG debuggers.

Why do I think this is important?

My mind goes back to a conversation with one of our early adopters of our hypervisor technology. The folks we met with were responsible for developing and stabilizing the software platform. The hypervisor technology is a critical component of their project as they are looking to consolidate multiple processors onto a single multicore device. They depend on our technology for memory protection to ensure the reliability of their system. One of the clearest memories I have from that day was when they asked "how do I debug my system running on the hypervisor"? The question was not asked out of curiousity - in their situation having assurances that we could address their needs in this area was critical for success.

So, what have done to help Kill Bugs when using virtualization?

The latest release of our Wind River On-Chip Debugging solutions enables debugging of a system running on the hypervisor without the need to add a target resident agent (see Figure 1). This new capability helps developers with tools to develop and debug hardware initialization code, virtual board bring-up, kernels, and drivers. When developing software for virtualized systems, these solutions give developers the confidence to know that they can set breakpoint, watch memory, or analyze stack frames.

Figure 1 - Overview of Hypervisor-aware debugging using Wind River On-Chip Debugging [click to enlarge]

This capability borrows from the concepts applied when debugging Linux processes using a JTAG debugger, but extends them to include the concepts of virtual boards running on the hypervisor. One of the special attributes of our solution is that we go beyond the MMU support provided by the processor to provide visibility and control over all of the virtual boards running on the system - not just the instance that is currently executing.

There's much more that can be discussed on this topic. More information and videos can be found here and there on our site. More importantly, we would like to hear your views on how to Kill Bugs when developing virtualized systems.

Stark Bloggin' Mad!

Wow!!! OK, so how is it that a few innocent tweets about the iPad gets me embroiled in a little Twitter / blogosphere skirmish with a colleague? Who would have thunk it? It's not just Schaefer, almost everyone seems to have formulated an opinion or a prediction about the future of mobile computing. Following last week's announcement of the Apple iPad there has been an impressive flood of blogs, microblogs, and YouTube from pundits and pranksters, alike.  I don't recall a technology preview that had as much publicly generated hype and excitement as the iPad announcement.

So how is it that the company that is focused on bringing us the best possible personal computing experience has generated so much emotion and excitement? iSchmad? Please...

I think that much of the excitement has to deal with the fact that many of us are not fully satisfied with the current state of mobile computing experience and we were looking to Apple - or anyone else - to show us a better way. I have to admit that when I first saw the iPad launch on Wednesday, I wasn't blown away. A few days later, I'm still not sure I completely get it, but I think it is a Good Thing for the industry for a couple of reasons:

1. The tablet form factor is well suited for rendering digital media. There have been several attempts over the years to tabletize the PC. The iPad is taking a different approach that just might work. Chip vendors, hardware vendors, software stacks - commercial and open source - will all optimize their offerings to follow Apple's lead and participate in the new market created by the iPad. I see this as a Good Thing. Within the next 18 months, we can expect both product choice and downward cost pressures which will make the technology broadly accessible and help bridge the digital divide.
2. Digital living is here. Wind River participates in the market of device vendors that are trying to connect all the devices in our homes - allowing you to get access to internet content anywhere. In his blog, Schaefer makes allusions to yours truly being an "Apple Fanboy". Yes, I have some of their gear and I like it very much because it works well and delivers on a great usage experience for listening to music, watching videos, sharing pictures, and surfing the web. And, oh yeah... the Apple stuff looks great, too. If the iPad is successful - and I predict it will be - then we will all benefit from the new standard for content sharing and device connectivity that will be set by Apple's innovation and the momentum it has built from the iPhone and iTunes.

Fascinating.

What exactly are you testing, and how?

"What exactly are you testing? How are you testing it?"

I'm sure that these are amongst the questions that product, business, and technical managers across the embedded software industry have asked themselves on several occasions throughout their careers. I know I have.

The importance of testing embedded software isn't new. In my "youth", I remember our VP of Product Development circulating a white paper that discussed the devastating impact resulting from the deployment of an untested "simple" patch to some switching software and declaring that "this cannot happen to us". I also recall reading and feeling sad about a software bug in a medical radiation instrument that when combined with operator error resulted in lethal doses of radiation being administered to patients.

There is no debate that software quality assurance is serious business. But how seriously are we really taking it, particularly in non-safety critical software applications? Survey data continues to tell us that we're spending upwards of 25% of our project budgets testing and fixing our software, yet we're still not satisfied that our testing practices are effective. Hmmm...

Transparency and visibility into what is being tested is a concern. One of my colleagues blogged about this not too long ago. Confidence that you've covered all your usage different scenarios is another big concern. I can tell you that when I run software that has "encountered an unexpected exception" and I lose work, I get irate and think about other software options.
[editor's note: I thought about including a link to a PC vs. Mac commercial here, but thought against it. Sort of...]

I have invested in partnering with Integrated Processing Limited (IPL) to provide an extension to Wind River Workbench that provides developers with a professional unit testing solution for C/C++ development. This week we are announcing and launching IPL Cantata++ version 5.3.2. Cantata++ provides software developers with unit testing capabilities for Wind River's run-times - including VxWorks and Wind River Linux. Cantata++ 5.3.2 introduces a cool new capability called "robustness testing" that is useful for automating the generation test cases - useful for reducing the likelihood of those "unexpected exceptions".

We are pretty jazzed about this product release. Several of the enhancements made should make it easier to integrate Cantata++ into your environment - including extending Cantata++ to support proprietary and 3rd party embedded operating systems. I anticipate that as we move further into the multicore era - the percentage of multi-OS systems will continue to increase as our customers look to consolidate the number of processors in their systems. One of our goals at Wind River is to provide our clients with a choice of development tools to help with these usage scenarios.

To learn more about our unit testing solutions, please come and check out our webinar on The Benefits of Effective Unit Testing for Embedded Development this Thursday November 5th. There still some great seats available. If we don't see you there, please feel free to "start the conversation" right here.

Simulation? Virtualization? What's in a name?

Check out this great panel round-table conversation about Virtualization & Simulation featuring a great set of panelists from Virtutech, Freescale, Cadence, GreenSocs, and our own Wind River CTO. The discussion touches on a number of important topics, including:

• the differences between virtualization and simulation
• the level of abstractions in models and their importance in predicting a system's behavior
• methodologies driving vendors solutions for virtualization and simulation
• programmable hardware and vanilla (i.e. standard) platforms

While the panelists do not agree on all points, I had a few great takeaways from the discussion:

1. Platform simulation is important to software engineers for managing complexity and risk when analyzing software performance - beyond MIPS and footprint. This is particularly the case when migrating existing software to multicore and dealing with race conditions resulting from parallelization of existing algorithms.
2. Multiple abstract models can be leveraged to ensure application integrity and timing, even before silicon is available. This involves abstract models ranging from cycle accurate instruction simulators to fast OS simulators. 
3. Software development can be fast-tracked using one or more abstract models of the hardware.

Proprietary and commercial simulation solutions have been around for many years. The need to shorten hardware-software integration time and achieve the goal of having working software when silicon/hardware arrives is a good reason to give virtual systems development solutions a good look.

 

2009 Autumnal Equinox

September 21st marks the last full day of the memorable summer of 2009, a summer that was memorable on several fronts - including personal and professional. My summer of 2009 includes memories of rainfalls of near biblical proportions, personal triumphs and losses, the passing of several social icons, and the evolution of the embedded software market.

There is some small debate as to when summer officially ends and when we must stop wearing white to work. If the sun could talk, I would guess that it would suggest that summer officially ends and fall begins at the moment of the autumnal equinox. This is the moment in the September when the sun is vertical to the equator. This year's autumnal equinox will occur on September 22nd at 17:18 ET (21:18 UTC) - just about five hours and eight minutes after the opening keynote at the 2009 Intel Developer Forum being held in San Francisco, California. This is also about nineteen and half hours before the day 2 keynote titled "Developing for the Continuum of Intel Platforms".

The team has been working intensely to prepare for IDF and the Day 2 keynote. We have contributed a short segment that we call "A Day in the Life of an Embedded Multicore Developer... in under 5 minutes". This portion of the keynote is intended to convey how Wind River's development tools and run-times work together cohesively to simplify dealing with the challenges of adapting existing uniprocessor code for multicore. The demo will showcase the Tilcon Graphics suite, Wind River Workbench, and a preview of soon to be available JTAG debugging solutions for Core 2 Duo and Xeon processors. (This is an extension of the Atom configuration of the solution that was made generally available this past July) I am very appreciative of the effort the team put in to create the demo systems.

These types of demonstrations are going to be important as we continue to evolve in the multicore era. A few months ago, I participated in a multicore development tools panel with Dave Stewart of Critical Blue. Dave made a comment to the effect that embedded developers "don't know what they don't know about multicore development" [editor's note: I am paraphrasing] and stressed the importance of education on the different technologies and techniques involved with multicore software. This comment resonates with me and I feel that it puts onus on vendors to support our "techno-speak" with demonstrable proof-points.

If you have an area of interest as it relates to our development tools support for multicore, drop me a comment and I'll put together a posting or video that helps to show the solution in action.

[You can follow Emeka's feed on Twitter - http://twitter.com/enwafor ]

What's the 4-1-1 on 3.1.1?

We have just announced that we are taking orders for Wind River Workbench On-Chip Debugging version 3.1.1. Here's some infomation (i.e. "the 4-1-1") on this release. Version 3.1.1 is a significant update to the software and firmware that powers our JTAG debugger units - the Wind River ICE 2 and the Wind River Probe. A couple of the highlights include our support for RMI Corporation's XLR and XLS processor lines, a new capability that strengthens our support for multicore and multithreaded processors. This component of the release was the result of some hard work and great teaming between the folks at RMI, our professional services organization, and our engineering team in Canton. The other thing that has our team excited is that version 3.1.1 introduces support for Intel Architecture starting with support for the Intel Atom product family. Intel Architecture support in our on-chip debugging solutions extends our existing coverage for ARM, MIPS, and PowerPC. These different architectures are supported using firmware updates to the same physical hardware - this is something that our clients have told me that they appreciate since the same debug solution can be used across multiple projects in there heterogeneous environments. I tend to feel a certain amount of pride everytime we execute a new release, but I'm particularly proud and nostalgic about this release. I'm proud because our clients are jazzed that we are expanding our optimized-for-multicore JTAG debugging solution to the Intel Architecture and are pleased that we have a non-intrusive debug solution for Intel that helps to abstract the debugging of operating systems and the stuff running on them - including operating systems like VxWorks and Wind River Linux.

So why the feelings of nostalgia? It is because the 80x86 instruction set was the first architecture I worked with.

Coming out of university, I started my professional and embedded software career with a company in Montreal that developed point-to-point TDMA-based wireless networks. The company had been primarily a hardware company, but because this "software thing" seemed important, they were ramping up their software engineering team. When I was brought in, I was asked to study if-and-how C++ could be used in this environment. We were also asked to investigate techniques to improve the quality of our software design processes by using formal requirments management techniques, OO-techniques, and state machines to improve the quality of our design activities.

In those first few weeks of my employment, my mornings would consist of researching new software engineering techniques and my afternoons were spent in the lab learning how to debug the distributed system. At this time, most of the system was running on an Intel 80C186 processor and the software consisted mostly of undocumented assembly language that had been written by a former employee who had left to pursue other interests because he had become independently wealthy. (This was the polite explanation that was given to me)

The afternoons were - let's just say - "difficult". My mentor "V" was a brilliant engineer, but not much of a talker. I would later describe "V" as a man who was put on earth with a limited number of words that he could speak for his entire life, and he was using each and every one of these words sparingly. So my afternoons would be spent watching "V" debug the system from a UNIX terminal that was connected to one or more HP 64700 emulators attached to either a central station or a terminal station. "V" was also a six-sigma UNIX blackbelt and a smoker. So, we would sit together at this terminal - with smoke billowing all around the screen - and I would watch as "V" would type a flurry of shell and emulator commands that would grab code from SCCS, compile it in the background, load it into the emulator, and then display assembly code in the debugger window. I would struggle to keep pace as "V" entered a flurry of commands - vi, grep, sed, make, get, sccsdiff, "yy", "set trigger on read/write 0F0B83E00H", "set trigger on I/O output to 040H"  - while navigating all over the filesystem. It was intense - so intense that it didn't feel appropriate to ask questions while all this activity was going on.

I was totally and completely lost.

Usually after an hour or so of analyzing the 80x86 assembly code and traces, "V" would speak saying something like:

"Ah-ha! Can't you see it?! The blah-blah hasn't received a message from the blah-blah, and the connection memory updates are not getting to the blah-blah. This is why we don't have dial-tone."

There would then be another flurry of typng activity culminating in a "make" command. And then he would walk away. I usually used this time to review the history of the commands "V" had typed in the shell. In hindsight, it really felt like trying to understand the hieroglyphics left behind my another civilization.

This went on for weeks (it seemed longer) and then due to some scheduling screw-up, all of the senior software engineers who knew the system were on vacation at the same time. At that time our engineering director - who looked like a Hellenic version of the Malboro Man - was a real "getting things done" kind of guy. There was an ASIC for our next generation terminal station that had to get done and they needed some software guys to test it out. The director walked into the pit where the new engineers were researching software engineering techniques and I remember the discussion going something like this:

Malboro Man: "Sonny, what are you doing for a living?"

Me: "I'm reading about C++ for our next generation software development efforts..."

Malboro Man: "Sonny, does this look like an [expletive deleted, gerund form] library?! 'M' needs to debug the ASIC in the lab before he can go on vacation. Go help him."

We walked to the lab together and I'll never forget the look on "M's" face when he saw that some new grad was between the debugging of his $100K ASIC and a much needed vacation with his family. The next week and a bit was challenging, but proved to be rewarding. We connected the HP 64700 emulator to the new board.This activity in itself was noteworthy just based on the size and cost of these emulators (see figure below). As a new grad, I always felt nervous about dropping one of these emulators that cost about the same amount as my annual salary at the time. Because they were so big, they were almost always mounted in some dangerous configuration.

With the emulator mounted to the board with the new ASIC, I sat down in front of the UNIX terminal, brought out my notebook of commands that I captured, and together with "M" we somehow (read: miraculously) verified the ASIC and "M" went on vacation with his family. For me the bigger accomplishment was learning from the experience of debugging the system at the HW-SW interface level. I quickly developed an appreciation for the visibility that on-chip debugging provides into the system. This knowledge of the system helped  me in later phases of my career as we designed both the hardware and the software for our next generation systems. Reading the specs is one thing, but using an on-chip debugger to gain insights into the execution of the software proved to be essential capability required to help verify and optimize our software.

This year's TechInsights Embedded Market Study by TechInsights confirms that "Testing and Debugging" activities continue to take up the most time in our embedded software development products. Embedded developers need quality debugging tools to help address these challenges and meet their schedules. What's also encouraging is that when asked the questions "Which of the following are your favorite/most important software/hardware tools", "debuggers" rated just a hair behind "compilers" and "JTAG/BDM" debuggers finished in the top-five. These on-chip debugging solutions are essential for embedded and device software development.

This summer's release brings this on-chip debugging capability to an avalance of [new uniprocessor and multicore processors from Freescale, Intel, Broadcom, and Texas Instruments] and puts Wind River in a great position to help engineers tackle their debugging challenges so that they can enjoy more of their summers.

E

P.S. If you have an on-chip debugging story that you want to share, I would be glad to hear it.

[You can follow me on Twitter - enwafor]

When will my Smartphone be as smart as a fifth grader?

Mobile mania is upon us.

In recent days, we've been introduced to the awesome looking Palm Pre. The Pre is Palm's attempt to share some of the spotlight with Apple's innovative and popular iPhone. Apple has had a big role to play in feeding the Mobile mania frenzy and it too was in the news this week. I have to acknowledge that I count myself amongst the multitudes that watched and waited in anticipation of The Big Announcement at this week's Apple Worldwide Developer's Conference where the Apple iPhone 3.0 was introduced. Since its first release, I have always admired the iPhone and its cousin the iPod Touch. I see the iPhone 3.0 at its pricepoint as being similar to a standing-up double in baseball; a great hit that puts them in scoring position, but it didn't bring the runner home. I was hoping for something different - more on this later in this post...

It seems like everyday I have some kind of conversation about the smartphones that are now close to the center of our lives. On the day of our big announcement, one of my Starbucks acquantances - Sherif - pulled me aside to tell me that as much as he loved his iPhone, he really wanted an iPhone that had a better lens on it and that can take better pictures. The other day one of our illustrious senior engineers (I have to say that since he develops stuff for my product division) was showing off his new HTC phone and rubbing his hands together in gleeful anticipation of writing an application for this platform. Today, I came across this great article by Steve Johnson that provides a view into how Intel is developing new Intel Atom microprocessors to power smartphones and mobile internet devices.

There are a number of elements that need to come together in harmony for our smartphones to meet our needs, including:

• low power devices that allow these devices we depend on to work with us through our waking hours
• rich platforms that address our needs around the convergence of the gagets that we use to capture and view pictures, capture and view video, play music, control our A/V equipment, or control the lighting in our homes
• killer applications that make our lives better, or at least more interesting

As a software guy, these are exciting times. Much of the "smarts" in these smartphones come from the software that powers the platforms that power these sleek devices. Without the software, our smartphone isn't so smart - they are really mostly just gadgets.

I often feel that many of the apps that we run on our handsets today are "clever" and more "superficially skillful" than "smart". There are clearly some great apps to help us manage our time, apps that help us communicate through Facebook and Twitter, data gathering apps, and stuff that could help me program my TV when I'm away (not yet available in Canada). There's also a bunch of apps that are great "party tricks" - I think of the cool little app for the iPhone/Touch that allows you to blow on the touch screen and puts mist on your picture or the one that reads MRI's.

But how smart is this software really? We have metrics that we can use to measure the popularity of an application - indicators like the number of downloads, application ratings, and the buzz in the forums help here. But have we really found a metric or an indicator that we can use to assess the quality of these applications with respect to their effectiveness in making our lives better? It might be out there and if it is, please let me know - I want to learn more about it.

I started thinking about the approaches be taken to developing applications for smartphones after seeing a recent tweet from David Pogue:

Yep, me too - I just want to TYPE IN "8:45".  The spin wheels is how it works on my Apple iPod Touch. I tried it out on my Blackberry - it too had a variation on the spin wheels. Both implementations seem to be patterned after the real alarm clock gadgets with those "modal >> | <<" buttons. I know these buttons well and get frustrated with them a couple of times a week when I have to adjust my wakeup time. You would think that a phone, especially a smartphone, would always require no more than 7 - maybe 8 - gestures to set ANY time. Wouldn't it be cool if I could just issue a voice command telling the phone to wake me up at 8:45am? And oh yeah, a Really Smartphone would also think to reprogram the coffee machine on its own - without me having to remind it.

I am nitpicking on the alarm clock since it is an application that most of us can relate to. The thing that excites me about the future of smartphones and Mobile Internet Devices are the possibilities that they create once some unconventional thinking and some hardwork are applied to developing capable and easy to use software on mobile device platforms. Once this is combined with a focus on simplicity of use, we'll be well on our way through this phase of the Technological Revolution. 

So here's what I was hoping for...

A slightly larger iPod Touch with a bigger screen and built-in stereo speakers that could sit on my night table with an easy programming interface to wake me up to one of my favorite tunes from my library. Maybe I could even jot down notes on it when an idea came into my head in the middle of the night (read: a tablet). When the device realized I was awake, it would then play all the tweets and e-mail that came through while I was sleeping.

Maybe for Christmas?

E

[You can follow me on Twitter - "enwafor"]

printf("Hello World!!! \n");

I have been putting off launching my corporate blog for what seems like months. So today, it is with great excitement I say "hello" to the extended Wind River community! The issue of trying to chose what to write about in my first post has been a bigger challenge than I expected. I chose the "printf" title for two reasons:

• it seemed like a cute and natural way to test if my blog account was set-up correctly;
• the continued popularity of printf statements for debugging embedded systems, and the motivations behing this popularity, is a topic worth touching on.

Using printf statements to debug software has been around forever and will probably never disappear completely. A quick google survey uncovered the colourful quote from ariels that says "I'll give up printf() when you pry my cold dead fingers from it". Ariels posting also highlights some of the challenges leading embedded software developers to resort to using printf for debugging.

When I talk with some of our clients and our sales team, the thing that often surprises me is not that developers use printf debugging, but the extent to which these developers are dependant on printf debugging - especially when other alternatives are available and qualify. I think back to my own experiences early in my embedded software development career and I wonder if a developers preference for printf debugging has less to with technical factors and more to do with preconceptions about using real debuggers to improve our effectiveness in developing embedded software.

If preconceptions have a role to play in how embedded software is developed, I believe that we are now in a period of change where these preconceptions will be challenged and refined. I also believe that the migration to multicore devices will be a driving factor for software engineers to revisit the way that things get done. Earlier this week, David Stewart from Critical Blue (and not of Eurythmics fame) reminded me that "software developers did not ask for multicore". Very true. Software developers have had the benefit of riding the power-performance curve to get more CPU power to handle new features and capabilities. Multicore is a result of us having hit the frequency ceiling combined with the pressure of embedded applications that consume less power and dissipate little heat. As a result the concerns of multicore - parallelism, concurrency, partitioning, and virtualization - have now becoming pervasive in the discussion with embedded software developers.

I think this is an exciting time in our industry and I look forward to a open discussion around an approach to development tools and education that will lead to an improved effectiveness in developing device software in the multicore era.

E

[You can follow me on Twitter - "enwafor"]