Tools, Testing, & Virtual Systems

Debug, multicore, and more debug

By Jakob Engblom

I recently gave a talk at an industry-academia collaboration called ICES, Innovative Center for Embedded Systems, at KTH in Kista, Stockholm, Sweden. The theme was embedded multicore, and I realized that my role at these events seems to have changed. A few years ago, I would be the "embedded guy", defending the collective of embedded systems against speakers assuming that everything was a homogeneous shared-memory multiprocessor. This time was different, though. I have become the "debug fanatic."

There seems to be less need to explain embedded multicore nowadays. The academic community and multicore event participants seem to have finally accepted that there are systems out there that use more than one operating systems on a single chip (thanks to hypervisors, AMP setups, local-memory configurations, etc) and that we have hardware that is highly heterogeneous (with different makes and different types of cores all being part of a single system). In the final panel discussion, I found myself saying "build the hardware anyway you like, as long as I have good hardware debug support and introspection features".

I think this is indicative of a general truth: if you cannot debug it, you cannot build it. For multicore in particular, debug is scrambling to catch up.

One joker in the audience made the age-old claim that you only need a debugger if you code errors. True enough. There are classes of code that can be written to be correct by construction and always work. I have written such code myself. For example, I wrote a memory allocation system for a simple OS that we created in a course project during my CS undergraduate days. I wrote the algorithm on paper and tested it on paper using various scenarios (brain-powered simulation, essentially). Once I was happy with the code, I went to the computer, typed it in, and it compiled and ran without a hitch for the rest of the project (module a single typing mistake). If only all problems were as easy as that.

Such nice code and nicely-defined problems tend to be the exception. In the real world today, your code will have to integrate with other people's code, as well as operating-systems and library APIs. In such circumstances, too much is out of your own control to allow correct-by-construction coding. I am not saying that you should be sloppy - but you have to be aware that there are factors beyond your control and insight that will affect the overall correctness of the combined system. There will be bugs in code that calls your code, and in code that your code calls. There will be timing-dependent and ordering-dependent errors as the operating system schedules threads on different cores in different orders.

In this world, debugging is a necessity, and we need to support in all ways we can. I prefer using virtual platforms like Simics for debugging as far as possible, since they provide a controlled and repeatable environment for most of the debugging of your project. When you get to the hardware, most bugs should already have been removed using other tools.

Note that there is no replacement for testing (and probably doing a bit of debug) on the actual physical hardware that will be used to run the software. No matter how you develop and debug your code, in the end, it has to run on a particular physical system. On that system, you really want the most powerful on-chip on-target debug tools you can find.

For more on Simics and debugging, see some of my previous blog posts:

• Repeatability and variability in Simics
• Bug transportation using Simics
• Concurrency and bugs

 

 

Intuitive & Efficient Debug of Multi-core Systems

By Brian Finkel

I reflect on a recent visit with my 84-year old Aunt Honey. As you might expect, she is not an avid cell phone user. She wanted help entering contacts into her phone. She has a basic phone so we had to enter text using the “9-up” keypad. Her objective is to make phone calls by selecting someone’s name instead of typing the telephone number.   

The event had me thinking about the way we used to make telephone calls – using an address book or a phone list and then typing in the number. In my youth, things weren’t so bad – everyone lived in the same town – with the same area code and only a few prefixes. For each friend you only had to remember the 4 digits and perhaps which prefix to use. Fast forward a few years and with the advent of cell phones, IP telephony, new area codes, and geographically-dispersed friends and family – you now need to remember 10 digits. 10 digits is beyond most people’s working memory (which is typically 7 digits). Address books and phone lists were required. 

[International Readers: The North America phone number is 10 digits. 3-digit Area Code + 3-digit Prefix + 4 digit Number]

Enter the cell phone. And the ability to program phone numbers and contact names into the phone’s memory. That enabled me to Dial-by-Name. Dial-by-Name is intuitive -- my Aunt Honey is a person not a 10-digit number. Dial-by-Name is efficient – no more rummaging through the desk drawer for the phone list, no more typing a wrong number, no need to memorize 10 digits. That saves me time! Dial-by-Name is intuitive and efficient compared to the old way.

Before I compare Dial-by-Name to what we do at Wind River, I’d like to take a minute to give kudos to the mobile phone developers for creating this seemingly simple user experience that we now take for granted.

Look at your cell phone. Under the covers things aren’t so simple. Your phone is a complex wonder of miniaturization enabled by numerous technology advances: low-power CPUs and communications chips, small-form factor batteries, cost-effective LCD displays, and thousands of lines of code ranging from device firmware to applications. There are dozens of components that had to be developed, integrated, and finely tuned:  the phone dialer application, the contact list application, the CPU, the baseband communications chip, the keypad, speaker, LCD display, headphones, Bluetooth, and more – all working together in harmony. It took 100’s of engineers, project managers, and quality assurance professionals (and a little luck) to deliver Dial-by-Name on your phone. So congratulations to the mobile phone developers out there for making our lives simpler!

So why does this all matter to you, to me, and to Wind River?

I assume if you are reading this blog you are probably closely tied to the embedded developer community. No doubt, you and your colleagues are using or considering multi-core SoCs.  These devices are becoming increasingly complex – integrated peripherals, co-processors, 3 levels of cache, and so-on. And to realize the benefits of these devices, you are considering multiple operating systems running on the same CPU potentially in a SMP (Symmetric Multi-Processing) or virtualized system configuration. Your world is complicated.  As an example, if you are designing network infrastructure gear, you are probably consolidating data plane and control plane functions onto a single board, blade, or SoC running a combination of Linux, an RTOS, and/or a low-level executive, partitioned across multiple cores or processors.   

As an embedded developer, how do you take control of a complex system like this? How do you debug and resolve issues that are prone to arise in this complex multi-core system environment?

As tools developers at Wind River, we strive to deliver solutions that are intuitive and efficient. 

Here are some of the challenges we think about on a daily basis:

• How do you simultaneously visualize and interact with the resources for 8 cores in a single IDE? 
• How do you provide visibility to multiple operating systems each running its own set of processes, threads, and applications?   
• How do you debug guest operating systems running on a hypervisor?
• How do you provide a flexible GUI so that different types of users – hardware engineers, device driver developers, application developers, system integrators – can view and control system elements important to the task at hand?
• What is the best way to represent an embedded system on a 1280 x 1024 display using graphical elements such as colors, fonts, tabular views, scroll bars, icons, expandable lists, progress indicators, windows, and more?   

In order to deliver innovation, we embrace a customer-oriented approach in developing our tools.  Through our local technical support organizations, we reach out to you to understand the problems you are trying to solve. We implement your usage models and desired “workflows” within our development and quality assurance processes. This approach is yielding innovation. 

A recent example is the SMP Debugging capability delivered with our On-Chip Debugging (JTAG) solution. When using Workbench 3.2 and our Wind River ICE 2 JTAG debugger, you can logically represent an OS running on multiple SoC cores as a single system. This enables an intuitive debug experience -- operations such as breakpoint management, kernel visibility, and run-control are managed at the system-level. Similar to how Dial-by-Name abstracts the 10-digit phone number, we’ve abstracted the details on each of the cores. It’s intuitive because it’s the way you think about your system. 

We look forward to hearing your feedback on how we can make your life easier through an intuitive and efficient development tools experience.     

Happy developing!

Brian Finkel is a Senior Product Line Manager for Wind River Development Tools and On-Chip Debugging Solutions. One of Brian’s current initiatives is to deliver a world-class debug experience for developers implementing complex multi-core and multi-OS systems.   Brian has 15-years experience in product management, technical marketing, program management, and business development roles at Wind River, Marvell Semiconductor, Intel, and IBM. Brian holds a Bachelor of Science Degree in Electrical Engineering from Rutgers University.

Virtual Trace

By Jakob Engblom

I spent most of last week at the S4D conference in Southampton, the UK. A small gathering of people interested in debug, it seems to be a good indicator of trends in debug. The dominant trends this year were clearly tracing and instrumentation. From the perspective of a virtual platform, tracing is a natural task, and it can be done with much less pain than on a physical platform.

Tracing on a piece of hardware requires that the processor or SoC at the heart of the system has an interface that can communicate a trace to an external trace box, and also that the chip has built-in tracing in the hardware. The trace units we have today are much more powerful than those of just a few year's ago, but trace is still not universal, not able to capture everything, and limited by the bandwidth of the few pins you can pull out of a target for the purpose of debug. The result is that you always end up tracing only certain aspects of the execution or only certain subcomponents of an SoC. In particular data trace is not always supported, and tracing the behavior and accesses to peripheral devices is usually hard.

In a virtual platform, tracing has unlimited bandwidth since it goes straight from the simulator to the host. As the time inside the virtual platform is virtual, it does not matter if tracing slows down the execution - the time seen by the virtual machine is the same regardless, and the tracing is invisible to the simulated system. You can trace any part of the target system, including the rare luxury of complete data access traces undisturbed by caches and buses hidden deep inside an SoCs.

Here are some tracing examples done with Simics, just to give a flavor what can be done. We start with a simple instructions + data trace (from a point about 2 seconds into the boot of VxWorks 6.8 on a Freescale P2020):

simics> trace0.start 
Tracing enabled. Writing text output to standard output.
simics> c 10
inst: [  51] CPU 0 <v:0x0100075c> <p:0x00100075c> 3b7b0002 addi r27,r27,2
inst: [  52] CPU 0 <v:0x01000760> <p:0x001000760> b3df0000 sth r30,0(r31)
data: [  467] CPU 0 <v:0xffef5eca> <p:0x0ffef5eca> Write 2 bytes 0xffff
inst: [  53] CPU 0 <v:0x01000764> <p:0x001000764> 3860000a li r3,10
inst: [  54] CPU 0 <v:0x01000768> <p:0x001000768> 4bfffe79 bl 0x10005e0
inst: [  55] CPU 0 <v:0x010005e0> <p:0x0010005e0> 9421ffd0 stwu r1,-48(r1)
data: [  468] CPU 0 <v:0x01316470> <p:0x001316470> Write 4 bytes 0x13164a0
inst: [  56] CPU 0 <v:0x010005e4> <p:0x0010005e4> 7c0802a6 mflr r0
inst: [  57] CPU 0 <v:0x010005e8> <p:0x0010005e8> bf61001c stmw r27,28(r1)
data: [  469] CPU 0 <v:0x0131648c> <p:0x00131648c> Write 4 bytes 0x132db1c
data: [  470] CPU 0 <v:0x01316490> <p:0x001316490> Write 4 bytes 0xaaaa
data: [  471] CPU 0 <v:0x01316494> <p:0x001316494> Write 4 bytes 0xf8000000
data: [  472] CPU 0 <v:0x01316498> <p:0x001316498> Write 4 bytes 0xffff
data: [  473] CPU 0 <v:0x0131649c> <p:0x00131649c> Write 4 bytes 0xffef5eca
inst: [  58] CPU 0 <v:0x010005ec> <p:0x0010005ec> 7c7f1b79 mr. r31,r3
inst: [  59] CPU 0 <v:0x010005f0> <p:0x0010005f0> 90010034 stw r0,52(r1)
data: [  474] CPU 0 <v:0x013164a4> <p:0x0013164a4> Write 4 bytes 0x100076c
inst: [  60] CPU 0 <v:0x010005f4> <p:0x0010005f4> 3ba00000 li r29,0

 

Simics can also trace accesses to devices in a system (and many many other things). Here is an example of keeping an eye on accesses to the main serial port in the P2020 system. The data reported include the processor accessing the device, as well as address accessed and the exact cycle when the access hits.

[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157a70 3064639465} 
 Read from p2020ds.soc.cpu[0]: PA 0x4504 SZ 1 0x8 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157bf0 3064639565} 
 Write from p2020ds.soc.cpu[0]: PA 0x4503 SZ 1 0x83 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157bf0 3064639582} 
 Write from p2020ds.soc.cpu[0]: PA 0x4500 SZ 1 0x58 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157bf0 3064639602} 
 Write from p2020ds.soc.cpu[0]: PA 0x4501 SZ 1 0x14 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157bf0 3064639622} 
 Write from p2020ds.soc.cpu[0]: PA 0x4503 SZ 1 0x3 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157bf0 3064639688} 
 Write from p2020ds.soc.cpu[0]: PA 0x4501 SZ 1 0x0 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157bf0 3064639713} 
 Write from p2020ds.soc.cpu[0]: PA 0x4503 SZ 1 0x3 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157bf0 3064639733} 
 Write from p2020ds.soc.cpu[0]: PA 0x4504 SZ 1 0x8 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157bf0 3064639753} 
 Write from p2020ds.soc.cpu[0]: PA 0x4502 SZ 1 0xc7 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157a70 3064639775} 
 Read from p2020ds.soc.cpu[0]: PA 0x4502 SZ 1 0xc1 (BE)
[p2020ds.soc.uart[0] trace-io] {p2020ds.soc.cpu[0] 0x157bf0 3064639801} 
 Write from p2020ds.soc.cpu[0]: PA 0x4502 SZ 1 0xf (BE)

Yet another example of tracing using Simics is the execution time line we have in the Simics Analyzer product. This traces the context switches of an operating system, providing a nice overview of the system execution. I wrote a blog post about Analyzer back in April, if you want to know more about that aspect.

The other theme was instrumentation. To get a grip on the execution of software stacks involving operating systems, hypervisors, middleware, custom devices drivers, and complex user-level programs, it is hard to beat inserting little snippets of instrumentation code into the software source code which report "now I am at place X in state Y". When the program is run and the instrumentation output traced, this results in a trace of events at a level very understandable to the programmers. It is "printf debugging" with less overhead and more sophistication. In theory, such a trace could be reconstructed from an instruction trace and using debug information, but in practice, inserting little application-specific tracers into the code gets us much more application-level information with much less programmer pain. If the instrumentation code is left inside a shipping system, it does not even amount to altering the nature of the programs - we are testing the same code as we ship! It is obviously not something for tight real-time code, but in many cases the time-savings in debug and development worth the slight execution overhead.

With a virtual platform, you can do interesting things with source-code instrumentation. Rather than pushing the instrumentation data out over a network or storing it in in-memory buffers, it can be sent to a special trace device that immediately saves the data onto the host disk (or send it across a network).

Interestingly for me, such an instrumentation device is the only hardware model written by me that is still shipping with Simics. The "simple-byte-dump" device maps a single byte in memory and copies anything written there into a file on the host. We used this with an aerospace customer to report results from on-target tests, without having to involve any serial or network drivers. Very simple, very effective, and a good example of what you can do with a virtual platform which is hard to do in hardware. If you have Simics, it is in there, and you can check it out yourself!

Kill Bugs: Volume 1

By Emeka Nwafor

The Bride: [Japanese] I need Japanese steel. 

Hattori Hanzo: [Japanese] Why do you need Japanese steel? 

The Bride: [Japanese] I have vermin to kill. 

Hattori Hanzo: [English] You must have big rats if you need Hattori Hanzo's steel. 

The Bride: [English] ... Huge. 

(from Kill Bill: Vol. 1)

I will probably need the help of a therapist to assist me in decoding why I think of that passage from Tarantino's great movie whenever I think of our portfolio of multicore development tools. When I ponder the personality of our development tools (i.e. the fearless Hattori Hanzo and his sword) and the risks presented to our customers in getting their multicore software to be correct (i.e. vermin - in reference to the challenges and bugs, not our customers) the exchange works for me. 

I digress.

We just announced a new release that extends our capabilities for multicore and virtualization for embedded applications. As always, I'm very proud of our development tools engineering team. A special shout-out to our teams in Canton and Vannes for their efforts in delivering this release.

Last week, my colleague - Bill "The Truth" Graham - blogged about the importance of development tools in reducing the inherent complexity of the task at hand. This latest release - actually, two product releases - is special because we have extended our hypervisor support to support Intel Xeon processors AND we have extended our development tools capabilities to now include "hypervisor-aware" debugging with our JTAG debuggers.

Why do I think this is important?

My mind goes back to a conversation with one of our early adopters of our hypervisor technology. The folks we met with were responsible for developing and stabilizing the software platform. The hypervisor technology is a critical component of their project as they are looking to consolidate multiple processors onto a single multicore device. They depend on our technology for memory protection to ensure the reliability of their system. One of the clearest memories I have from that day was when they asked "how do I debug my system running on the hypervisor"? The question was not asked out of curiousity - in their situation having assurances that we could address their needs in this area was critical for success.

So, what have done to help Kill Bugs when using virtualization?

The latest release of our Wind River On-Chip Debugging solutions enables debugging of a system running on the hypervisor without the need to add a target resident agent (see Figure 1). This new capability helps developers with tools to develop and debug hardware initialization code, virtual board bring-up, kernels, and drivers. When developing software for virtualized systems, these solutions give developers the confidence to know that they can set breakpoint, watch memory, or analyze stack frames.

Figure 1 - Overview of Hypervisor-aware debugging using Wind River On-Chip Debugging [click to enlarge]

This capability borrows from the concepts applied when debugging Linux processes using a JTAG debugger, but extends them to include the concepts of virtual boards running on the hypervisor. One of the special attributes of our solution is that we go beyond the MMU support provided by the processor to provide visibility and control over all of the virtual boards running on the system - not just the instance that is currently executing.

There's much more that can be discussed on this topic. More information and videos can be found here and there on our site. More importantly, we would like to hear your views on how to Kill Bugs when developing virtualized systems.

What's the 4-1-1 on 3.1.1?

By Emeka Nwafor

We have just announced that we are taking orders for Wind River Workbench On-Chip Debugging version 3.1.1. Here's some infomation (i.e. "the 4-1-1") on this release. Version 3.1.1 is a significant update to the software and firmware that powers our JTAG debugger units - the Wind River ICE 2 and the Wind River Probe. A couple of the highlights include our support for RMI Corporation's XLR and XLS processor lines, a new capability that strengthens our support for multicore and multithreaded processors. This component of the release was the result of some hard work and great teaming between the folks at RMI, our professional services organization, and our engineering team in Canton. The other thing that has our team excited is that version 3.1.1 introduces support for Intel Architecture starting with support for the Intel Atom product family. Intel Architecture support in our on-chip debugging solutions extends our existing coverage for ARM, MIPS, and PowerPC. These different architectures are supported using firmware updates to the same physical hardware - this is something that our clients have told me that they appreciate since the same debug solution can be used across multiple projects in there heterogeneous environments. I tend to feel a certain amount of pride everytime we execute a new release, but I'm particularly proud and nostalgic about this release. I'm proud because our clients are jazzed that we are expanding our optimized-for-multicore JTAG debugging solution to the Intel Architecture and are pleased that we have a non-intrusive debug solution for Intel that helps to abstract the debugging of operating systems and the stuff running on them - including operating systems like VxWorks and Wind River Linux.

So why the feelings of nostalgia? It is because the 80x86 instruction set was the first architecture I worked with.

Coming out of university, I started my professional and embedded software career with a company in Montreal that developed point-to-point TDMA-based wireless networks. The company had been primarily a hardware company, but because this "software thing" seemed important, they were ramping up their software engineering team. When I was brought in, I was asked to study if-and-how C++ could be used in this environment. We were also asked to investigate techniques to improve the quality of our software design processes by using formal requirments management techniques, OO-techniques, and state machines to improve the quality of our design activities.

In those first few weeks of my employment, my mornings would consist of researching new software engineering techniques and my afternoons were spent in the lab learning how to debug the distributed system. At this time, most of the system was running on an Intel 80C186 processor and the software consisted mostly of undocumented assembly language that had been written by a former employee who had left to pursue other interests because he had become independently wealthy. (This was the polite explanation that was given to me)

The afternoons were - let's just say - "difficult". My mentor "V" was a brilliant engineer, but not much of a talker. I would later describe "V" as a man who was put on earth with a limited number of words that he could speak for his entire life, and he was using each and every one of these words sparingly. So my afternoons would be spent watching "V" debug the system from a UNIX terminal that was connected to one or more HP 64700 emulators attached to either a central station or a terminal station. "V" was also a six-sigma UNIX blackbelt and a smoker. So, we would sit together at this terminal - with smoke billowing all around the screen - and I would watch as "V" would type a flurry of shell and emulator commands that would grab code from SCCS, compile it in the background, load it into the emulator, and then display assembly code in the debugger window. I would struggle to keep pace as "V" entered a flurry of commands - vi, grep, sed, make, get, sccsdiff, "yy", "set trigger on read/write 0F0B83E00H", "set trigger on I/O output to 040H"  - while navigating all over the filesystem. It was intense - so intense that it didn't feel appropriate to ask questions while all this activity was going on.

I was totally and completely lost.

Usually after an hour or so of analyzing the 80x86 assembly code and traces, "V" would speak saying something like:

"Ah-ha! Can't you see it?! The blah-blah hasn't received a message from the blah-blah, and the connection memory updates are not getting to the blah-blah. This is why we don't have dial-tone."

There would then be another flurry of typng activity culminating in a "make" command. And then he would walk away. I usually used this time to review the history of the commands "V" had typed in the shell. In hindsight, it really felt like trying to understand the hieroglyphics left behind my another civilization.

This went on for weeks (it seemed longer) and then due to some scheduling screw-up, all of the senior software engineers who knew the system were on vacation at the same time. At that time our engineering director - who looked like a Hellenic version of the Malboro Man - was a real "getting things done" kind of guy. There was an ASIC for our next generation terminal station that had to get done and they needed some software guys to test it out. The director walked into the pit where the new engineers were researching software engineering techniques and I remember the discussion going something like this:

Malboro Man: "Sonny, what are you doing for a living?"

Me: "I'm reading about C++ for our next generation software development efforts..."

Malboro Man: "Sonny, does this look like an [expletive deleted, gerund form] library?! 'M' needs to debug the ASIC in the lab before he can go on vacation. Go help him."

We walked to the lab together and I'll never forget the look on "M's" face when he saw that some new grad was between the debugging of his $100K ASIC and a much needed vacation with his family. The next week and a bit was challenging, but proved to be rewarding. We connected the HP 64700 emulator to the new board.This activity in itself was noteworthy just based on the size and cost of these emulators (see figure below). As a new grad, I always felt nervous about dropping one of these emulators that cost about the same amount as my annual salary at the time. Because they were so big, they were almost always mounted in some dangerous configuration.

With the emulator mounted to the board with the new ASIC, I sat down in front of the UNIX terminal, brought out my notebook of commands that I captured, and together with "M" we somehow (read: miraculously) verified the ASIC and "M" went on vacation with his family. For me the bigger accomplishment was learning from the experience of debugging the system at the HW-SW interface level. I quickly developed an appreciation for the visibility that on-chip debugging provides into the system. This knowledge of the system helped  me in later phases of my career as we designed both the hardware and the software for our next generation systems. Reading the specs is one thing, but using an on-chip debugger to gain insights into the execution of the software proved to be essential capability required to help verify and optimize our software.

This year's TechInsights Embedded Market Study by TechInsights confirms that "Testing and Debugging" activities continue to take up the most time in our embedded software development products. Embedded developers need quality debugging tools to help address these challenges and meet their schedules. What's also encouraging is that when asked the questions "Which of the following are your favorite/most important software/hardware tools", "debuggers" rated just a hair behind "compilers" and "JTAG/BDM" debuggers finished in the top-five. These on-chip debugging solutions are essential for embedded and device software development.

This summer's release brings this on-chip debugging capability to an avalance of [new uniprocessor and multicore processors from Freescale, Intel, Broadcom, and Texas Instruments] and puts Wind River in a great position to help engineers tackle their debugging challenges so that they can enjoy more of their summers.

E

P.S. If you have an on-chip debugging story that you want to share, I would be glad to hear it.

[You can follow me on Twitter - enwafor]