Mike Deliman

Quit Bugging Me: Making Maps

A tool commonly used in embedded debugging is a linker map - a map of where all the symbols are in the runtime image.  These maps are useful as they turn raw addresses reported by some exception stubs (etc) into offsets into the data or text (program routines) in the computer's RAM.  They give you an idea of what may have been happening when the error occurred.

Producing a linker map is fairly easy.  Most linkers include command line options to produce a map.  This works fine and is very clear when used from a command line.  But things can get a little confusing from within an integrated gui environment.

The Workbench environment makes it very easy to create and use projects, and glue them together.  A problem that can occur, is since you build all projects by selecting "build project", you may not be sure of what tools are being used by the builder "under the hood".

A recent question we had was "How do I produce a linker map?  I provided the following flags to get the linker to make a map:  -Xlinker -map mapfile.  Instead of getting my map file, I got an error."  The error the customer was getting was "ldppc: cannot find -linker".

Thinking about the problem a bit I came to the realization that the customer was calling ldppc with the flags that tell ccppc "hand these options to the linker".  The problem is that the linker, ldppc, called with these flags directly, decided they meant something else.  The real problem is a misunderstanding that the project being modified uses the linker directly, instead of the compiler front-end.  I decided it would help to demonstrate.

To the compiler, "-Xlinker blah blah" tells the compiler "call the linker with the flags blah blah".

-Xlinker  means something completely different to the linker.  -X tells the linker "discard symbols".  -l<something> tells the linker "link in the library named <something>".  The kind of funny thing is that using -Xlinker, the linker was looking for a library named "inker", libinker.a .  Since the directive failed, the linker sort-of seemed to be complaining that it couldn't find itself!  Here's a demonstration of it: note that to complete the demonstration I needed to add a directive "look here for libraries" (-L.)

For the demonstration, I launched the Developer's Environment:

Q:\WIND\VX67WB31>ldppc --help | grep -i map
  -M, --print-map             Print map file on standard output
  -Map FILE                   Write a map file
Q:\WIND\VX67WB31>ldppc --help | grep X
  ...
  -X, --discard-locals        Discard temporary local symbols (default)
  ...
Q:\WIND\VX67WB31>touch libinker.a

Q:\WIND\VX67WB31>ldppc -L. -Xlinker
ldppc: warning: cannot find entry symbol _start; not setting start address

Q:\WIND\VX67WB31>rm libinker.a

Q:\WIND\VX67WB31>ldppc -L. -Xlinker
ldppc: cannot find -linker

Quit Bugging Me: Another Surprise NaN!

An Earlier QBM, "Surprise NAN" covered how floating point computations may become corrupted from unexpected sources.  Today's Quit Bugging Me is about... another Surprise! NAN!

An application runs with several tasks.  One task, which computes a set of floating point values, periodically comes up with bad values, and sometimes the name becomes corrupted.  

Here is a short example...  done with the solaris simulator.  Kernel options do -not- include any hardening, nor any guard pages / stack protection.  In fact, only the options related to "MMU_BASIC" are included.  I have a breakpoint set just before the exit so I can catch overflows.  surpriseNAN recurses on it's argument, easily overflowing the default stack size.  I call logMsg first thing, it's a cheap way to get the task ID printed out.  I'm going to provide edited kernel shell output to show the points.

Symptoms:  the task names are valid on entry but suddenly go away, there's a crash that involves "reschedule", the values contained in the FPregs are... corrupted badly:

-> task spawned: id = 0x40b4820, name = t2

task spawned: id = 0x40bb968, name = t3

task spawned: id = 0x40ad6d8, name = t4

task spawned: id = 0x40a6590, name = t5

task spawned: id = 0x409f448, name = t6

task spawned: id = 0x4098300, name = t7

0x40bb968 (t3): alive.

0x40ad6d8 (t4): alive.

0x40a6590 (t5): alive.

sometime later the BP is hit.  Several of the tasks are "missing", though none completed.

> i

  NAME      ENTRY     TID    PRI   STATUS      PC       SP     ERRNO  DELAY

-------- ---------- -------- --- ---------- -------- -------- ------- -----

tExcTask   excTask   40d8250   0 PEND          a89cc  40d8058       0     0

tLogTask   logTask   40d37b8   0 PEND          a89cc  40d35c0       0     0

tWdbTask   5eea4     40ce308   3 PEND          664c8  40cdfd8       0     0

!       surpriseNAN  40bb968 100 READY         6a80c  40b4cb8       0     0

!       surpriseNAN  40ad6d8 100 READY         6a80c  40a6a28       0     0

       surpriseNAN   40a6590 100 SUSPEND       6a810  409f950       0     0

value = 0 = 0x0

Notice - the task t5 hit the breakpoint and was suspended.  The other two tasks are "ready" and never hit the break point.  They are no longer running though.  And what happened to the names?  Let's look at one of the tasks...

-> ti 0x40bb968

NAME       ENTRY      TID    PRI   STATUS      PC       SP     ERRNO  DELAY

-------- ----------- ------- --- ---------- -------- -------- ------- -----

        surpriseNAN  40bb968 100 READY         6a80c  40b4cb8       0     0

stack: base 0x40bb968  end 0x40b4c08  size 27856  high 27856  margin 0    

options: 0x1d

VX_SUPERVISOR_MODE  VX_DEALLOC_STACK    VX_FP_TASK          VX_STDIO            

%pc    =    6a80c   %npc   =    6a810   %psr   =        0   %mask  =        0

%tbr   =        0   %y     =        0                                        

%g0    =        0   %g1    =       5f   %g2    =        0   %g3    =  40ad6d8

%g4    =        0   %g5    =        0   %g6    =        0   %g7    =        0

%i0    =        1   %i1    =    db800   %i2    =        0   %i3    =        0

%i4    =        0   %i5    =        0   %fp    =  40b4d30   %i7    =  40d9d60

%l0    =  40bb968   %l1    =        0   %l2    =        0   %l3    =        0

%l4    =        0   %l5    =        0   %l6    =        0   %l7    =        0

%o0    =        0   %o1    =    db800   %o2    =        0   %o3    =        0

%o4    =        0   %o5    =        0   %sp    =  40b4cb8   %o7    =    6a804

%fsr   = eeeeeeee
%f00   = 0.793335   %f02   =        4   %f04   =        8   %f06   =        8
%f08   =  7.87933   %f10   = 0.00819755   %f12   = 0.223435   %f14   = -0.668483
%f16   = -0.902314   %f18   =        0   %f20   =        0   %f22   =      NaN
%f24   =      NaN   %f26   =      NaN   %f28   =      NaN   %f30   =      NaN
value = 67819224 = 0x40ad6d8

What happened here?  The FP-REGS are trashed?  and the name is trashed... something must have corrupted the system.  Is it a stack over-run?

-> checkStack

  NAME        ENTRY        TID     SIZE   CUR  HIGH  MARGIN

------------ ------------ -------- ----- ----- ----- ------

tExcTask     excTask      40d8250  15984   504  3016  12968 

tLogTask     logTask      40d37b8  15984   504  2104  13880 

tWdbTask     0x000005eea4 40ce308  16040   816 11376   4664 

!           surpriseNAN  40bb968  27856 27824 27856      0 OVERFLOW

!           surpriseNAN  40ad6d8  27856 27824 27856      0 OVERFLOW

            surpriseNAN  40a6590  27856 27712 27856      0 OVERFLOW

So in fact there is not only one stack overflow, but 3.  What has happened here?

Looking back at the TaskIDs returned from taskSpawn, we can see these were all spawned with their OS data structures right in a row.  With no memory protection (no memory protections included!) the task stacks are not protected.

When the surpriseNAN recursively writes over the top of it's own stack, it corrupts the area where the floating point data are saved.  Some of the values written work-out to be valid floating point numbers, others are not numbers.  This is near the area where the task name is stored.  With a sufficient overflow, the task name is corrupted as well as the FP data.

When the task overflows it's stack far enough, it will eventually over-write other data.  In this case the other data is the register set of the task immediately adjacent (in lower addresses) in RAM.  

Something to notice in the corrupted FP-REGS - the FSR is set to a stack-fill value (so the FSR screams "eeeeeeee!").   This is one more tell-tale that the context for this task has been corrupted.

Quit Bugging Me: Revalidated

One day, I get this phone call.  "We're working with the system, we see the calls that update the exception handlers early on - connecting the clock routines, etc.  Then not very much farther on we see the system has run past where tasking should be running but it's not.  When we check, we're not getting any clock interrupts, and the clock exception handler isn't there any more.  How could the system do that?  Is there any way the OS could remove it's clock connections?"

The complete story was even more eyebrow raising.  "We're testing for a launch, this vehicle is a one-of-a-kind, there's no backups.  We're doing environmental limits testing.  We're still within spec, but operating at the low end of the electrical ratings.  We need to explain this anomalous behavior or scrub."

This was the start of a frantic week of work.  On PowerPC, the architecture stores it's exception vectors down in low ram.  A valid method of updating these vectors is to write new data to the vector address as data, then force the data cache to be pushed to the RAM, and finally tell the processor "any instructions you have cached for this area are invalid" (invalidate the I-cache).  This last step ensures that the CPU will fetch the newly-updated instructions the very next time they're called.

The hardware involved was fairly rare and expensive. We didn't have a lot of options available for hardware debug assist.  In fact, I had never even seen the hardware in person, all my work was theoretical and experiments were done with "similar" boards.

I used what I had to trace the execution sequence to the point in question, where we should have been seeing clock ticks come through. With all the hardware I had, there was no way I could recreate the problem.  In every case, I had clock ticks come through as expected, and with code to set the clock to an expected value earlier on, every single run showed the same exact results.  It was impossible for it to /not/ work.

The sequence, boiled down, that was under investigation, runs like this:

The runtime starts.  Vectors are written-to / populated with default values and routines.  Caches are invalidated and enabled. Board hardware is initialized.  A clock is set up, it's exception vector is populated via a data-write; the D-cache is "flushed" to RAM, the instruction cache is told "this area of RAM is updated" (invalidated), and we continue running.  1/60th of a second later we should get a clock tick, it should be processed through this exception vector and ....

The problem seemed to be that the exception vector was running the same old code - not the new code.  It could be checked that the new code had actually been pushed all the way into RAM - except we didn't have that capability at the time nor enough time to set it up.

In every other test setup, we saw the expected behavior - the clock tick delivered and tasking running exactly as expected.  In this particular setup, it was as if the clock tick never happened.  The CPU side of things clearly ran as if the code had never, ever been invalidated.

As a point of interest, we found the code executed was smaller than the instruction cache.  We'd never exhausted the cache.  We formed a theory, hard to prove, but interesting.  Running at the lower voltage end, the units under test were all the -same-, hardware wise.  With the lower voltage, was it possible that some anomaly of the hardware caused the instructions that had been invalidated to... somehow.. not be invalid?  The only way we could still be executing those instructions is if there were still in the I-cache, and we'd invalidated them.  Perhaps the hardware somehow revalidated the cache when run at low voltages?

Bringing power back up to spec showed normal execution.  Working with the hardware vendor we learned that there was a probability that the lab grade hardware might show anomalies not exhibited by deployment grade hardware, and indeed the lab grade hardware would fail tests that the deployment grade hardware passed without problems.

Testing was repeated on deployment grade hardware, which ran without incident.

Looking back, had I known then what I know now, it would have been expeditious to execute a field of no-ops the size of the I-cache immediately after installing the timer vector.  This would have caused the entire I-cache to be invalidated/populated with new entries, flushing the timer vector code completely.  The next execution of that vector would have forced the I-cache to re-read the whole vector, we would have seen the clock ticks where expected, and we could have explicitly tested the I-cache in this way.

Quit Bugging Me: Driving Me Backwards

You're working on a highly precise instrument.  The kind of instrument that relies on extremely accurate measurements of time.   Work is proceeding normally.  Suddenly.. events come in in a backwards order, time jumps around - but only for a few ticks.

The heart of the system - in this case - is a hyper-accurate clock.  The clock's accuracy can be verified by external sources.  The clock itself is not only part of the computer, but also part of the scientific apparatus.  It is crucial that the timing marks in the data are as accurate as possible, or the data becomes worthless.

Everything seems to work perfectly.  All the calibration is done, data sets are processing exactly as intended, then suddenly....  time goes backwards.  For months testing has been going on, tests haven't found problems, data all looks good, but on the longest test runs, once in a long while, the clock seems to skip a beat... and move backwards.

An unlikely test is suggested: turn up the clock rate, either in actuality or virtually.  It's easy to say "The OS must have dropped the ball somewhere, perhaps delayed processing of some critical bit of information at just the wrong time.  The timer driver runs for days at a time with no errors, how can this find a problem with the system?"

In cases like this it's almost always best to work backwards from the symptoms.  The main symptom is that once in a long while, regardless of what the rest of the system is doing, time drifts a couple of ticks backwards all at once.

The best way to debug a situation like this is to start with what was happening when the system took it's trip into the past.  Software normally isn't sentimental and doesn't care what happened in ancient history a fraction of a second ago.  Look for patterns.  In this particular case, the most common pattern in the failed tests was that the system had been running for many weeks without failure, when it suddenly took a step backward in time.  What happened before those brief steps back in time was varied, there are no obvious patterns exhibited.

Since in each case when the problem occurred, the system was in a different state, the most obvious commonality between the incidents was: the system had been running for a "longer" test.  This means re-creating the test may be a very long term endeavor.  What do you do in these cases?  (Problems like this used to be difficult to debug before there were system visualization tools like the System Viewer.)

One thing you can do is speed time up.  If your system is not processor limited, increase the system tick rate and increase data rates on other tests you can run simultaneously.   Since it takes a long time for the problem to be exhibited, compressing time may help debug it.

The other thing to do is review the code involved.  Start small in scope - what things were running at the time the problem occurred.  Use the assumption that the code has not become corrupted. Places to look for culprits when an otherwise stable driver starts going awry: look at everything that could affect that driver, especially itself.

I've actually seen this problem - and others similar to it in nature - more than once.  In the cases I've seen it's mostly been race conditions - either "mistakes" made in the code sequence or timing problems imposed by hardware that the coder needs to be aware of to work around.

This is a bug known as a Race Condition.   A critical point is not properly protected, an irq re-enabled too early, hardware access re-enabled too soon, something along these lines.  The general feeling is that of a figure-eight race track.  As long as the drivers are spaced just right, the cars go speeding through the cross-roads without colliding, but it only takes the smallest bit of miss-timing to cause a small collision and precipitate a gigantic crash.

The timer driver had a miss-sequence.  The global tick count was incremented immediately after the interrupts were re-enabled.  Once every trillion ticks or so the clock tick comes through and right behind it a second tick comes through.  With the IRQ-re-enabled, a second tick completes first, then a first tick, and the tick value is seen to "go backwards" by one tick.  This makes time seem to jump forward, backward, and forward again - two forward, one back, and continue normally.  Looking at the disassembled code, there was only a window of perhaps 5 instructions where a second interrupt could have possibly triggered the bug, which was why the system could run for months without exhibiting the problem.

The first time I saw a hardware induced race-condition was on what I think of now as a vintage VME computer.  It was a 68010 based board with a serial chip (and a bunch of discrete electronics!).  A programming example for "implementing an adjustable buffer feature" (heh.. turning on the 4-char buffer) included a number of "nops" between writes to the serial chip.  We had a customer try to return the hardware once, because when they removed all the "nops" the serial driver stopped working ("they don't do anything, right? we need it to run fast!").  We explained that the no-ops actually created a delay just long enough for the serial chip to "become ready again".  Removing these delay-instructions actually induced a race condition where the CPU (serial driver) was too fast for the serial chip.  (there were only perhaps a dozen or so no-ops.)  In this case, the serial chip itself needed some time to "settle-out" after changing modes in the chip.  Accessing it too soon gave unpredictable results.

Quit Bugging Me: ABI

What's in an ABI?  An Application Binary Interface is a lot like a API - Applications Programming Interface, except instead of just telling you how to make a call, what parameters to provide, and what returns and errnos to expect, an ABI also tells you something about how the interfaces work.  Understanding an ABI for a CPU means you understand how the registers are used, what they're for, and what they mean under different contexts.

Among the things ABIs give you is how your CPU formats memory (the stack) when it makes a call to a routine. An easy way to take a look at an ABI is to set a break point on a routine, and look at what the stack contains.  This is on a PowerPC target.  First I'll spawn a task, then extract the current information about the task - stack pointer, stack base, etc, using target-side debug routines.

The PowerPC ABI gives specifications for how and when routines use the stack to store values.  I'll use some knowledge of the ABI and values from the target-side debug routines to show you a simple ABI trick.

I've spawned:  printf("this is a test. %d %d %d %d %d %d \n", 1, 2, 3, 4, 5, 6) .

I've set break a break point on "fioFormatV" for this demonstration, so I can show you how printf was called.  When printf() calls fioFormatV() it has to store some of it's work on the stack.  It does this so fioFormatV() can do work without destroying the data printf is supposed to handle.  Let's look at some task information and the contents of the stack.  I've removed some output we're not concerned with for this example.

-> Break at 0x00053068: fioFormatV          Task: 0xfff9c20 (t1)

-> ti

  NAME         ENTRY       TID    PRI   STATUS      PC       SP     ERRNO  DELAY
----------  ------------ -------- --- ---------- -------- -------- ------- -----
t1          printf        fff9c20 100 STOP          53068  fff9b80       0     0

task stack: base 0xfff9c20   end 0xfff4e00   size 20000  high 160    margin 19840

r0         = 0x0002cae4   sp         = 0x0fff9b80   r2         = 0x00000000
r3         = 0x0fff4dbc   r4         = 0x0fff9ba8   r5         = 0x000545a4
r6         = 0x00000001   r7         = 0x00000004   r8         = 0x00000005
r9         = 0x00000000   r10        = 0x00000000   r11        = 0x0fff9b88
r12        = 0x0000000d   r13        = 0x00000000   r14        = 0x00000000
r15        = 0x00000000   r16        = 0x00000000   r17        = 0x00000000
r18        = 0x00000000   r19        = 0x00000000   r20        = 0x00000000
r21        = 0x00000000   r22        = 0x00000000   r23        = 0x00000000
r24        = 0x00000000   r25        = 0x00000000   r26        = 0x00000000
r27        = 0x00000000   r28        = 0x00000000   r29        = 0x00000000
r30        = 0x00054474   r31        = 0x00000000   msr        = 0x02029230
lr         = 0x000544d8   ctr        = 0x00054474   pc         = 0x00053068

-> d 0xfff9b80
0x0fff9b80:  0fff9bc0 eeeeeeee 0fff4dbc 00000001  *..........M.....*
0x0fff9b90:  00000002 00000003 00000004 00000005  *................*
0x0fff9ba0:  00000006 00000000 01000dee 0fff9bc8  *................*
0x0fff9bb0:  0fff9b88 eeeeeeee eeeeeeee eeeeeeee  *................*
0x0fff9bc0:  0fff9be0 0002cae4 00000000 00000000  *................*
0x0fff9bd0:  eeeeeeee eeeeeeee 00000000 00000000  *................*
0x0fff9be0:  00000000 00000000 eeeeeeee eeeeeeee  *................*
0x0fff9bf0:  eeeeeeee eeeeeeee 0fff4dbc 00000001  *..........M.....*
0x0fff9c00:  00000002 00000003 00000004 00000005  *................*
0x0fff9c10:  00000006 00000000 00000000 00000000  *................*
0x0fff9c20:  00000000 0fff9c20

-> tt t1
0x0002cae4 vxTaskEntry  +0x5c : printf (0xfff4dbc, 0x1)
0x000544d8 printf       +0x64 : fioFormatV ()

Okay... looking at the task trace, we see printf was called with some values.  The first value doesn't look like our string, it looks like an address.  This should be the address where our string is... that's what the ABI says:

-> d 0xfff4dbc
NOTE: memory values are displayed in hexadecimal.
0x0fff4db0:                             74686973  *            this*
0x0fff4dc0:  20697320 61207465 73742e20 20256420  * is a test.  %d *
0x0fff4dd0:  25642025 64202564 20256420 25640a00  *%d %d %d %d %d..*

Yes... that's the format string for printf.  From the task trace there's only one additional argument. But looking at values stored on the stack - starting at the address in the "stack pointer" address from the "ti" task information we see some things we recognize... all in a "row" there is the address of the string above, and the numbers 1 through 6 - this is the arguments passed in to printf.  At the very first address, there's an odd value - 0fff9bc0 - this address is between the stack pointer and stack base.  It's called a back-chain: it points back to the previous stack frame.  Following back-chains back, fff9bc0 leads to 0fff9be0, which contains 0.  But.. we're still not back to the base of the stack yet.  What gives?  This area - between the "0" valued back-chain and the base of the stack, is the initial stack frame which is used to preserve the arguments originally handed to taskSpawn().  The first stack frame is the part of the stack used by a routine referred to as "vxTaskEntry" in the "task trace" output.  This previous part is where taskSpawn() holds the original values for taskRestart().

From the PowerPC ABI, we can see that when one routine calls another, standard stack usage is to preserve the data the caller is going to need later on, by pushing it onto the stack.  printf is calling fioFormatV, printf is responsible for saving some registers, so it puts them on the stack (look up "caller-saves" and "callee-saves"). It's easy enough for us to identify the arguments we handed to taskSpawn for printf to use on the stack.

Looking back at the top stack frame - between fff9b80 and fff9bc0, we can change these values and affect what gets printed.  It should print "1 2 3 4 5 6". Let's change the 6 to a 9.  After changing the value, we'll "c" continue the execution and see what it prints.  In that top stack frame, "00000006" is stored at 0xfff9ba0, so we'll modify memory there:

-> m 0xfff9ba0
0x0fff9ba0:  0x0000-
0x0fff9ba2:  0x0006-0009
0x0fff9ba4:  0x0000-.

value = 0 = 0x0
-> c
value = 0 = 0x0
-> this is a test.  1 2 3 4 5 9

By changing the value printf() had saved on the stack according to the ABI, we changed the value printf actually printed out.

By using the ABI and the contents of memory, you can see if something has happened to corrupt the arguments given to a routine you're interested in testing.  You can also use this kind of method to force erroneous values into a routine for testing purposes.  Given that the ABI specifies a routine will store values to the stack when it calls other routines, you may need to find the first call-out performed by the routine you are interested in (as I did above using fioFormatV, which is called by printf).

Quit Bugging Me: Induction

Working on a customer problem once, we had an interesting phenomena.  Upgrading a system with a large VME cage and several boards, the customer replaced older processor boards with what were then "new" boards.  The old boards ran at (I think) 33 MHz, the new ones at more like 133MHz.

The overall system included motor control functions and sensor feedback, there were D-to-A, A-to-D, and other boards in the system.  The problem was, with a straight-and-simple rebuild of software for the new boards, the system was giving anomalous readings.  Checking the registers for values from the A-to-D boards, there were numbers showing up that were not possible.  Readings indicated the motors were on and moving their load when in fact the motors were not connected.  Switching back to the old computer boards eliminated the problem.  We added some hardware to examine VME bus activity, to see if the new CPUs were reading the bus wrong.  We had to put the test board between the CPUs and the sampling boards.

The sampling boards (A-to-D) were in the chassis right next to the CPUs.  They had a sampling rate not far off from the new CPU's clock rates.  Moving the sampling boards farther down the chassis reduced the severity of the anomalous data reads, but did not eliminate the problem.  The bus analiser showed the data on the VME bus was consistent with the data reported by the new CPUs.  This provided the final clues needed to pinpoint the problem and fix the system.

The new boards have clocks that run at a rate close to the sampling rate of the A-to-D boards.  The clocks were physically next to the A-to-D  boards in the VME chassis.  For whatever reason, the new boards oscillators were radiating enough energy at the right frequency to corrupt samples taken by the A-to-D boards.  By moving the boards farther apart we reduced the effect (inverse square law) but not by enough to eliminate it.  To test this theory we placed steel plates in a VME slot, as a separator between the rest of the boards in the bus and the A-to-D board.  At this point, both old and new CPUs read the same consistent readings from the A-to-D boards.  Removing the metal plates brought back anomalous readings with the new CPUs.

In this case it seemed like a new system of software and boards was improperly accessing a device.  Switching back to the old baseline eliminated all the new problems.  Analytical equipment verified the readings reported by the CPU were consistent with the data retrieved from the sampling hardware, so it wasn't a problem "within" the new software or hardware. The real problem was a completely unexpected interaction between legacy hardware and upgrade hardware.

Serial Intent

Howdy out there,

I realize this is a blog that's supposed to be about real-time programming issues, and mostly I've posted about planetary and space based projects, with a few announcements about technology and news items.  Though these combine subjects near and dear to my heart (space, and VxWorks), these were mostly "interest stories", not solid real-time issues.  (It must be kind of like tuning into a financial news station and seeing stories about some new exotic fluffy breed of dog.)

In line with the Real-Time issues theme, I'm going to try to post more often about debugging systems and common issues I've seen over the past 20+ years of working with VxWorks.  This is not because of feedback from the readership, it's something I just decided needs to be done.  *smile* please don't be confused by the subject lines...

Starting tomorrow, I'll try to post at least one story a month under the title: Quit Bugging Me.

Kind regards,
  mike