All XL220 controllers

Enhancements

SCN 5388: XL2; Debug code added to help troubleshoot Loss of Memory.

We have a customer complaining that several of their controllers are resetting and powering up with loss of memory, reporting that the configuration has changed.

The most likely cause is that a power or software glitch is causing the reset. Something in the software is apparently corrupting settings or Global Back data, which requires a memory clear.

To help determine what software is losing control, we added a switch-on diagnostic that arms the 386's hardware breakpoints on the backup copy of the setup table. If anything writes to it unexpectedly, the controller records a task error with a code (0x000F0002) and a stack trace pinpointing the offending code — the tool to catch, in the act, whatever is corrupting configuration on the affected customer machines.

In addition, every powerup, the controller reports to the debug console which copies of the setup and global data passed their integrity check. If a "configuration changed" ever happens again, this tells us exactly which data went bad (setup table vs. order/global data) so a targeted fix can follow. No effect on machine operation — it's reporting only.

If the Global Data backup is being corrupted, a new error message will be displayed to distinguish it from the Setup Data. Knowing which block of data that is being corrupted will help us target that memory with more debug resources, if necessary.

SCN 5389: XL2; Transition to 12-bit encoder counter chip hardware.

The 5387 board, since V4, has used two 32-bit quadrature encoder counter chips to support the four incremental encoder ports. These parts are obsolete and we have no source for them any longer.

We have found a source for the 12-bit version of these chips. It is apparently a custom build of the 12-bit version that is housed in the 32-bit package. This allows us to use the chip without a board layout change. The only issue is that we have to modify our software to use the 12-bit version. We have to read the encoder often enough so that we never see more than 2048 counts between reads or our math will not work properly to calculate the change between reads.

Through empirical testing we have determined the Max Encoder rate to be 2MHz. If we read the encoder every 100usec, that calculates out to a maximum number of counts equal to 200 per read, well below the 2048 limit.

There will be a new limitation introduced. Any 5387 board with the new 12-bit chip will no longer be compatible with older software versions. The cutoff will be V4.110.0 and V5.110.0. Any software older than this will be incompatible with 5387 boards containing the new chip. Software versions newer than these will work with either version of the chip.

On the old chip we have a bogus encoder count read of 10,000 per read. We will be lowering that threshold to 2,000 counts per read.

We have modified the Closed Loop Controller code to read at 100usec, even when there are no Open Loop Targets.

The OL gets similar treatment but with a much simpler change since it was already written to read the encoder more than once per sample.

Bug Fixes

SCN 5369: XL2; PEG library build issue.

The PEG library was not being built with the correct flags to successfully link with the application. It appears as if the project file for the hardware build was not checked in properly some time after the library was built and saved.

There was a Peg_i80386EX_Base.optset file that was not being correctly included in the project configuration.

SCN 5386: XL2; Debug Logger Class bugs.

Fixed three long-standing bugs in the debug console's formatted-print routine so numbers print with the right size and column width. Debug/diagnostic output only — no effect on how the machine runs.

SCN 5387: XL2; Engineering Build (BAYCB) Setup Protection Bug.

On engineering controllers — which have no model/options programmed into the chip and are configured at the bench — the main copy of the setup table was being wiped and rebuilt on every powerup, leaving only the backup copy valid. That left those units one glitch away from a "configuration changed." This change preserves the configured model on those units so both copies stay valid, while still bringing up the model-select screen when a model genuinely needs to be chosen (fresh unit or after a flash/memory clear). Production controllers behave exactly as before.

SCN 5391: XL2; Fixed illegitimate F0002 Task Errors.

Aligned the four 386 debug-register addresses to dword boundaries so the backup-setup write watch no longer false-trips on the adjacent main-copy CRC store, which was boot-looping the OpenLoop build. It may have cause issues on the V4 projects as well.

SCN 5392: XL2; Fix language validation test.

There was a language validation test that made sure the master copy of the language selection was good. However, it also forced the backup copy to be changed as well. This created a situation where the backup copy could not be used, even if it was good. This could cause unnecessary memory clears due to Configuration Changes.

This is was an unlikely, but possible scenario that we wanted to fix, in case it is happening in the field.

SCN 5393: XL2; Fixed potential race conditions with Deletion of Coil records.

While search for causes or reported memory clears on a coil tail out, we discovered some remote possibilities of deleting coil records and leaving dangling pointers behind. Each of these were remote possibilities but potentially possible nonetheless.

When the controller cleans up deleted coil records in the background, there was a small chance it could free a coil that some other part of the system — a screen dialog, or the host-computer/Eclipse browse position — was still pointing at, which can corrupt memory. The cleanup now waits if a pop-up dialog is open, refuses to delete the coil the host is currently looking at, and briefly locks the coil list the same way every other record type already does. This is extra insurance against a hard-to-reproduce field report about coils.

SCN 5396: XL2; "Corruption" of master setup copy.

On every power-up, the routine that reads the machine's model and options from its hardware bit code was erasing more of the model record than it rebuilt — including the model-name text. That made the primary copy of the machine settings look corrupted on each boot, so the controller quietly fell back to its backup copy every time. Survivable alone, but it meant the machine was always running on a single good copy; if power dropped at the wrong instant while the backup was being refreshed, both copies could be lost at once and the controller would report "Configuration Changed" and clear its memory. The fix limits that startup routine to clearing only the fields it actually re-reads, so the primary copy stays valid and the machine always keeps two good copies.

Version 5 only

New Features

SCN 5354: XL2; USB Hub support

USB Hub Support for the XL200 ARM (lets several USB devices share the controller's single USB port)

The XL200's USB port could previously work with only one device at a time. It can now work through a standard USB hub, so several devices can be used at once. This was confirmed on the bench with a network adapter and a flash drive plugged into a hub at the same time - both fully working, while the machine's normal EtherCAT motion communication continued without any disturbance.

Reaching that point required fixing four long-standing defects in the USB software supplied by the chip vendor. None had ever caused trouble before, because no hub had ever been connected to this product, so these code paths had never actually run. The most significant of them quietly consumed an internal resource every time a hub was unplugged; left alone it would eventually have stopped USB working until the controller was power cycled. That path has now been unplugged and reconnected ten times in a row with no sign of degradation.

A separate and more serious problem was found and fixed along the way. The coprocessor sets aside a small workspace for handling hardware interrupts, and the extra activity a hub creates overflowed it - which then corrupted the operating system's own internal data. When that happened the coprocessor detected the fault, logged a task error with its diagnostic details, and restarted itself, which is the designed behavior. What made it hard to trace was that the corruption struck whatever the coprocessor happened to be doing next, so the details recorded pointed at an unrelated part of the system rather than at the real cause. That workspace is now four times larger, in all five controller programs.

One follow-up item came out of this: the way these faults are recorded can be improved so that the logged details identify the true source when the fault happens inside interrupt handling, which would have saved considerable time here.

Groundwork for USB keyboard and mouse support is included but is not yet active. Those devices will be recognized when plugged in, but they do not yet send keystrokes or pointer movement to the display. That work is fully specified and is the next step.

Current limitation: testing used a hub without its own power supply and two devices. How many devices the port can support on its own power has not yet been measured, so a hub with its own power supply is the safer recommendation until that is characterized.

SCN 5368: XL2; USB Keyboard and Mouse Support

USB keyboards and mice now work on the controller, both during startup and in normal operation.

Two paths were needed. Once the machine is running, keyboard and mouse events ride along with the data the ARM coprocessor already sends every millisecond, costing nothing extra when nobody is typing. But the machine can stop during startup and wait for a keypress, before that once-per-millisecond exchange begins — so a second path lets the startup code collect events directly until the normal one takes over, handing off without dropping or duplicating anything.

Three faults found and fixed along the way. Leftover test code on the ARM was consuming events before the main processor could read them, which made the mouse jerky. A startup flag was stored in battery-backed memory and so kept its value from the previous power-up, permanently disabling the startup path after the first boot. And diagnostic printing shared a task with keyboard handling, where it could stall it — that would have affected machines in the field with no PC attached, so it is now off by default.

Existing keyboard behavior is untouched: the ARM produces exactly the same messages the keyboard processor has always produced, so the display software needed no changes.

SCN 5372: XL2; USB Barcode Scanner Support

Barcode scanner support over USB. A USB scanner set to serial (COM) mode is now recognized, and its scans arrive exactly as if they had come from the existing RS-232 scanner port — so all the existing barcode handling, including macros, the character filtering and entry into fields, works unchanged.

Three things had to be enabled or corrected to get there. The USB serial device class was not built into the controller's USB software at all, so the scanner was seen but nothing could claim it. A header the application needed was not being copied out of the USB library. And the buffer that receives scanner characters was only created when no PC was connected to the diagnostic port, so it was missing on any machine with a debugger attached.

SCN 5373: XL2; USB diagnostic screen.

Version 5 controllers now have a USB Devices screen in the Diagnostics menu, below System Info. It shows which USB devices the controller has found, listing the hub, keyboard, mouse, barcode scanner and Ethernet adapter, and for each one the manufacturer and product identification codes. Those codes are what let support confirm that a customer is using the device they believe they are using.

If a USB hub is connected, the screen shows how many ports it has and which of them have something plugged in. If a port is occupied but no device appears in the list, the controller found something it does not support, so "I plugged it in and nothing happened" now has a visible explanation.

The screen also shows whether the barcode scanner has ever sent any characters and how many, which separates a broken scanner from a working scanner with a problem elsewhere. It reports a count of USB errors, and while errors are actually occurring the USB Status field shows the specific error code in place of OK. Finally it shows how much USB memory is free, which is how a slow internal leak would reveal itself.

None of this was visible before. Diagnosing a USB problem meant connecting a laptop to a service port and reading technical output, which is not practical in the field. Just as importantly, the screen was previously capable of reporting everything as normal while USB was failing hundreds of times a second, because it only watched for one specific kind of failure. It now watches for errors of any kind and keeps the count after the problem clears, so an intermittent fault leaves a trace that someone can find later. The screen refreshes itself about once a second while it is open, and is written so that having it open does not slow the machine down.

One bug was fixed along the way. A barcode scanner plugged directly into the controller, with no USB hub in between, would be detected correctly and then have its power cut a few seconds later, over and over, which was the clicking noise heard from the scanner. A feature that recovers poorly seated USB devices had never been told that scanners exist, so it treated a perfectly working scanner as a failed one and kept trying to fix it. Scanners plugged into a hub were unaffected, which is why this went unnoticed for so long. Scanners now work in either arrangement.

Enhancements

SCN 5357: XL2; ARM Task Error parsing improvements.

Improved Fault Diagnosis on the XL200 ARM Coprocessor (accurate task error records, and keeping the information needed to read them later).

When the ARM coprocessor detects a fault it records a task error with diagnostic details and restarts itself, which is the designed behavior. Two problems were making those records misleading, and both are now fixed.

First, if the fault happened while the coprocessor was in the middle of handling a hardware interrupt, the recorded details described an unrelated part of the system instead of the code that actually failed. The record looked perfectly plausible, which is worse than no record at all - it points an investigation in the wrong direction. This was a significant part of why the recent USB hub fault took a full day to track down.

Second, two of the recorded values were stored under each other's labels, so anyone reading a record was told the failure was in one area when the evidence actually pointed somewhere else. That also contributed to the same wasted day.

Fault records now identify the code that actually failed, whether the fault occurred in normal program flow or inside interrupt handling, and they include which interrupt was active at the time. The record's format and size are unchanged, so everything that reads or displays these records continues to work exactly as before. Records logged before this change still have the two values reversed.

The build now also keeps a second piece of information under version control. A fault record identifies the failing code by its numeric address, which by itself means nothing - translating an address into the name of the function requires the "map file" the build produces, and that map is only valid for the exact program image it was built with. Until now only the program image was kept and the map existed solely on the machine that built it, so a fault reported from the field months later could not reliably be traced back to the code that caused it. The map is now saved alongside the program image automatically at build time, and a copy also remains on the build machine for immediate use.

Both fault cases were verified on the bench by deliberately causing each kind of fault and confirming the recorded details matched the known cause.

This is diagnostic accuracy only - it changes nothing about how the product runs, and it does not prevent faults. It makes the next one much faster to diagnose, including well after the software has shipped.

SCN 5359: XL2; Moved to newer version of USB stack for Keyboard and Mouse support.

The ARM coprocessor's USB software was a version the supplier had stopped developing and replaced. We have shipped releases on it and it worked — the USB serial console, telnet, and the USB-to-Ethernet adapter all functioned. The trouble only began when we started hot-plugging devices for the keyboard and mouse work: that exercises parts of the software a device present from power-up never touches, and there the old design does too much work with interrupts disabled. Long enough to interfere with the timing of machine control.

Since the supplier had already replaced that software, and the replacement is built around not doing that, we moved to it rather than patching a version with no future.

The move is now complete and tested on the bench. The USB-to-Ethernet adapter identifies itself, negotiates a network link, takes its address from the 386, and answers pings — the deciding test, since that adapter is how the controller reaches the network in EtherCAT mode, where the built-in Ethernet port is taken over by EtherCAT itself.

Two faults are fixed along the way. Plugging in a USB device could stall the ARM long enough that the 386 stopped receiving responses and reported an error, which in turn stopped EtherCAT and faulted the drive; plugging and unplugging no longer disturbs machine control. Separately, high-speed USB devices could appear to disconnect and reconnect repeatedly on their own.

The diagnostic USB serial console had to be moved to the new software at the same time, because the old and new versions could not coexist in one build. It works as before.

Keyboard and mouse support remains switched off and is the next stage.

SCN 5363: XL2; USB Stack now enumerates USB Keyboard and Mouse

USB keyboards and mice now work on the ARM board, and they can be unplugged and plugged back in while the machine is running. The mouse and Keyboard events still need to be wired into the the PEG task to be used. That will be the next step.

Three problems were fixed:

1. The board could lock up when a mouse was unplugged and reconnected. Some USB devices describe themselves incorrectly on reconnect, and the manufacturer-supplied USB software got stuck in an endless loop reading that bad description. Because that software runs at a high priority, nothing else could run — the diagnostic console went silent even though the board itself was still alive and still talking to the drive. It now recognizes the bad description and stops rather than getting stuck.
2. Keypresses were not being reported at all. Two causes: the board asked the keyboard for more data at once than a keyboard is able to send, and it never told the keyboard to use the simple standard reporting format. Both corrected.
3. After three plug/unplug cycles, nothing further could connect. Internal resources were not being released when a device was removed, so the board gradually ran out of them. They are now released properly.

Keyboard and mouse are tracked separately, so both can be connected at once and either can be removed without disturbing the other. That separation is also the groundwork for barcode scanner support, since scanners connect to the system as keyboards.

Testing: about a dozen unplug/replug cycles with no lockup, followed by confirmed live mouse movement and keystrokes. EtherCAT communication stayed running throughout.

One note for future maintenance: two pieces of the manufacturer's USB code look like straightforward bugs but are actually load-bearing — correcting them introduces real defects. Both are now commented in place so a future cleanup doesn't reintroduce the problem.

SCN 5370: XL2; USB Mouse, rebase to touch screen.

To prevent mouse jumps when switching from using the touchscreen to a USB mouse, the USB mouse coordinates are rebased to the touchscreen position after a touch screen event.

SCN 5375: XL2; More USB diagnostics.

The USB Devices diagnostics screen now reports hubs correctly. Many hubs sold as "7-port" contain two 4-port controllers wired together inside one shell, and the controller reports itself to the software one half at a time. The screen was describing only the first half: on a 7-port hub with four devices connected it showed 4 ports with 2 occupied, and one of those two was the hub's own internal wiring rather than anything a customer had plugged in. It now counts the whole hub and reports 7 sockets with 4 occupied.

The screen shows how many sockets are occupied rather than which ones. Identifying individual sockets turned out not to be possible in a way we can trust. USB tells the controller which internal chip and which internal port a device is on, but nothing tells it how the sockets are numbered on the outside of the case, and different hubs are built differently. A numbering was worked out that matched the hub on the bench exactly, but it would have been guesswork on any other hub, and sending a technician to the wrong socket is worse than giving no socket at all. The count is correct on any hub, and read against the port total it still answers the question that matters: if more sockets are occupied than devices are listed, something is plugged in that the controller did not recognize.

The error count on the screen now shows the error code as well as the total. The code was previously visible only while errors were actively occurring, so after a problem cleared, the screen could report that hundreds of errors had happened without saying what they were.

A USB Ethernet adapter failure was found and fixed. With a hub, keyboard, mouse, barcode scanner and Ethernet adapter all connected, the Ethernet adapter would not work and the network connection never came up. The cause was an internal limit on how many simultaneous USB connections the software could track. The limit was one short of what that combination needs, and the shortfall happened to land on the Ethernet adapter because it was the last device to be set up. The limit has been raised with room to spare, and the software now measures and reports how close it comes to that limit so the margin is a known number rather than an assumption.

All five USB devices now work together, along with the machine's EtherCAT motion network, and devices can be unplugged and moved between sockets without the controller losing track of memory or needing a restart.

SCN 5378: XL2; EtherCAT Indradrive WKC-miss Investigation.

EtherCAT WKC-miss investigation instrumentation (IndraDrive).

Conclusion (2026-08-18): the misses are the drive not publishing a fresh feedback (AT) sample ~0.6% of cycles, in time-varying eras, on both bench IndraDrive units, present since the family's first contact with our master. The master was measured clean at every layer; the ctrlX showed zero misses under matched conditions. Follow-ups tracked separately: Bosch support case, and feedback extrapolation-on-skip for velocity-loop axes.

Instrumentation added (permanent field diagnostics, one stats block):

• Miss profile: category (noframe / untouched / partial), spacing, run length, last-miss timing context; counters reset per session.
• Arrival phase: frame arrival vs the drive's Sync0 from the drive's own DC timestamp, printed at lock and every report (verifies frame placement independently of the sync PI).
• Frame accounting: sent / received / cycle passes (proves one frame per cycle, no double-sends).
• Acyclic mailbox request counters (diagnostics / clear-error).
• ESC hardware error counters (0x0300 block) read per report, printed on change only: wire errors, processing-unit rejects, PDI errors, lost link.
• Config manifest: one line per bring-up with every value written to the drive (mode, telegram, scaling, modulo, cycle time) for comparing runs, builds, and machines.
• SoE procedure-command handshake logging (start/poll/cancel/end). Loop B trim re-enabled (experiment concluded); miss-experiment flag gbLoopBTrimEnable remains, default true.

SCN 5385: XL2; EtherCAT reliability and CPU usage improvements.

EtherCAT reliability and diagnostics improvements. The controller now recovers from a drive communication interruption (for example a disconnected cable) on the first attempt, in about half the previous time. The root cause of the failed recoveries was found in our own software: it was rejecting valid messages from the drive because it assumed the drive numbers its messages the same way we number ours. The official EtherCAT specifications say the two numbering sequences are independent, and the drive was following the rules. The message handling was corrected to match the specification, and the drive error-clearing procedure was rebuilt around the specified completion signal. The controller also no longer wastes processor time waiting for network responses (it is now interrupt driven), procedures that used to be able to wait forever now have time limits, and the serial log now reports lost or repeated drive messages so field problems of this kind can be diagnosed from a log capture instead of a bench investigation.

SCN 5398: XL2; EtherCAT Improvements. Mainly speed but some reliablility.

EtherCAT Startup: DC Clock Synchronization, Drive-Config Reliability, and Diagnostics

Major Improvement to DC Clock Synchronization Time at Startup

Two changes cut worst-case synchronization from tens of seconds to under one second:

• Convergence-based clock stabilization. Startup used to propagate the reference clock with a fixed 10,000 writes (a magic number copied from the reference driver), costing the same few seconds every boot no matter how many drives were present. It now propagates the clock while polling each drive's System Time Difference register and stops as soon as every drive has settled within tolerance and stayed there, auto-scaling to the actual network. A single drive converges in a fraction of the writes and a one-drive setup skips it entirely; the 8-drive ctrlX measured 2,350 writes / 421 ms instead of the old fixed multi-second loop.

• Robust phase-lock detector (leaky accumulator). The master's check for whether it is locked to the drive clock used to require a run of consecutive in-tolerance cycles and reset to zero on any single out-of-tolerance cycle. On a busy multi-drive network, occasional send-jitter spikes kept resetting that run, so lock time grew exponentially: an 8-drive segment could take 25,000+ cycles (about 25 s), within 5 s of the 30 s timeout that aborts startup. Replaced with an accumulator that gains on good cycles and only leaks slightly on bad ones, making lock time scale linearly with jitter instead of exponentially. It now locks in about 250 cycles (about 0.28 s) even on the noisy 8-drive segment, removing a real risk of startup timing out and failing to come up.

Fixed Intermittent Drive-Configuration Failures During Multi-Drive Startup

The EtherCAT mailbox reads and writes that configure each drive were using a 20 ms response timeout (EC_TIMEOUTSAFE, which is actually meant for wireless frame return) instead of the 700 ms mailbox timeout that operation is supposed to use. With eight ctrlX drives coming up together, a drive occasionally took just over 20 ms to answer; the controller gave up early, and the drive's late reply then landed in the next transaction's slot, shifting the whole message stream by one and failing a drive timing-parameter write (S-0-1007). Corrected all 49 mailbox calls (SoE and CoE) to the proper timeout. As a backstop, the shared communication library now recognizes and discards a stale or mismatched reply and re-reads for the correct one, so even a rare slow drive cannot desynchronize the setup sequence. Startup is now clean, error-free, and quicker.

Made the Startup Timing Read-outs Accurate

The clock-sync lock time was being sampled well after the lock actually happened, which made it look far slower than reality; it now captures the true time-to-lock and reports it in both cycles and milliseconds. Jitter and cycle-deviation figures are reset at the moment of lock, so the operating numbers reflect steady running rather than the momentary startup transient. The stabilization step also now reports how long it took.

Production Cleanup and Reduced Startup Noise

Removed a diagnostic read of parameter S-0-0028 (a drive missed-frame counter) that neither supported Bosch drive family implements; it only logged a "no such parameter" error every boot for a value nothing used. Bench-only diagnostic probes were placed behind a compile switch so they do not run in production, and a leftover manual tuning enable was removed.