VxWorks Development – when hardware doesn’t support Ethernet
Since its earliest days, one of the key features of Wind River’s VxWorks real-time operating system has been its support for networking. This applies not only to the run-time system, but also to the development environment. Even before Tornado and Workbench, Wind River’s tools have traditionally placed a strong emphasis on using a high-bandwidth network connection between the host and target systems.
But what happens when the target hardware doesn’t support Ethernet?
System software based on VxWorks is developed in a host-target environment, where the connection between the host and target is usually via an Ethernet interface on the target hardware. Today, Ethernet has become such a widely-used interface that it’s integrated on many of the processors that might run VxWorks; if not, it’s usually provided as part of the board design anyway.
In a traditional VxWorks development cycle, the target boots the kernel in a few seconds using an Ethernet file transfer protocol to load a file stored on the host system. Once the kernel is running, you can easily download and run object modules via Ethernet. If necessary, the code can even be replaced by a new version, simply by re-loading an updated module.
There’s also a wide range of tools to analyse the behaviour of the software you’re testing, from traditional source-level debugging, through to system analysis and tuning. All of these normally take advantage of the high bandwidth of an Ethernet connection to transfer data quickly between the host and target.
Using Ethernet in this way has become almost a given for VxWorks developers. But working on a recent project, I re-discovered how to use VxWorks without it.
Surprisingly, this was not a stripped-down, tightly embedded system, but a rather complex application running on a “big” processor (a Freescale i.MX 6). Although the processor had integrated Ethernet interfaces, they were not needed in the final system and weren’t connected to the outside world.
The first question to tackle was how to boot the VxWorks kernel, or at least how to load it into memory. Normally the target would have a VxWorks boot loader, which could load a kernel file from a USB mass storage device, making quite a convenient boot mechanism. In this project, although the target hardware had USB interfaces, instead of running the VxWorks boot loader, it had a version of U-Boot that didn’t support the USB interfaces. In fact the only boot mechanism available to me was to download the VxWorks kernel via a serial connection – it’s a long time since I’ve had to do that!
Loading the kernel via a serial connection took several minutes each time. That changed my approach to development quite noticeably: with a quick re-boot cycle through an Ethernet connection, I’d have tried many more experiments safe in the knowledge that there wasn’t much time lost if they didn’t work out. Booting via the serial connection made me much more cautious and considered when trying new builds – I had to adopt more of a “measure twice and cut once” approach. On reflection, that’s a discipline I’d like to continue even when the penalty for getting it wrong is small.
Testing, Debugging and Tuning
Once the kernel was booted, loading software modules onto the target was quick and easy enough. VxWorks supports the USB interfaces on the target, so I could simply copy my object modules onto a USB memory stick and use the VxWorks shell to load them into the target. Without USB, one option would have been to link my code with the kernel, but a better approach might have been to configure the Wind Debug Agent (WDB) to use a serial connection to the target. With that in place, I could have dynamically loaded object code into the target without having to re-boot the complete kernel.
Configuring WDB to use a serial link to the target would also have allowed me to use the source-level debugging tools built into Workbench. An alternative that I use more often is the target-resident shell. That provides a good environment for experimenting with and debugging code – provided you’re happy to adapt your debugging process a little, since it doesn’t have any knowledge of source code.
After debugging, unit testing and integration, the next step is normally tuning and optimisation, whether on a small-scale or system-wide basis. This is the area I found most challenging when working without the Ethernet connection. The tool I’d normally reach for first at this point is Wind River’s System Viewer, which traces the execution and sequence of events through the different contexts in the complete system software. System Viewer collects this data on the target and ships it up to the host, where it’s displayed in a timeline – it’s the software analogue of a logic analyser. In a busy system, System Viewer can collect quite large amounts of data quickly, so it naturally benefits from a high-bandwidth connection between host and target. System Viewer is a great tool both for identifying problems in the sequence of execution, and for measuring the time it takes to execute specific sections of code.
In my case, using System Viewer over the serial link would have been quite limiting, so instead I used some much leaner techniques – the timing and profiling tools built into the target shell. Although those are much simpler and less sophisticated than System Viewer, they’re still good enough to identify CPU usage hotspots and time the execution of individual functions down to microsecond accuracy.
All in all, my recent experience developing on a VxWorks system without Ethernet reminded me what a well-crafted system it is. Even with a fairly limited set of interfaces, the development tools worked quite effectively. Nevertheless, my advice to the customer for next time would be to design an Ethernet interface into the hardware to better support the software development process. The parts needed outside the processor could be populated for the prototype/development boards, but left off the production hardware to save cost.