Our engineers debug Spotters from our global network across some of the most remote regions on Planet Ocean. Software designers and hardware developers virtually gathered at the ArmDev Summit for technical programs on energy-efficient computing solutions, from Edge AI to cloud native architecture, with a keynote session on open source software. Wondering how engineers at Sofar Ocean troubleshoot Arm Cortex-M devices through product development with limited bandwidth and resources? In Troubleshooting At Sea - Debugging Remote ARM Cortex-M Devices Without Physical Access, Alvaro Prieto talks through tips and tricks for doing so. Read on to learn more about what it takes to maintain the largest private network of distributed marine sensors.
Pre-prototype Stage
Once a microcontroller/platform is selected, I highly recommend getting a development board from the vendor to get a head start on firmware development. In our case, we used an STM32L4 processor, so I got some STM32-Nucleo development boards to get started.
If this is a brand new project, this is the time where you select your toolchain, build system, and Real-time Operating System (RTOS), if any. In our case, we're using gcc, CMake, and FreeRTOS. If you have the time, you can make sure all of this is working before you even get the development boards. Nothing makes a team happier than having firmware ready the day hardware arrives! Even if it's not fully functional, being able to flash something onto a new board will make the bringup process much more pleasant. Later on, we'll cover how to make device bring-up faster with a Command Line Interface (CLI).
In our case, I was able to get many device drivers up and running with a development board. These included I2C, SPI, SDIO, UART, USB, and I2S. I was also able to get FreeRTOS and our debugging environment working. In this case, we're using STLink-v3 adapters with OpenOCD and GDB.
Reproducible Builds
I briefly mentioned reproducible builds in the presentation. This means being able to compile the exact same binary on different computers - maybe even different operating systems. This is harder than it sounds. At the very least, you need to make sure all systems have the exact same version of all of the tools. It is very common for system paths to get pulled into symbols as well, which makes the output change depending on where it is.
Prototype Stage
This is where most of the development should happen. If possible, get a prototype PCB with as many test points as possible. Don't forget to put lots of ground test points to reference signals to! Make sure you can measure every signal and bring out spare microcontroller pins to headers. This will make troubleshooting firmware/hardware issues much simpler. It may not be feasible to have everything mentioned here, but it helps to stay aspirational.
Having extra pins broken out can be helpful if you want to try out a new sensor or fix a possibly misrouted signal.
Debugging
At this stage, your board should have a SWD/JTAG connector available. This will allow you to quickly flash your device as well as using debugging software to step through code, read out memory, set breakpoints, and watchpoints. As I mentioned earlier, we mostly use GDB. Other alternatives include SEGGER's Ozone, CLion, and Eclipse, to name a few.
Having all of the previously mentioned test points and headers allows you to easily use oscilloscopes, multimeters, and logic analyzers to help in debugging firmware and hardware issues.
JTAG/SWD Tips and Tricks
Breakpoint if debugger is connected
Cortex-M devices have a nice feature that lets you check whether or not a debugger is connected. (Check bit-0 of the DHCSR register). One neat way to use this is to hit a breakpoint on failed asserts, instead of restarting, but only if a debugger is connected. If there's no debugger, the normal assertion failure procedure can be followed.
GDB Python
If you're using GDB for anything more than stepping through code and looking at a few variables, I highly recommend that you start using Python. GDB has optional Python support which enables some really neat debugging features.
Catching Null Pointer Dereferences
Unfortunately, with Arm processors, Null pointers are just pointing to 0, which is a perfectly valid address! In the STM32 part we're using, it's the beginning of the Flash memory. If you have a null pointer and you try to dereference it and write to it, you won't get an error (but it may have an unintended effect if you do want to write to flash later on!). If your device has a memory protection unit (MPU), you can set it up to cause a memory/bus fault when any code (or peripheral) tries to write to that address. This can be extremely helpful for debugging.
For a deep dive into ARM Cortex-M Debug Interfaces, check this article out.
Pre-production
At this point, your hardware might be in an enclosure, so no more easy access to JTAG/SWD. We'll instead use USB and/or a serial port to continue debugging.
Printf Debugging
You've probably heard of "printf debugging". This is where you start putting print statements around the code you suspect isn't working to try and narrow down on where the problem is. This can be very useful, but can also introduce additional delays and even temporarily fix the problem!
CLI
Command-line interfaces can be incredibly useful for both debugging and board bringup. They allow you to interact with the hardware and do more interactive debugging, versus just inserting a printf, recompiling, flashing, running, which is very slow.
Here are some commands I've found helpful in the past:
GPIO Control
Whenever I'm bringing up new hardware, I like exposing GPIO control over the command line. This allows me to read, write, and sometimes even configure some (or all) of the IO pins in the microcontroller. This lets you do things like:
•Toggle the enable/disable line on a power switch
•Turn on LEDs
• Anything else you could do by toggling a pin!
Serial Interface Access
Another useful set of commands is exposing access to various serial interfaces. These include SPI, I2C, UART, etc... One extremely useful command is an
command, which tries every possible I2C address and returns a list of the ones that responded with an ACK. This can immediately tell you if an I2C device is present and if the addresses are what you expect. Another possible command could be something like:
This would let you generate a SPI transaction with a particular chip select pin and arbitrary data. (Pretend that the xx's are the bytes you plan on transmitting) This command lets you communicate with and test SPI peripherals without having written the drivers. It can be extremely useful during board bring up.
Finally, I like to be able to redirect various serial input/outputs to the serial terminal. For example, if our CLI is coming in over USB, I can redirect the GPS serial port to USB and see all of the data that is coming in over that port. I can also send data to it with a command like
where
is the data I want to send.
Peek/Poke
For the brave, you can add direct memory read/write commands. This way, you can use the CLI to access any device memory. Yes, this is incredibly powerful, but also dangerous. Use with care!
Reset Into Bootloader
Some Cortex-M devices have a built-in ROM bootloader. The STM32, for example, has a built-in USB bootloader that can be accessed by holding the BOOT pin high during boot. Thankfully, there are ways to access it directly from software, so we don't need access to that boot pin to enter the bootloader! In the case of the STM32 we are using, the ROM bootloader starts at address 0x1FFF0000, so we can jump to it to enter the bootloader - here’s an example.
If you're implementing this functionality, the best time to jump into the bootloader is right after boot, when all peripherals are in their reset state. The bootloader might not work properly if, for example, the systick(or other) interrupt is enabled and configured. The best way to do this is to set a magic number in a '.noinit' variable that is checked(and cleared!) right after boot. For example:
The attribute .noinit tells the compiler not to initialize this variable during boot. In some devices, certain RTC registers can also be used instead of .noinit sections to persist data across reboots. Until next time, happy debugging!
If you're interested in learning more about Sofar or related services, visit our website. An example of an interesting project we're working on now is Bristlemouth. With the Office of Naval Research, DARPA, and Oceankind, our engineering team is building Bristlemouth: a hardware connector that works everywhere. Bristlemouth presents a new ”plug-and-play” standard for ocean sensing with streamlined software and firmware customizations. Universal marine hardware integration with Bristlemouth reduces the cost and complexity of ocean data collection. The easy-to-adopt connectivity standard enables interoperable applications of state-of-the-art marine technologies. Join the effort here: https://www.bristlemouth.org/.
