DE

Solo developer

Dull Edge Technologies

Documentation

10/29/26 - OOM SSH Stall

Home

A document for 10/29/26 - OOM SSH Stall.

10/29/26 - OOM SSH Stall

By refactoring from software architecture V0.5 -> V0.6 the memory foot print increased to 580Mb. This was not known at the time. At the time it was believed that the migration decreased the memory usage. This proved to not be the fact when the Pi stopped responding to ssh, the web interface stopped being served. The fix for this was to reflash the OS configure 800Mb of swap and prevent systemd from rebooting the program after initial boot.

How did OOM cause SSH to stall? in theory the kernel would start killing processes once memory starts to run out. By analyzing the logs it was discovered that it was, however systemd was restarting the nodes immediately. In this case systemd should have realized that the program was dying on repeat, however the exact process killed was slightly randomized between the different ros node. Making the possibly burst detection fail. It is possible that after sufficient time systemd would have recognized the boot loop.

How did this progress to reflashing? the ideal remediation would have been to kill the ros service, configure a swap file, and gather a base line. However this requires shell access. With SSH not stalling the option was to make a physical connection. This partially worked, when a keyboard was plugged in it would input to the default shell environment. The issue is that pcb_projector would not stay dead or consistently be killed, meaning that the display would be overwritten by the program as intended. Meaning the option was to wait several minutes to get a couple seconds of seeing what the shell was currently saying, or typing blindly. I am not familiar enough with the default behavior to blindly navigate killing a system.

Next step, mount the file system on a separate computer. The next idea was to access the file system, investigate systemd-journald and remove the ros process service file. This proved more difficult because every other computer on hand runs windows. So modify the existing ext4 formatted drive was not feasible. Leading to reformatting as the only solution.

Root cause: increased use of memory by software refactor. Solution: add swap file, reconfigure system revival behavior, investigate adding memory usage to testing.