Edge Computing

Embedded IoT Linux won't reboot - how to fix and repair

By

I have a lot of embedded systems running in my product testing lab or at home, all of them either as Raspberries or self-made PCB with Yocto. Sometimes I can't reboot some systems, I think its the journald which causes some issues with SSD Cards, the error-message usually is:

Failed to open /dev/initctl

Anyhow, if you have this issue - a reboot can be force-forced:

systemctl --force --force reboot

Since a forced reboot does not sync the journal, the system typically comes up with a damaged FS. The remote fsck can be initiated by extending the command above with

sudo tune2fs -i 1m /dev/DISK && touch /forcefsck && systemctl --force --force reboot

(assumed you have access to a shell, via SSH or local access). When all goes fine, the system comes up with a clean FS. All this fuss comes from the SSD r/w actions, a well designed IoT embedded devices should have a flash mem part for the running OS.

Need help with distributed systems, backend engineering or data platforms? See my consulting services.

Alexander Alten

Alexander Alten

Founder of Scalytics, co-creator and PMC member of Apache Wayang, and author of KafScale. I write about Kafka, streaming, data platforms, IoT and AI agent infrastructure in production.