I have four Raspberry Pis that were running PiAware 3.5.0.
Two are running Raspbian Jessie Lite and two run Raspbian Stretch Lite. They are all configured to reboot daily at 3am. Automatic updates are disabled, except for flightaware updates.
They all use different complicated (24+ character) passwords and are not directly accessible from the internet.
At 3am this morning they all went down for a reboot as expected, but they didn’t come back up as expected.
The ones running Stretch Lite rebooted, started sending data to Flightaware again and external services (Web view on port 8080) came back up, but I can no longer SSH into them. The SSH negotiation starts, but then I get the cryptic error
-bash: relocation error: -bash: symbol strcm▒, version GLIBC_2.4 not defined in file libc.so.6 with link time reference
and it then disconnects.
The ones running Jessie Lite rebooted, got an IP address via DHCP, are pingable, not sending any data to flightaware and are actively refusing all connections including SSH.
Any ideas? Was a flightaware update pushed out yesterday?
Edit: They are not running the prebuilt piaware images, these are all running Raspbian with PiAware and Fr24Feed installed and configured by me. dump1090-fa does all the hard work, while Fr24feed just accesses the dump1090-fa datafeed on port 30002.
They have all been running like this (including the daily reboots) for at least 2 months
There have been no recent FA auto-updates pushed out (last was around the time of the 3.5.0 release, a few months ago) so it sounds like something specific to how you are configuring your systems. Perhaps you have an unattended upgrade configured (generally a bad idea for remote systems). Sounds like version skew between libc and sshd bash.
(Scheduled reboots are also not necessarily a great idea…)
I’ve confirmed that it’s not related to Flight monitoring (FlightAware or Fr24Feed) as I have two other RPi’s running as LoRaWAN gateways which are also inaccessible (but fortunately still running). Maybe I need to consider netboot or something similar.
We run a rural wireless ISP, two of these Pis are at the top of very remote hills. Normally it would be accessible by quad bike by September, but we’ve had record rainfall and the tracks are still unusable. I’ll prep some replacement RPis, but will ultimately have to wait for the weather to improve, or for a network critical failure to justify either the hike in on foot, or if it’s heavy, the helicopter trip.
Photos are for anyone interested. In the 2nd photo you can (just) see the ADS-B antenna (Homemade coaxial collinear) on the horizontal bar between the two dishes. I’ve got some proper flightaware antennas sitting here ready to go up the next time I visit.
One tactic we use on the FlightFeeder images is to run them readonly most of the time, switching to readwrite only when we really need to (upgrades, etc). That minimizes the chance of something going wrong (at worst a powercycle will bring them back) & almost eliminates write load on the sdcard. I can probably share the scripts that set it up if there’s interest.
I can confirm that doing a “reboot now” on a Raspberry pi v4, with the latest Flight Aware software version, hung up with a fix red led.
Power cycling make reboot ok. No obvious messages on /var/log/syslog nor dmesg.
Researching.
Tried with Raspberry PI 3: also hung.
