FlightAware Discussions

RasPi airspy_adsb process hang?

Hi - I wondered if anyone has seen this behaviour with the latest version of airspy_adsb?

airspy_adsb seemed to hang on my RasPi4 overnight. This was evident from graphs1090:

However when I ran systemctl status, it was not listed as failed:


jrg@raspi4:~ $ sudo systemctl status
[sudo] password for jrg: 
● raspi4
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 1970-01-01 10:00:02 AEST; 50 years 7 months ago
   CGroup: /
           ├─user.slice
           │ └─user-1001.slice
           │   ├─session-c6.scope
           │   │ ├─17845 sshd: jrg [priv]
           │   │ ├─17901 sshd: jrg@pts/0
           │   │ ├─17904 -bash
           │   │ ├─27970 sudo systemctl status
           │   │ ├─28054 systemctl status
           │   │ └─28055 pager
           │   └─user@1001.service
           │     └─init.scope
           │       ├─17881 /lib/systemd/systemd --user
           │       └─17885 (sd-pam)
           ├─init.scope
           │ └─1 /sbin/init splash
           └─system.slice
             ├─collectd.service
             │ └─523 /usr/sbin/collectd
             ├─timelapse1090.service
             │ ├─  473 /bin/bash /usr/local/share/timelapse1090/timelapse1090.sh
             │ ├─  481 /bin/bash /usr/local/share/timelapse1090/timelapse1090.sh
             │ └─28036 sleep 10
             ├─graphs1090.service
             │ ├─  301 /bin/bash /usr/share/graphs1090/service-graphs1090.sh
             │ ├─27975 /bin/bash /usr/share/graphs1090/graphs1090.sh 7d 0.7
             │ └─28035 sleep 0.7
             ├─dbus.service
             │ └─329 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
             ├─chrony.service
             │ ├─822 /usr/sbin/chronyd -F -1
             │ └─823 /usr/sbin/chronyd -F -1
             ├─ssh.service
             │ └─687 /usr/sbin/sshd -D
             ├─airspy_adsb.service
             │ └─353 /usr/local/bin/airspy_adsb -v -f 1 -t 300 -w 3 -e 9.4 -l 29999:beast -l 47806:asavr -c localhost:30004:beast -g 20 -m 20
             ├─tar1090.service
             │ ├─  491 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090 /run/dump1090-fa
             │ ├─  511 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090 /run/dump1090-fa
             │ ├─  875 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090 /run/dump1090-fa
             │ ├─27953 sleep 8
             │ └─28043 sleep 10
             ├─system-getty.slice
             │ └─getty@tty1.service
             │   └─560 /sbin/agetty -o -p -- \u --noclear tty1 linux
             ├─wpa_supplicant.service
             │ └─334 /sbin/wpa_supplicant -u -s -O /run/wpa_supplicant
             ├─tar1090-persist.service
             │ ├─  487 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090-persist /run/dump1090-fa
             │ ├─  490 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090-persist /run/dump1090-fa
             │ ├─  872 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090-persist /run/dump1090-fa
             │ ├─27942 sleep 14
             │ └─27964 sleep 10
             ├─lighttpd.service
             │ └─637 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
             ├─systemd-logind.service
             │ └─318 /lib/systemd/systemd-logind
             ├─polkit.service
             │ └─11670 /usr/lib/policykit-1/polkitd --no-debug
             ├─cron.service
             │ └─311 /usr/sbin/cron -f
             ├─systemd-udevd.service
             │ └─148 /lib/systemd/systemd-udevd
             ├─rsyslog.service
             │ └─338 /usr/sbin/rsyslogd -n -iNONE
             ├─systemd-journald.service
             │ └─119 /lib/systemd/systemd-journald
             ├─dhcpcd.service
             │ └─466 /sbin/dhcpcd -q -b
             ├─dump1090-fa.service
             │ └─4666 /usr/bin/dump1090-fa --net-only --max-range 360 --fix --net --net-heartbeat 60 --net-ro-size 1300 --net-ro-interval 0.2 --net-ri-port 0 --net-ro-port 30002 -
             ├─piaware.service
             │ ├─ 649 /usr/bin/piaware -p /run/piaware/piaware.pid -plainlog -statusfile /run/piaware/status.json
             │ ├─1982 /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,
             │ └─4738 /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat -38.296 --lon 144.391
             └─pfclient.service
               └─578 /usr/bin/pfclient -d -i /var/run/pfclient.pid -z /etc/pfclient-config.json -y /var/log/pfclient $


jrg@raspi4:~ $ sudo systemctl status airspy_adsb
● airspy_adsb.service - Airspy ADS-B receiver
   Loaded: loaded (/etc/systemd/system/airspy_adsb.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-08-21 09:17:04 AEST; 4 days ago
     Docs: https://discussions.flightaware.com/t/howto-airspy-mini-piaware-dump1090-fa-configuration/44343/2
 Main PID: 353 (airspy_adsb)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/airspy_adsb.service
           └─353 /usr/local/bin/airspy_adsb -v -f 1 -t 300 -w 3 -e 9.4 -l 29999:beast -l 47806:asavr -c localhost:30004:beast -g 20 -m 20

Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 262144 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 262144 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 131072 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 131072 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 393216 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 262144 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 131072 samples /!\
Aug 21 09:17:08 raspi4 airspy_adsb[353]: Push client connected to localhost:30004 (beast)
Aug 23 04:44:02 raspi4 airspy_adsb[353]: Push client disconnected from localhost:30004 (beast)
Aug 23 04:44:15 raspi4 airspy_adsb[353]: Push client connected to localhost:30004 (beast)

As the graphs show, rebooting restored normal operations.

If this happens again, are there are other things I should check before rebooting? (I probably should have tried just restarting airspy_adsb, as some logs may have been lost in the reboot)

There is nothing in /var/log/messages or /var/log/syslog at the relevant time that gives any hint of a problem. The piaware log reports no messages received and no mlat data every five minutes, and restarts of dump1090 every hour or so.

My set up is a Raspi 4 fed from an Airspy Mini attached to a Uputronics SAW filter/Pre-amp - powered separately from the Pi via 50cm 24awg USB cables. The Pi is powered via POE through a RasPi POE hat. Latest version of Raspbian Buster, airspy_adsb v1.85

I have run Wiedehopf’s undervoltage script - that shows no problems.

Any thoughts on possible causes or diagnostics would be appreciated.

Thanks, John

This happened earlier this year on the MTG receiver but even though the graphs showed zero, it was actually still running. I know this because I didn’t look at it for a couple of weeks and it had been down all that time.

I took it as a graph glitch, not an airspy_adsb glitch.

The overall CPU usage wouldn’t glitch like that.

Cool the airspy mini?
Run 12 MHz sample rate?

Not much you can do i think, not sure what the root cause might be.

The log you posted has messages from the 23rd, but according to the graphs you seem to have rebooted on the 26nd, so … check the logs again?

sudo journalctl -u airspy_adsb --no-pager

Thanks for the quick reply.

In my case it was a real airspy_adsb glitch. My Flightaware stats show no messages for 8 hours prior to rebooting, but nearby sites with similar coverage were receiving some messages (only about 15 aircraft as there is very little traffic here due to the time of day and Covid second wave lockdowns) Will elaborate in my reply to Wiedehopf.

John

Good stuff, it was worth checking :slight_smile:

Hello Weidehopf,

I don’t think cooling or sample rates are the problem.

It is winter down here in southern Australia at the moment and my setup is in an uninsulated roof space below and aluminium sheet roof - according to my weather station data, the ambient temperature was 8 degrees Celsius at the time of the “hang”. Also, the Airspy mini is directly connected to the Uputronics filter/pre-amp, and that has a a large aluminium case which should act as good heatsink.

I’ve been running at 20Mhz sample rate, and higher reported temperatures without problems for about 3 months - you can see in the following graphs when I cranked up the sample rate, etc.

The RasPi temperatures are more stable since I installed the POE hat, which includes a cooling fan, in mid June.

The logs I posted were from immediately before I rebooted - the RasPi had last been rebooted on 21 August. Here is the current output as requested:


-- Logs begin at Wed 2020-08-26 07:47:09 AEST, end at Wed 2020-08-26 19:54:38 AEST. --
Aug 26 07:47:12 raspi4 systemd[1]: Started Airspy ADS-B receiver.
Aug 26 07:47:12 raspi4 airspy_adsb[325]: airspy_adsb v1.85
Aug 26 07:47:12 raspi4 airspy_adsb[325]: Listening for beast clients on port 29999
Aug 26 07:47:12 raspi4 airspy_adsb[325]: Listening for asavr clients on port 47806
Aug 26 07:47:12 raspi4 airspy_adsb[325]: Acquired Airspy device with serial B8B067DC3623901F
Aug 26 07:47:12 raspi4 airspy_adsb[325]: Decoding started at 20 MSPS
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 262144 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 393216 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 917504 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 786432 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 524288 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 1048576 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 786432 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 393216 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 393216 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 262144 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 131072 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 262144 samples /!\
Aug 26 07:47:13 raspi4 airspy_adsb[325]: /!\ Lost 131072 samples /!\
Aug 26 07:47:13 raspi4 airspy_adsb[325]: /!\ Lost 131072 samples /!\
Aug 26 07:47:16 raspi4 airspy_adsb[325]: Push client connected to localhost:30004 (beast)

Does the disconnection and reconnection shown on 23 August on the earlier log have any significance? I have seen that when monitoring the systemctl status airspy_adsb in recent weeks, but assumed it has something to do with periods when there were no aircraft within range due to Covid flight reductions.

I grateful for your response - I just wanted to check that I wasn’t missing something obvious. My system is back to normal, so will just hope that this was a random glitch.

John

1 Like

Don’t think so that looks like dump1090-fa might have been restarted?
Oh actually if there are no planes at all, piaware likes to restart dump1090-fa after not receiving any aircraft for some time, so that could be it?
Check the dump1090-fa / piaware logs.

Yeah … if it doesn’t happen again then that’s just fine.
It shouldn’t affect it, but running the higher sample rate makes hangs more likely due to the Airspy using more power.

Even without the RPi having undervolt errors i’ve seen the airspy mini go unstable unless the RPi provides perfect voltage via USB.
20 MHz is running the Airspy Mini at the limit, so it needs the full 5.0 V and not even 4.95 :slight_smile:

Thus if it happens again and you don’t absolutely need the 20 MHz, 12 MHz might provide more stability.

1 Like

I was having the same problems with airspy_adsb a while ago and I tracked it down to a power problem. It would stop for me even at 12MSPS. If this keeps happening I’d look at the POE hat. If possible try running your RPi 4 using the USB-C power brick for a while and see if that fixes it. You can keep the HAT on while using the power brick for cooling. This is exactly what I saw. Airspy just stopping out of the blue. I didn’t have any undervolt messages either. Do what you can to diagnose a power issue. Its the most likely.

Based on your and Hampstershorts’ advice, I’ve ordered a cheap USB voltage/current meter. It’s only accurate to +/- 1%, so I will not be able to tell the difference between 4.95V and 5.00V :slight_smile:

That will let me get an idea of voltage differences across my set-up. If necessary, I will try reducing the sample rate.

I had not understood that it was the Airspy Mini that had hung rather than the airspy_adsb process. I’d assumed that the ADC would still be sending noise of some kind to airspy_adsb, which should still be trying to process it, and just not finding any signals - in that case graphs1090 would still have shown cpu utilisation.

Although systemctl reported airspy_adsb as active(running) 8 hours after it last received any data from the Airspy, I had assumed that if it was still running, it would have disconnected or flagged an error. Am I right in thinking that, once connected, airspy_adsb will wait indefinitely for the Airspy to send data?

I’ve come up with a crude workaround concept. My unix scripting skills are pretty weak, but I will try to write a script attached to a chron job that regularly looks at the airspy_adsb parent process cpu utilisation, and if it falls below, say 5%, has systemctl restart it. In my case, the graph looks like it fell to close to zero. (I’m assuming that restarting airspy_adsb will have the same effect as a full reboot in terms of getting the airspy running again.)

You are right that piaware includes this sort of logic when it doesn’t receive any aircraft messages for a period of time. It’s log showed that it restarted dump1090-fa approximately every 65 minutes when the Airspy was hung.

If I get the script working, and have another hang to test it on, I will post it to this thread.

Thanks Hampstershorts,

I will do a proper check of voltage available to the Airspy Mini.

I’ve been experimenting with the POE setup because I want to move my antenna from the house to the top of a tree that will give it an unobstructed view over a sand dune out across Bass Strait and the Southern Ocean (I’m on the surf coast of Victoria, Australia). To do that I need a minimum cable run of 50m.

Prior to making the move, I’m testing an ideal cable setup that involves 75m of cat6 ethernet cable:

UPS -> 802.3at POE Switch/injector -> 25m cable -> 802.3at POE powered switch with 802.3af POE pass through -> 50m cable -> RasPi 802.3af POE hat

In theory this should be able to deliver up to 20 Watts to the RasPi, and the POE hat’s rated output is 12.5 Watts, however given your experience, I may be pushing the envelope.

I’m trying to avoid having to install a new mains electricity outlet in my garage - the structure closest to the tree where the antenna is going. But if I have to, I can do that and ditch the POE passthrough switch and 25m of cable. The current setup worked for 4 days before the Airspy hang, so I will keep it running to see if the problem repeats while I wait for my USB voltage tester to arrive.

John

Yeah that’s my guess to what happens.
I’d have to check the code. (i’ve worked on it a bit with the author (called prog here on the forum))

I’m not sure how hung the airspy mini becomes, it might even need a reboot.

Just put a LNB there with power over coax and be done. Add ground wire to an electrode by the tree root and a gas-filled surge arrester.
I am running 45 meters RG6 with a 30dB LNA that has internal power regulation (5V). So I just send 7.5V to it…

1 Like

In my case the problem was actually the RPi itself. Initially I was running a 3B+, and I was having the problem with the Mini just stopping without error. During my troubleshooting I swapped in a RPi 3B, and it worked fine. I am now running a RPi 4B. So there was some kind of silent power problem on that 3B+ board that was interfering with the Mini. I know that’s not necessarily a satisfying answer. You could also try ordering another POE hat from a different source, hoping to get one from a different production run. There could be a problem with the hat as well that isn’t showing up in the logs. I’d just start changing 1 component of your setup at a time and see where you get.

In my experience, you just need to crank up the voltage.
If you want to go with POE, get something that will get the 48 V out of the ethernet or some jig that converts it to 12 V.
Then get a converter either from 48 V or from 12 V to 5 V that is adjustable.
Then under load running airspy_adsb you adjust it so you have 5.0 V on the USB.

Maybe there is a way to slightly crank up the voltage for the POE thing you’re running currently?
Adding a capacitor might also help to reduce the ripple.

I’m pretty sure it was something with the RPi 3B+ itself because I even ran it on a 5.1V 2.5amp USB power supply and it did the same thing.

That’s basically insufficient to get 5.0 V at the USB port.
For that with load, you need something like 5.2 V or even 5.25 V at the RPi3B+ power input.

Well for what it’s worth I had a 5.25V power supply as well and tried that. But it had the same problem on my 3B+.

The label doesn’t necessarily say much.
Measuring at the RPi input and the USB port is the only thing that matters.

Anyhow the power design for the RPi4 is quite a bit better and one shouldn’t have any issues with the official power supply.

@wiedehopf - you were correct about needing to reboot.

This is to provide a follow-up on airspy_adsb process hang issue. I’ve now identified a possible cause, and have written a script that can monitor airspy_adsb and restart/reboot if needed.

As background, after the last hang I left my setup running for 20+ days without the problem recurring. In that time I have been learning a little about bash scripting, writing my script and waiting for a USB voltage tester to be delivered. The tester finally arrived yesterday, and indirectly pointed to a possible cause for the hang.

My Pi4 is located in a roof space, and I was already up there when I realised that I had not shut down the Pi. Not feeling like crawling all the way back, I simply disconnected the Airspy Mini from the the USB port and connected the USB tester into the circuit. Once I got out of the roof I discovered that I had zero aircraft on tar1090 and again appeared to have airspy_adsb hung.

This is a 2-hour graphs1090 detail of the event at around 13:25 - the earlier break at around 12:37 was an intentional reboot:

My script was running and detected the hang - it first restarted airspy_adsb. As that did not work it then rebooted the Pi. The script saves the airspy_adsb and dump1090-fa systemd journal entries before rebooting:

-- Logs begin at Mon 2020-09-21 12:37:17 AEST, end at Mon 2020-09-21 13:40:11 AEST. --
Sep 21 12:37:20 raspi4 systemd[1]: Started Airspy ADS-B receiver.
Sep 21 12:37:20 raspi4 airspy_adsb[357]: airspy_adsb v1.85
Sep 21 12:37:20 raspi4 airspy_adsb[357]: Listening for beast clients on port 29999
Sep 21 12:37:20 raspi4 airspy_adsb[357]: Listening for asavr clients on port 47806
Sep 21 12:37:20 raspi4 airspy_adsb[357]: Acquired Airspy device with serial B8B067DC3623901F
Sep 21 12:37:20 raspi4 airspy_adsb[357]: Decoding started at 20 MSPS
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 131072 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 131072 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 524288 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 131072 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 262144 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 131072 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 262144 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 131072 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 131072 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 262144 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 131072 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 655360 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 1441792 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 655360 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 131072 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 524288 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 131072 samples /!\
Sep 21 12:37:21 raspi4 airspy_adsb[357]: /!\ Lost 262144 samples /!\
Sep 21 12:37:24 raspi4 airspy_adsb[357]: Push client connected to localhost:30004 (beast)
Sep 21 13:24:18 raspi4 airspy_adsb[357]: Decoding stopped
Sep 21 13:24:18 raspi4 airspy_adsb[357]: Push client disconnected from localhost:30004 (beast)
Sep 21 13:37:39 raspi4 airspy_adsb[357]: Caught signal 15
Sep 21 13:37:39 raspi4 systemd[1]: Stopping Airspy ADS-B receiver...
Sep 21 13:39:09 raspi4 systemd[1]: airspy_adsb.service: State 'stop-sigterm' timed out. Killing.
Sep 21 13:39:09 raspi4 systemd[1]: airspy_adsb.service: Killing process 357 (airspy_adsb) with signal SIGKILL.
Sep 21 13:39:09 raspi4 systemd[1]: airspy_adsb.service: Main process exited, code=killed, status=9/KILL
Sep 21 13:39:09 raspi4 systemd[1]: airspy_adsb.service: Failed with result 'timeout'.
Sep 21 13:39:09 raspi4 systemd[1]: Stopped Airspy ADS-B receiver.
Sep 21 13:39:09 raspi4 systemd[1]: Started Airspy ADS-B receiver.
Sep 21 13:39:09 raspi4 airspy_adsb[29043]: airspy_adsb v1.85
Sep 21 13:39:09 raspi4 airspy_adsb[29043]: Listening for beast clients on port 29999
Sep 21 13:39:09 raspi4 airspy_adsb[29043]: Listening for asavr clients on port 47806
Sep 21 13:39:09 raspi4 airspy_adsb[29043]: airspy_open() failed: AIRSPY_ERROR_NOT_FOUND (-5)
Sep 21 13:39:10 raspi4 airspy_adsb[29043]: airspy_open() failed: AIRSPY_ERROR_NOT_FOUND (-5)
Sep 21 13:39:11 raspi4 airspy_adsb[29043]: airspy_open() failed: AIRSPY_ERROR_NOT_FOUND (-5)
Sep 21 13:39:12 raspi4 airspy_adsb[29043]: airspy_open() failed: AIRSPY_ERROR_NOT_FOUND (-5)
Sep 21 13:39:13 raspi4 airspy_adsb[29043]: airspy_open() failed: AIRSPY_ERROR_NOT_FOUND (-5)
Sep 21 13:39:14 raspi4 airspy_adsb[29043]: airspy_open() failed: AIRSPY_ERROR_NOT_FOUND (-5)
Sep 21 13:39:15 raspi4 airspy_adsb[29043]: airspy_open() failed: AIRSPY_ERROR_NOT_FOUND (-5)
Sep 21 13:39:16 raspi4 airspy_adsb[29043]: airspy_open() failed: AIRSPY_ERROR_NOT_FOUND (-5)
                           [[MESSAGE continues every second until reboot]]

-- Logs begin at Mon 2020-09-21 12:37:17 AEST, end at Mon 2020-09-21 13:40:11 AEST. --
Sep 21 12:37:21 raspi4 systemd[1]: Started dump1090 ADS-B receiver (FlightAware customization).
Sep 21 12:37:21 raspi4 dump1090-fa[477]: Mon Sep 21 12:37:21 2020 AEST  dump1090-fa 3.8.1 starting up.
Sep 21 12:37:21 raspi4 dump1090-fa[477]: Net-only mode, no SDR device or file open.

-- Logs begin at Mon 2020-09-21 12:37:17 AEST, end at Mon 2020-09-21 13:40:11 AEST. --
Sep 21 12:37:20 raspi4 systemd[1]: Started airspy_defib1090: Periodically check the status of airspy_adsb, restart/reboot if required.
Sep 21 12:37:20 raspi4 airspy_defib1090[365]: airspy_defib1090 running with parameters:
Sep 21 12:37:20 raspi4 airspy_defib1090[365]: cycletime= 1800
Sep 21 12:37:20 raspi4 airspy_defib1090[365]: cpu_samples= 10
Sep 21 12:37:20 raspi4 airspy_defib1090[365]: cpu_threshold= 5
Sep 21 12:37:20 raspi4 airspy_defib1090[365]: max_restarts= 1
Sep 21 13:07:37 raspi4 airspy_defib1090[365]: airspy_adsb operating within set parameters: CPU%= 72.8
Sep 21 13:37:39 raspi4 airspy_defib1090[365]: airspy_adsb is unhealthy! - CPU utilisation 0.0% is below 5%
Sep 21 13:39:09 raspi4 airspy_defib1090[365]: airspy_adsb process restarted by airspy_defib1090, restart number: 1
Sep 21 13:40:11 raspi4 airspy_defib1090[365]: airspy_adsb is unhealthy! - CPU utilisation 0.0% is below 5%
Sep 21 13:40:11 raspi4 airspy_defib1090[365]: Machine rebooted by airspy_defib1090 - airspy_adsb process had degraded

If I’m reading the airspy_adsb log correctly, pulling the USB cable (at time 13:24:18) stops decoding and disconnects, but leaves the process unresponsive - the script’s attempt to stop airspy_adsb (at time 13:37:39) doesn’t exit cleanly, and the subsequent start command fails to restart the process. The script recognises that the restart was unsuccessful and reboots the Pi4 (at time 13:40:11)

So in summary, it appears that a transient USB disconnection of the Airspy Mini may have caused the hang that I originally observed. This leaves the airspy_adsb process unresponsive but is not flagged as failed by systemctl status. Rebooting fixes the problem.

I don’t have an explanation for the underlying cause of the transient disconnection.

The following may fall under the category of “a solution in search of a problem”, but I will post my script in case anyone needs a starting point for addressing similar issues. I apologise in advance for the standard of coding - I am a novice with bash scripts, and learnt to “program” a long time ago.

The main part of the script checks the CPU utilisation of the airspy_adsb process at regular intervals (30 minutes in what’s posted below) and if its lower than a set threshold (5% of one core) it stops, then starts the process. It then waits 60 seconds and checks again. If it is still below the threshold, the script saves the logs and then reboots. The parameters can all be changed by editing the values at the top of the script.

#!/bin/bash

# Parameters for airspy_defib1090 - edit as needed

cycletime=1800  #approx time in seconds between tests for airspy_adsb process health [integer]
cpu_samples=10  #number of top samples to calculate airspy_adsb average cpu utilisation [integer] 
cpu_threshold=5 #minimum CPU% indicating that airspy_adsb is healthy [floating point or integer]
max_restarts=1  #number of restarts allowed before rebooting instead [integer]

restart_airspy() {
        # function to restart airspy_adsb, or reboot if that doesn't work
        if [ $restartflag -gt $((max_restarts-1)) ]; then
                #reboot branch:
                #save relevant journal entries to a date/time stamped file to preserve across reboot
                #file location depends on WorkingDirectory setting in airspy_defib1090.service file
                filename="defib_saved_logs"$(date +"%Y%m%d-%H%M")
                journalctl -u airspy_adsb >> $filename 
                echo "" >> $filename
                journalctl -u dump1090-fa >> $filename
                echo "" >> $filename
                echo "Machine rebooted by airspy_defib1090 - airspy_adsb process had degraded"
                journalctl -u airspy_defib1090 >> $filename
                systemctl reboot        #to shutdown, use "halt --now"
        fi
        #try restarting branch:
        systemctl stop airspy_adsb
        systemctl start airspy_adsb
        restartflag=$((restartflag+1))
        sleeptime=60    #reduce sleeptime, check if restart worked after 60 seconds
        echo "airspy_adsb process restarted by airspy_defib1090, restart number: "$restartflag
}

#write parameters to systemd journal (view with "journalctl -u airspy_defib1090"):
echo "airspy_defib1090 running with parameters:"
echo "cycletime= "$cycletime
echo "cpu_samples= "$cpu_samples
echo "cpu_threshold= "$cpu_threshold
echo "max_restarts= "$max_restarts

sleeptime=$cycletime
restartflag=0   #initialise restart counter, begin indefinite loop:

while :
do
        sleep $sleeptime

        # Check status of airspy_adsb, stop then start if it has failed
        # This "is-failed" test is probably redundant, as airpsy_adsb.service includes "restart=always"
        systemctl is-failed --quiet airspy_adsb
        if [ $? -eq 0 ]; then
                echo  "airspy_adsb state = is-failed, restarting/rebooting"
                restart_airspy
        continue                        #truncate loop - see if is-failed condition resets
        fi

        # Get the PID of the airspy_adsb parent process - see dump1090.py in graphs1090 on github
        mnPID=$(systemctl show -p MainPID airspy_adsb | cut -f 2 -d=)

        # graphs1090 airspy_adsb CPU usage is for the third PID listed with "ls /proc/mnPID/task"
        # This is line 10 of the top output, when sorted by ascending PID
        # Read top for the number of times defined by "cpusamples" to allow for potential volatility 
        # Accumulate the CPU% field (9) and take the average
        cpu_sum=0
        for i in $(seq 1 $cpu_samples)
        do
                string1=$(top -b -n1 -p$mnPID -H -o-PID | awk NR==10)
                cpu_percent=$(echo $string1  | cut -d ' ' -f 9)
                cpu_sum=`echo $cpu_sum $cpu_percent | awk '{print $1 + $2}'`
        done

        avcpu_util=`echo $cpu_sum $cpu_samples | awk '{printf "%.1f", $1 / $2}'`
        test=`echo $avcpu_util $cpu_threshold | awk '{print ($1 > $2)}'`
        if [ $test -eq 0 ]; then
                echo "airspy_adsb is unhealthy! - CPU utilisation "$avcpu_util"% is below "$cpu_threshold"%"
                echo "restarting/rebooting"
                restart_airspy
        continue                    #truncate loop, test again after reduced sleeptime
        else
                echo "airspy_adsb operating within set parameters: CPU%= "$avcpu_util 
                sleeptime=$cycletime    #if get to this line, restart worked, so sleeptime can revert
                restartflag=$((restartflag>0 ?(restartflag-1):0))       #also back-off the restart trigger
        fi
done

I’ve called the script airspy_defib1090.sh (sorry for the presumption of using the 1090 suffix - no offence intended).

As provided here it should be installed in a directory of the same name: /usr/local/share/airspy_defib1090 - it requires execute permissions to be set, and need to be run as root.

A systemd service file is needed to run the script. This file should be saved as airspy_defib1090.service in /etc/systemd/system

[Unit]
Description=airspy_defib1090: Periodically check the status of airspy_adsb, restart/reboot if required
Wants=airspy_adsb.service
After=airspy_adsb.service
[Service]
ExecStart=/usr/local/share/airspy_defib1090/airspy_defib1090.sh
Type=simple
WorkingDirectory=/usr/local/share/airspy_defib1090
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=airspy_defib1090
User=root
Group=root
Nice=19
[Install]
WantedBy=multi-user.target

After installing the files, follow the the standard systemctl update procedure:

systemctl daemon-reload
systemctl enable airspy_defib1090
systemctl start airspy_defib1090

The usual caveats apply: use at your own risk, no warranties, etc.

The script has been tested on a RasPi 4B running RaspberryPi OS fully updated 32bit Buster. Note that the first check of airspy_adsb does not take place until after the cycle time set in the script. I’m assuming people follow best practice, and check the status of key processes themselves immediately after reboots!

In case anyone is interested, the observed voltage at the Airspy Mini when operating was 5.01V to 4.98V (+/- ???) and the average current measured over 19 hours = 134mA

1 Like