FlightAware Discussions

RasPi airspy_adsb process hang?

Hi - I wondered if anyone has seen this behaviour with the latest version of airspy_adsb?

airspy_adsb seemed to hang on my RasPi4 overnight. This was evident from graphs1090:

However when I ran systemctl status, it was not listed as failed:


jrg@raspi4:~ $ sudo systemctl status
[sudo] password for jrg: 
● raspi4
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 1970-01-01 10:00:02 AEST; 50 years 7 months ago
   CGroup: /
           ├─user.slice
           │ └─user-1001.slice
           │   ├─session-c6.scope
           │   │ ├─17845 sshd: jrg [priv]
           │   │ ├─17901 sshd: jrg@pts/0
           │   │ ├─17904 -bash
           │   │ ├─27970 sudo systemctl status
           │   │ ├─28054 systemctl status
           │   │ └─28055 pager
           │   └─user@1001.service
           │     └─init.scope
           │       ├─17881 /lib/systemd/systemd --user
           │       └─17885 (sd-pam)
           ├─init.scope
           │ └─1 /sbin/init splash
           └─system.slice
             ├─collectd.service
             │ └─523 /usr/sbin/collectd
             ├─timelapse1090.service
             │ ├─  473 /bin/bash /usr/local/share/timelapse1090/timelapse1090.sh
             │ ├─  481 /bin/bash /usr/local/share/timelapse1090/timelapse1090.sh
             │ └─28036 sleep 10
             ├─graphs1090.service
             │ ├─  301 /bin/bash /usr/share/graphs1090/service-graphs1090.sh
             │ ├─27975 /bin/bash /usr/share/graphs1090/graphs1090.sh 7d 0.7
             │ └─28035 sleep 0.7
             ├─dbus.service
             │ └─329 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
             ├─chrony.service
             │ ├─822 /usr/sbin/chronyd -F -1
             │ └─823 /usr/sbin/chronyd -F -1
             ├─ssh.service
             │ └─687 /usr/sbin/sshd -D
             ├─airspy_adsb.service
             │ └─353 /usr/local/bin/airspy_adsb -v -f 1 -t 300 -w 3 -e 9.4 -l 29999:beast -l 47806:asavr -c localhost:30004:beast -g 20 -m 20
             ├─tar1090.service
             │ ├─  491 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090 /run/dump1090-fa
             │ ├─  511 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090 /run/dump1090-fa
             │ ├─  875 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090 /run/dump1090-fa
             │ ├─27953 sleep 8
             │ └─28043 sleep 10
             ├─system-getty.slice
             │ └─getty@tty1.service
             │   └─560 /sbin/agetty -o -p -- \u --noclear tty1 linux
             ├─wpa_supplicant.service
             │ └─334 /sbin/wpa_supplicant -u -s -O /run/wpa_supplicant
             ├─tar1090-persist.service
             │ ├─  487 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090-persist /run/dump1090-fa
             │ ├─  490 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090-persist /run/dump1090-fa
             │ ├─  872 /bin/bash /usr/local/share/tar1090/tar1090.sh /run/tar1090-persist /run/dump1090-fa
             │ ├─27942 sleep 14
             │ └─27964 sleep 10
             ├─lighttpd.service
             │ └─637 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf
             ├─systemd-logind.service
             │ └─318 /lib/systemd/systemd-logind
             ├─polkit.service
             │ └─11670 /usr/lib/policykit-1/polkitd --no-debug
             ├─cron.service
             │ └─311 /usr/sbin/cron -f
             ├─systemd-udevd.service
             │ └─148 /lib/systemd/systemd-udevd
             ├─rsyslog.service
             │ └─338 /usr/sbin/rsyslogd -n -iNONE
             ├─systemd-journald.service
             │ └─119 /lib/systemd/systemd-journald
             ├─dhcpcd.service
             │ └─466 /sbin/dhcpcd -q -b
             ├─dump1090-fa.service
             │ └─4666 /usr/bin/dump1090-fa --net-only --max-range 360 --fix --net --net-heartbeat 60 --net-ro-size 1300 --net-ro-interval 0.2 --net-ri-port 0 --net-ro-port 30002 -
             ├─piaware.service
             │ ├─ 649 /usr/bin/piaware -p /run/piaware/piaware.pid -plainlog -statusfile /run/piaware/status.json
             │ ├─1982 /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,
             │ └─4738 /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat -38.296 --lon 144.391
             └─pfclient.service
               └─578 /usr/bin/pfclient -d -i /var/run/pfclient.pid -z /etc/pfclient-config.json -y /var/log/pfclient $


jrg@raspi4:~ $ sudo systemctl status airspy_adsb
● airspy_adsb.service - Airspy ADS-B receiver
   Loaded: loaded (/etc/systemd/system/airspy_adsb.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-08-21 09:17:04 AEST; 4 days ago
     Docs: https://discussions.flightaware.com/t/howto-airspy-mini-piaware-dump1090-fa-configuration/44343/2
 Main PID: 353 (airspy_adsb)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/airspy_adsb.service
           └─353 /usr/local/bin/airspy_adsb -v -f 1 -t 300 -w 3 -e 9.4 -l 29999:beast -l 47806:asavr -c localhost:30004:beast -g 20 -m 20

Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 262144 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 262144 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 131072 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 131072 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 393216 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 262144 samples /!\
Aug 21 09:17:05 raspi4 airspy_adsb[353]: /!\ Lost 131072 samples /!\
Aug 21 09:17:08 raspi4 airspy_adsb[353]: Push client connected to localhost:30004 (beast)
Aug 23 04:44:02 raspi4 airspy_adsb[353]: Push client disconnected from localhost:30004 (beast)
Aug 23 04:44:15 raspi4 airspy_adsb[353]: Push client connected to localhost:30004 (beast)

As the graphs show, rebooting restored normal operations.

If this happens again, are there are other things I should check before rebooting? (I probably should have tried just restarting airspy_adsb, as some logs may have been lost in the reboot)

There is nothing in /var/log/messages or /var/log/syslog at the relevant time that gives any hint of a problem. The piaware log reports no messages received and no mlat data every five minutes, and restarts of dump1090 every hour or so.

My set up is a Raspi 4 fed from an Airspy Mini attached to a Uputronics SAW filter/Pre-amp - powered separately from the Pi via 50cm 24awg USB cables. The Pi is powered via POE through a RasPi POE hat. Latest version of Raspbian Buster, airspy_adsb v1.85

I have run Wiedehopf’s undervoltage script - that shows no problems.

Any thoughts on possible causes or diagnostics would be appreciated.

Thanks, John

This happened earlier this year on the MTG receiver but even though the graphs showed zero, it was actually still running. I know this because I didn’t look at it for a couple of weeks and it had been down all that time.

I took it as a graph glitch, not an airspy_adsb glitch.

The overall CPU usage wouldn’t glitch like that.

Cool the airspy mini?
Run 12 MHz sample rate?

Not much you can do i think, not sure what the root cause might be.

The log you posted has messages from the 23rd, but according to the graphs you seem to have rebooted on the 26nd, so … check the logs again?

sudo journalctl -u airspy_adsb --no-pager

Thanks for the quick reply.

In my case it was a real airspy_adsb glitch. My Flightaware stats show no messages for 8 hours prior to rebooting, but nearby sites with similar coverage were receiving some messages (only about 15 aircraft as there is very little traffic here due to the time of day and Covid second wave lockdowns) Will elaborate in my reply to Wiedehopf.

John

Good stuff, it was worth checking :slight_smile:

Hello Weidehopf,

I don’t think cooling or sample rates are the problem.

It is winter down here in southern Australia at the moment and my setup is in an uninsulated roof space below and aluminium sheet roof - according to my weather station data, the ambient temperature was 8 degrees Celsius at the time of the “hang”. Also, the Airspy mini is directly connected to the Uputronics filter/pre-amp, and that has a a large aluminium case which should act as good heatsink.

I’ve been running at 20Mhz sample rate, and higher reported temperatures without problems for about 3 months - you can see in the following graphs when I cranked up the sample rate, etc.

The RasPi temperatures are more stable since I installed the POE hat, which includes a cooling fan, in mid June.

The logs I posted were from immediately before I rebooted - the RasPi had last been rebooted on 21 August. Here is the current output as requested:


-- Logs begin at Wed 2020-08-26 07:47:09 AEST, end at Wed 2020-08-26 19:54:38 AEST. --
Aug 26 07:47:12 raspi4 systemd[1]: Started Airspy ADS-B receiver.
Aug 26 07:47:12 raspi4 airspy_adsb[325]: airspy_adsb v1.85
Aug 26 07:47:12 raspi4 airspy_adsb[325]: Listening for beast clients on port 29999
Aug 26 07:47:12 raspi4 airspy_adsb[325]: Listening for asavr clients on port 47806
Aug 26 07:47:12 raspi4 airspy_adsb[325]: Acquired Airspy device with serial B8B067DC3623901F
Aug 26 07:47:12 raspi4 airspy_adsb[325]: Decoding started at 20 MSPS
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 262144 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 393216 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 917504 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 786432 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 524288 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 1048576 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 786432 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 393216 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 393216 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 262144 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 131072 samples /!\
Aug 26 07:47:12 raspi4 airspy_adsb[325]: /!\ Lost 262144 samples /!\
Aug 26 07:47:13 raspi4 airspy_adsb[325]: /!\ Lost 131072 samples /!\
Aug 26 07:47:13 raspi4 airspy_adsb[325]: /!\ Lost 131072 samples /!\
Aug 26 07:47:16 raspi4 airspy_adsb[325]: Push client connected to localhost:30004 (beast)

Does the disconnection and reconnection shown on 23 August on the earlier log have any significance? I have seen that when monitoring the systemctl status airspy_adsb in recent weeks, but assumed it has something to do with periods when there were no aircraft within range due to Covid flight reductions.

I grateful for your response - I just wanted to check that I wasn’t missing something obvious. My system is back to normal, so will just hope that this was a random glitch.

John

1 Like

Don’t think so that looks like dump1090-fa might have been restarted?
Oh actually if there are no planes at all, piaware likes to restart dump1090-fa after not receiving any aircraft for some time, so that could be it?
Check the dump1090-fa / piaware logs.

Yeah … if it doesn’t happen again then that’s just fine.
It shouldn’t affect it, but running the higher sample rate makes hangs more likely due to the Airspy using more power.

Even without the RPi having undervolt errors i’ve seen the airspy mini go unstable unless the RPi provides perfect voltage via USB.
20 MHz is running the Airspy Mini at the limit, so it needs the full 5.0 V and not even 4.95 :slight_smile:

Thus if it happens again and you don’t absolutely need the 20 MHz, 12 MHz might provide more stability.

1 Like

I was having the same problems with airspy_adsb a while ago and I tracked it down to a power problem. It would stop for me even at 12MSPS. If this keeps happening I’d look at the POE hat. If possible try running your RPi 4 using the USB-C power brick for a while and see if that fixes it. You can keep the HAT on while using the power brick for cooling. This is exactly what I saw. Airspy just stopping out of the blue. I didn’t have any undervolt messages either. Do what you can to diagnose a power issue. Its the most likely.

Based on your and Hampstershorts’ advice, I’ve ordered a cheap USB voltage/current meter. It’s only accurate to +/- 1%, so I will not be able to tell the difference between 4.95V and 5.00V :slight_smile:

That will let me get an idea of voltage differences across my set-up. If necessary, I will try reducing the sample rate.

I had not understood that it was the Airspy Mini that had hung rather than the airspy_adsb process. I’d assumed that the ADC would still be sending noise of some kind to airspy_adsb, which should still be trying to process it, and just not finding any signals - in that case graphs1090 would still have shown cpu utilisation.

Although systemctl reported airspy_adsb as active(running) 8 hours after it last received any data from the Airspy, I had assumed that if it was still running, it would have disconnected or flagged an error. Am I right in thinking that, once connected, airspy_adsb will wait indefinitely for the Airspy to send data?

I’ve come up with a crude workaround concept. My unix scripting skills are pretty weak, but I will try to write a script attached to a chron job that regularly looks at the airspy_adsb parent process cpu utilisation, and if it falls below, say 5%, has systemctl restart it. In my case, the graph looks like it fell to close to zero. (I’m assuming that restarting airspy_adsb will have the same effect as a full reboot in terms of getting the airspy running again.)

You are right that piaware includes this sort of logic when it doesn’t receive any aircraft messages for a period of time. It’s log showed that it restarted dump1090-fa approximately every 65 minutes when the Airspy was hung.

If I get the script working, and have another hang to test it on, I will post it to this thread.

Thanks Hampstershorts,

I will do a proper check of voltage available to the Airspy Mini.

I’ve been experimenting with the POE setup because I want to move my antenna from the house to the top of a tree that will give it an unobstructed view over a sand dune out across Bass Strait and the Southern Ocean (I’m on the surf coast of Victoria, Australia). To do that I need a minimum cable run of 50m.

Prior to making the move, I’m testing an ideal cable setup that involves 75m of cat6 ethernet cable:

UPS -> 802.3at POE Switch/injector -> 25m cable -> 802.3at POE powered switch with 802.3af POE pass through -> 50m cable -> RasPi 802.3af POE hat

In theory this should be able to deliver up to 20 Watts to the RasPi, and the POE hat’s rated output is 12.5 Watts, however given your experience, I may be pushing the envelope.

I’m trying to avoid having to install a new mains electricity outlet in my garage - the structure closest to the tree where the antenna is going. But if I have to, I can do that and ditch the POE passthrough switch and 25m of cable. The current setup worked for 4 days before the Airspy hang, so I will keep it running to see if the problem repeats while I wait for my USB voltage tester to arrive.

John

Yeah that’s my guess to what happens.
I’d have to check the code. (i’ve worked on it a bit with the author (called prog here on the forum))

I’m not sure how hung the airspy mini becomes, it might even need a reboot.

Just put a LNB there with power over coax and be done. Add ground wire to an electrode by the tree root and a gas-filled surge arrester.
I am running 45 meters RG6 with a 30dB LNA that has internal power regulation (5V). So I just send 7.5V to it…

1 Like

In my case the problem was actually the RPi itself. Initially I was running a 3B+, and I was having the problem with the Mini just stopping without error. During my troubleshooting I swapped in a RPi 3B, and it worked fine. I am now running a RPi 4B. So there was some kind of silent power problem on that 3B+ board that was interfering with the Mini. I know that’s not necessarily a satisfying answer. You could also try ordering another POE hat from a different source, hoping to get one from a different production run. There could be a problem with the hat as well that isn’t showing up in the logs. I’d just start changing 1 component of your setup at a time and see where you get.

In my experience, you just need to crank up the voltage.
If you want to go with POE, get something that will get the 48 V out of the ethernet or some jig that converts it to 12 V.
Then get a converter either from 48 V or from 12 V to 5 V that is adjustable.
Then under load running airspy_adsb you adjust it so you have 5.0 V on the USB.

Maybe there is a way to slightly crank up the voltage for the POE thing you’re running currently?
Adding a capacitor might also help to reduce the ripple.

I’m pretty sure it was something with the RPi 3B+ itself because I even ran it on a 5.1V 2.5amp USB power supply and it did the same thing.

That’s basically insufficient to get 5.0 V at the USB port.
For that with load, you need something like 5.2 V or even 5.25 V at the RPi3B+ power input.

Well for what it’s worth I had a 5.25V power supply as well and tried that. But it had the same problem on my 3B+.

The label doesn’t necessarily say much.
Measuring at the RPi input and the USB port is the only thing that matters.

Anyhow the power design for the RPi4 is quite a bit better and one shouldn’t have any issues with the official power supply.