Having problems with 978 - no data making it to FA

FlightAware ADS-B Flight Tracking
1

I have run a 1090 Pi for over a year, no problems. I decided to try out 978, so I started with a second Pi running an RTL stick, dump978, etc. Keeping them separate from the 1090 unit so I can see the 978 stats and to prevent any messups to the 1090 unit. I consider myself to Pi/Linux capable, but not an expert. These are manually installed, not the FA image.

It did not work at first. I found two problems. I had to add --net-only to the /etc/default/dump1090-fa file to prevent it from using the RTL dongle and feeding 1090. I also added a --net-ri-port 30001 to accept incoming data from dump978 (via netcat, see dump978-maint.sh).

It then seemed to work fine for a day, I was sending data to FA according to my FA stats page. Then it stopped. I’m not sure if I did something or what, but I have yet to be successful in getting it back for more than a quick burst. There is not a lot of 978 activity here (and I am NOT in range of a base station), but there is some.
I added a tee into my dump978 command and I did log some 978 data to a text file, and when put through uat2text it was readable so I know it was good. However, none of that data made it to FA according to my FA stats. I took that tee back out just to eliminate a step.
I have since disabled mlat just to make sure that wasn’t causing a problem, since I don’t think that is relevant to 978 (??).

Here are some relevant files:
/etc/default/dump1090-fa



RECEIVER_OPTIONS="--net-only --net-bo-port 30005"
DECODER_OPTIONS="--max-range 300"
NET_OPTIONS="--net --net-heartbeat 60 --net-ro-size 1000 --net-ro-interval 1 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005 --net-ri-port 30001"
JSON_OPTIONS="--json-location-accuracy 1"

tail of /var/log/piaware.log



Jan 21 16:09:48 raspberrypi4 piaware[444]: faup1090 exited with SIG SIGHUP
Jan 21 16:09:48 raspberrypi4 piaware[444]: attempting to restart dump1090..
Jan 21 16:09:52 raspberrypi4 piaware[444]: attempting to restart dump1090-fa using 'systemctl --no-block try-restart dump1090-fa.service < /dev/null'...
Jan 21 16:09:52 raspberrypi4 piaware[444]: dump1090 restart appears to have been successful
Jan 21 16:10:03 raspberrypi4 piaware[444]: ADS-B data program 'dump1090-fa' is listening on port 30005, so far so good
Jan 21 16:10:03 raspberrypi4 piaware[444]: Starting faup1090: /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat  --lon 
Jan 21 16:10:03 raspberrypi4 piaware[444]: Started faup1090 (pid 1004) to connect to dump1090-fa
Jan 21 16:10:22 raspberrypi4 piaware[444]: 0 msgs recv'd from dump1090-fa (0 in last 5m); 0 msgs sent to FlightAware
Jan 21 16:15:18 raspberrypi4 piaware[444]: 0 msgs recv'd from dump1090-fa (0 in last 5m); 0 msgs sent to FlightAware
Jan 21 16:20:18 raspberrypi4 piaware[444]: 0 msgs recv'd from dump1090-fa (0 in last 5m); 0 msgs sent to FlightAware

dump978-maint.sh (how dump978 is started)



/usr/bin/rtl_sdr -f 978000000 -s 2083334 -g 48 - | /home/pi/dump978/dump978 | /home/pi/dump978/uat2esnt | /bin/nc -q1 127.0.0.1 30001 &

Running processes



root       458 39.5  0.2   2424  1288 ?        R    15:09  29:15 /home/pi/dump978/dump978
root       459  0.0  0.1   2312   864 ?        S    15:09   0:00 /home/pi/dump978/uat2esnt
dump1090   989  0.8  0.5   6468  2440 ?        S<s  16:09   0:07 /usr/bin/dump1090-fa --net-only --net-bo-port 30005 --max-range 300 --net --net-heartbeat 60 --net-ro-size 1000 --net-ro-interval 1 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005 --net-ri-port 30001 --json-location-accuracy 1 --lat 3 --lon  --write-json /run/dump1090-fa --quiet

root       457  7.8  1.2  14808  5576 ?        Rl   15:09   5:50 /usr/bin/rtl_sdr -f 978000000 -s 2083334 -g 48 -

piaware   1004  0.1  0.3   2772  1720 ?        S    16:10   0:00 /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat  --lon -

piaware-status



PiAware master process (piaware) is running with pid 444.
PiAware ADS-B client (faup1090) is running with pid 1004.
PiAware mlat client (fa-mlat-client) is not running.
Local ADS-B receiver (dump1090-fa) is running with pid 989.

dump1090-fa (pid 989) is listening for connections on port 30005.
faup1090 is connected to the ADS-B receiver.
piaware is connected to FlightAware.

dump1090 is producing data on localhost:30005.

port 30001 test



pi@raspberrypi4:~ $ nc -zv 127.0.0.1 30001
Connection to 127.0.0.1 30001 port [tcp/*] succeeded!

I guess my first question, because this what I’m most unsure of, is my dump1090-fa correct? And nc (netcat) does not show in a process list. Should it, or will it only when data is being piped to 30001 since the -q1 option is there?

Thank you, before I pull what is left of my hair out!

2

I have to restart my dump978s every few hours. For some reason they just seem to lock randomly.

Here is what I had from a previous post

change to root user

Create a file called /root/978restart.sh
Add this to it and make it executable
/usr/bin/pkill -f dump978-maint.sh; /usr/bin/pkill -f rtl_sdr; /usr/bin/pkill -f dump978; /usr/bin/pkill -f uat2esnt;/usr/bin/pkill -f uat2json; /bin/sleep 9s; /home/pi/adsb-receiver/build/dump978/dump978-maint.sh & > /dev/null

add it to crontab -e
11 */4 * * * /root/978restart.sh &
This runs the script every 4 hours at 11 minutes past the hour. Change it to suite your needs.
I am not sure why my system locks up, but it does. A restart instantly shows traffic.

3

I failed to mention that I have a cronjob that reboots the Pi twice daily (I do that on all my Pi’s).

But if dump978 had locked up, it seems like I would not have had anything logged to the log file I was creating via the pipe to tee. If it made it to the file, wouldn’t that have meant that dump978 was working and had decoded 978 messages?

4

What do the logs indicate?

5

Which logs? I quoted the tail of the piaware log in my 1st post. Are there others or should I include more of the log?

6

Sorry, I forgot that dump1090-fa doesn’t seem to generate logs.

Try stopping dump979 and running this for a while
/usr/bin/rtl_sdr -f 978000000 -s 2083334 -g 48 - | /home/pi/dump978/dump978 | /home/pi/dump978/uat2text

I notice that the traffic really drops off during bad weather.

You may also want to try it with the FA filter in place. I know it really helped my situation.

7

Sure it does, they go via the systemd journal to syslog.

8

I can try that again…but based on my earlier test (described in my first post), I was logging traffic when I “tee” to a text file. I ran that file through uat2text and it was good decodes. But at the same time they were logging, for some reason those locations never made it to FA, based on the hour by hour stats and the date/time on the Data Feed line.

9

syslog (these same lines appear several times in syslog so I only included one set)



Jan 21 23:20:57 raspberrypi4 systemd[1]: Stopping dump1090 ADS-B receiver (FlightAware customization)...
Jan 21 23:20:58 raspberrypi4 dump1090-fa[1275]: Sat Jan 21 23:20:58 2017 UTC  Caught SIGTERM, shutting down..
Jan 21 23:20:58 raspberrypi4 dump1090-fa[1275]: Sat Jan 21 23:20:58 2017 UTC  Normal exit.
Jan 21 23:20:58 raspberrypi4 systemd[1]: Starting dump1090 ADS-B receiver (FlightAware customization)...
Jan 21 23:20:58 raspberrypi4 systemd[1]: Started dump1090 ADS-B receiver (FlightAware customization).
Jan 21 23:20:58 raspberrypi4 dump1090-fa[1469]: Sat Jan 21 23:20:58 2017 UTC  dump1090-fa v3.1.0 starting up.
Jan 21 23:20:58 raspberrypi4 dump1090-fa[1469]: Net-only mode, no RTL device or file open.
10

If dump1090-fa restarts (and piaware does try to restart it, as you see above, if it doesn’t see any traffic for a while) then the netcat will lose the connection, and there’s no mechanism in your pipeline there to restart it.

11

Ahhh! I did not realize that netcat wouldn’t reopen the connection when new data came through.

Is it possible to prevent piaware from stopping and restarting dump1090-fa? Lack of messages received for a time is not unusual for 978.

I just put back in a do while sleep loop into dump978-maint (from the original file, but is that just restarting it every 30 seconds or is looking for an exit status)…but if I can stop dump1090-fa from being restarted over and over just because of lack of messages I’d rather do that.

12

There is no mechanism in piaware to disable that behaviour short of taking away its sudo rules.

It’s not the right solution either, you should make your dump978 pipeline tolerate connection loss.

13

Any suggestions on how to do that? I tried switching from netcat to socat, but by itself it also terminates after a while. I tried adding ,fork to it’s command line and that just spawns new sessions every few seconds…I ended up with a page full in the process list after a minute or two.

I’m thinking if there is a way to detect that dump1090-fa has auto restarted, then a script can restart the rest of the utilities.

14

Wrap the pipeline in a script, don’t background it, have something else handle the restarts e.g. write a systemd service. Or if you like duct tape, wrap it in a shellscript loop.

I generally find something like “socat -u -t1 STDIN TCP-CONNECT:localhost:30001” to be more reliable than nc.

15

TBH someone needs to take dump978 and turn it into something more robust / fully featured if you are going to use it in anger.
It is very much a proof-of-concept at the moment, and since there is no UAT here I don’t really have much of a reason to improve it myself…

16

I would make crontab runt a shell script that checks if the connection is up by greping the netstat table.

if the connection is up - exit

if not - check if nc/socat is running by greping ps -a and if it is kill it

  • restart nc/socat

I’m sure there are more elegant ways to do this, but I like Duct tape, and it works :slight_smile:

/M

17

Thanks for the suggestions. I’ll work on which color duct tape I want to use when I have a little time. At the moment I went with the bailing wire approach and set a cronjob to reboot every 59 minutes (just before the dump1090-fa restart). :wink: At least this way if I see a worthwhile amount of 978 traffic, I know to spend the time working on a permanent solution.

18

Just for future reference by anyone who finds this thread…
I haven’t had time to work on it since my last posting, but it’s still humming along just fine with the 59 minute crontab reboots. I am getting only about 3 or 4 978MHz aircraft a day.