Raspberri PI randomly stops feed every few days

[foxhunter]

@dcoffin1

Got this message.

Looks like you’re trying to download the file to a folder where you do not have write access.

I am always using the home folder (Raspberry typically uses /home/pi) as long as i am not logged in via root directly.

The suggestion of using “sudo” in front of it might help, but then you need to remember where you have downloaded it to.

[dcoffin1]

Thanks. That worked.

[dcoffin1]

Thanks. All of the commands succeeded and its back up live. Is there a command to see if in fact the new code is running?

[abcd567]

@dcoffin1

Is there a command to see if in fact the new code is running?

What is output of this command?

apt-cache policy tcl-tls

 

[dcoffin1]

@abcd567

apt-cache policy tcl-tls

pi@piaware:~ $ apt-cache policy tcl-tls
tcl-tls:
Installed: 1.7.22-1+fa1~bpo10+1
Candidate: 1.7.22-1+fa1~bpo10+1
Version table:
*** 1.7.22-1+fa1~bpo10+1 100
100 /var/lib/dpkg/status
1.7.16-1+fa1 500
500 http://flightaware.com/adsb/piaware/files/packages buster/piaware armhf Packages
1.7.16-1 500
500 http://flightaware.com/mirror/raspbian/raspbian buster/main armhf Packages

[abcd567]

@dcoffin1

tcl-tls:
Installed: 1.7.22-1+fa1~bpo10+1

Above is what is installed and operating NOW. It is the one you downloaded and installed from my github site. (ver 1.7.22-1)

The previous one you had was ver 1.7.16-1

Hopefully the piaware crashes will stop now.

EDIT:
Please reboot the RPi if you have not rebooted it after installing the tcl-tls ver 1.7.22-1

 

[dcoffin1]

Thank you abcd567. Seems to be working smoothly so far.

[abcd567]

@dcoffin1

Thank you abcd567. Seems to be working smoothly so far

My pleasure. Actually thanks should go to @obj who gave me the tip to rebuild the tcl-tls package with the configure option --enable-ssl-fastpath, using latest upstream source code.

Please keep an eye on piaware and after few days post as to how it is going now. Your feedback will help others having similar problem.

 

[PeterHR]

on my P2, the usb-wifi dongle died … it’s now got a netgear one from spares box and has been happy for maybe six months.
That said, the usual problem people have is power / or the cable between the power brick and the Pi

[abcd567]

@PeterHR

the usual problem people have is power / or the cable between the power brick and the Pi

Here OP’s problem was NOT power supply as the problem continued even after changing power supply (please see quote from his post below). It was concluded by @obj that this was due to a bug in package tcl-tls. The OP replaced the original tcl-tls package by a rebuild bug-free package, and after that his problem has stopped.

@dcoffin1

I’ve been having random feed stoppages (site #42787) ever few days or weeks since last September. I’ve replaced the charger (CanaKit) with a new first thinking it might be failing but no joy. Then I replaced the Raspberry PI 3 board with a new Raspberry 4 board thinking it was failing but it still behaves the same way. This evening after about 3 days of use the feed stopped again. while this is all occurring, PiAware Skyware has continuously worked fine.

Each time it happens I shh into it and issue a systemctl restart piaware command to get it going again.

[dcoffin1]

I’ve had two more feeder losses. One was three days ago and another this morning. Here is the last few lines of the log file.

3 13:58:50 piaware piaware[1848]: mlat-client(8515): Server: 0.1 kB/s from server 0.0kB/s TCP to server 1.5kB/s UDP to server
Mar 3 13:58:50 piaware piaware[1848]: mlat-client(8515): Results: 46.6 positions/minute
Mar 3 13:58:50 piaware piaware[1848]: mlat-client(8515): Aircraft: 43 of 116 Mode S, 28 of 31 ADS-B used
Mar 3 14:00:51 piaware piaware[1848]: 5776888 msgs recv’d from dump1090-fa (9916 in last 5m); 5775819 msgs sent to FlightAware
Mar 3 14:05:51 piaware piaware[1848]: 5787894 msgs recv’d from dump1090-fa (11006 in last 5m); 5786825 msgs sent to FlightAware
Mar 3 14:10:51 piaware piaware[1848]: 5795750 msgs recv’d from dump1090-fa (7856 in last 5m); 5794681 msgs sent to FlightAware
Mar 3 14:13:51 piaware piaware[1848]: mlat-client(8515): Receiver status: connected
Mar 3 14:13:51 piaware piaware[1848]: mlat-client(8515): Server status: synchronized with 410 nearby receivers
Mar 3 14:13:51 piaware piaware[1848]: mlat-client(8515): Receiver: 308.6 msg/s received 106.3 msg/s processed (34%)
Mar 3 14:13:51 piaware piaware[1848]: mlat-client(8515): Server: 0.1 kB/s from server 0.0kB/s TCP to server 1.3kB/s UDP to server
Mar 3 14:13:51 piaware piaware[1848]: mlat-client(8515): Results: 53.8 positions/minute
Mar 3 14:13:51 piaware piaware[1848]: mlat-client(8515): Aircraft: 63 of 121 Mode S, 26 of 26 ADS-B used
Mar 3 14:15:51 piaware piaware[1848]: 5804876 msgs recv’d from dump1090-fa (9126 in last 5m); 5803807 msgs sent to FlightAware
Mar 3 14:20:51 piaware piaware[1848]: 5816244 msgs recv’d from dump1090-fa (11368 in last 5m); 5815175 msgs sent to FlightAware
Mar 3 14:25:51 piaware piaware[1848]: 5824611 msgs recv’d from dump1090-fa (8367 in last 5m); 5823542 msgs sent to FlightAware
Mar 3 14:28:51 piaware piaware[1848]: mlat-client(8515): Receiver status: connected
Mar 3 14:28:51 piaware piaware[1848]: mlat-client(8515): Server status: synchronized with 444 nearby receivers
Mar 3 14:28:51 piaware piaware[1848]: mlat-client(8515): Receiver: 329.9 msg/s received 115.4 msg/s processed (35%)
Mar 3 14:28:51 piaware piaware[1848]: mlat-client(8515): Server: 0.1 kB/s from server 0.0kB/s TCP to server 1.4kB/s UDP to server
Mar 3 14:28:51 piaware piaware[1848]: mlat-client(8515): Results: 40.5 positions/minute
Mar 3 14:28:51 piaware piaware[1848]: mlat-client(8515): Aircraft: 48 of 109 Mode S, 36 of 44 ADS-B used
Mar 3 14:30:51 piaware piaware[1848]: 5832420 msgs recv’d from dump1090-fa (7809 in last 5m); 5831351 msgs sent to FlightAware
Mar 3 14:35:51 piaware piaware[1848]: 5843523 msgs recv’d from dump1090-fa (11103 in last 5m); 5842454 msgs sent to FlightAware
Mar 3 14:40:51 piaware piaware[1848]: 5855523 msgs recv’d from dump1090-fa (12000 in last 5m); 5854454 msgs sent to FlightAware
Mar 3 14:43:51 piaware piaware[1848]: mlat-client(8515): Receiver status: connected
Mar 3 14:43:51 piaware piaware[1848]: mlat-client(8515): Server status: synchronized with 458 nearby receivers
Mar 3 14:43:51 piaware piaware[1848]: mlat-client(8515): Receiver: 460.8 msg/s received 151.6 msg/s processed (33%)
Mar 3 14:43:51 piaware piaware[1848]: mlat-client(8515): Server: 0.1 kB/s from server 0.0kB/s TCP to server 1.9kB/s UDP to server
Mar 3 14:43:51 piaware piaware[1848]: mlat-client(8515): Results: 35.1 positions/minute
Mar 3 14:43:51 piaware piaware[1848]: mlat-client(8515): Aircraft: 56 of 128 Mode S, 46 of 56 ADS-B used
Mar 3 14:45:51 piaware piaware[1848]: 5866491 msgs recv’d from dump1090-fa (10968 in last 5m); 5865422 msgs sent to FlightAware
Mar 3 14:50:51 piaware piaware[1848]: 5878202 msgs recv’d from dump1090-fa (11711 in last 5m); 5877133 msgs sent to FlightAware
Mar 3 14:55:51 piaware piaware[1848]: 5887354 msgs recv’d from dump1090-fa (9152 in last 5m); 5886285 msgs sent to FlightAware
Mar 3 14:58:52 piaware piaware[1848]: mlat-client(8515): Receiver status: connected
Mar 3 14:58:52 piaware piaware[1848]: mlat-client(8515): Server status: synchronized with 462 nearby receivers
Mar 3 14:58:52 piaware piaware[1848]: mlat-client(8515): Receiver: 562.0 msg/s received 171.6 msg/s processed (31%)
Mar 3 14:58:52 piaware piaware[1848]: mlat-client(8515): Server: 0.1 kB/s from server 0.0kB/s TCP to server 2.1kB/s UDP to server
Mar 3 14:58:52 piaware piaware[1848]: mlat-client(8515): Results: 18.1 positions/minute
Mar 3 14:58:52 piaware piaware[1848]: mlat-client(8515): Aircraft: 51 of 122 Mode S, 54 of 69 ADS-B used
Mar 3 15:00:51 piaware piaware[1848]: 5899464 msgs recv’d from dump1090-fa (12110 in last 5m); 5898395 msgs sent to FlightAware
Mar 3 15:05:51 piaware piaware[1848]: 5911002 msgs recv’d from dump1090-fa (11538 in last 5m); 5909933 msgs sent to FlightAware
Mar 3 15:10:51 piaware piaware[1848]: 5923911 msgs recv’d from dump1090-fa (12909 in last 5m); 5922842 msgs sent to FlightAware
Mar 3 15:13:52 piaware piaware[1848]: mlat-client(8515): Receiver status: connected
Mar 3 15:13:52 piaware piaware[1848]: mlat-client(8515): Server status: synchronized with 447 nearby receivers
Mar 3 15:13:52 piaware piaware[1848]: mlat-client(8515): Receiver: 695.7 msg/s received 190.6 msg/s processed (27%)
Mar 3 15:13:52 piaware piaware[1848]: mlat-client(8515): Server: 0.1 kB/s from server 0.0kB/s TCP to server 2.1kB/s UDP to server
Mar 3 15:13:52 piaware piaware[1848]: mlat-client(8515): Results: 28.5 positions/minute
Mar 3 15:13:52 piaware piaware[1848]: mlat-client(8515): Aircraft: 86 of 137 Mode S, 65 of 89 ADS-B used
Mar 3 15:15:51 piaware piaware[1848]: 5935428 msgs recv’d from dump1090-fa (11517 in last 5m); 5934359 msgs sent to FlightAware
Mar 3 15:20:51 piaware piaware[1848]: 5948742 msgs recv’d from dump1090-fa (13314 in last 5m); 5947673 msgs sent to FlightAware
Mar 3 15:25:51 piaware piaware[1848]: 5961424 msgs recv’d from dump1090-fa (12682 in last 5m); 5960355 msgs sent to FlightAware
Mar 3 15:28:52 piaware piaware[1848]: mlat-client(8515): Receiver status: connected
Mar 3 15:28:52 piaware piaware[1848]: mlat-client(8515): Server status: synchronized with 460 nearby receivers
Mar 3 15:28:52 piaware piaware[1848]: mlat-client(8515): Receiver: 723.2 msg/s received 205.4 msg/s processed (28%)
Mar 3 15:28:52 piaware piaware[1848]: mlat-client(8515): Server: 0.1 kB/s from server 0.0kB/s TCP to server 2.1kB/s UDP to server
Mar 3 15:28:52 piaware piaware[1848]: mlat-client(8515): Results: 31.0 positions/minute
Mar 3 15:28:52 piaware piaware[1848]: mlat-client(8515): Aircraft: 52 of 136 Mode S, 70 of 89 ADS-B used
Mar 3 15:30:51 piaware piaware[1848]: 5973623 msgs recv’d from dump1090-fa (12199 in last 5m); 5972554 msgs sent to FlightAware
Mar 3 15:34:21 piaware piaware[1848]: Lost connection to adept server at piaware.flightaware.com/1200: server closed connection

[dcoffin1]

It looks like I’m still having the same problem. 3 disconnects in the last three days.

[abcd567]

@dcoffin1

It looks like I’m still having the same problem. 3 disconnects in the last three days

So the cause is not tcl-tls, as you have already installed tcltls-rebuild.

May be something to do with network or FA servers. The FA staff @obj or @eric1tran may suggest a solution.

[dcoffin1]

I have another Raspberry feeder on my network providing 978 Mhz feed and it has not had and disconnect at all. Both are wired connections.

[obj]

When it’s in the bad state, can you gather the output of:

systemctl status piaware

ps auwxf | grep piaware

top few lines of top (I’m mostly interested in if piaware is taking CPU or not)

contents of /var/log/syslog for the time range immediately before/after the last line logged in /var/log/piaware.log (let’s say at least +/- 5 mins)

@abcd567

May be something to do with network or FA servers.

Network issues between the feeder and the server are relatively common and expected. The issue here is that piaware is not attempting to reconnect.

[dcoffin1]

I restarted it about 4:00 PST.

(edit) I’ll post the results when it stops again.

[abcd567]

@obj

The issue here is that piaware is not attempting to reconnect

What was the issue here?
I tried to fix it by rebooting, but Pi failed to reboot. I fixed it by pulling out and replugging wall wart.

@abcd567

pi@raspberrypi:~ $ journalctl -u dump1090-fa -e
Feb 24 14:19:40 raspberrypi systemd[1]: Stopped dump1090 ADS-B receiver (FlightAware customization).
Feb 24 14:19:40 raspberrypi systemd[1]: Started dump1090 ADS-B receiver (FlightAware customization).
Feb 24 14:19:40 raspberrypi dump1090-fa[3426]: Thu Feb 24 14:19:40 2022 EST  dump1090-fa 7.1 starting up.
Feb 24 14:19:40 raspberrypi dump1090-fa[3426]: rtlsdr: error querying device #0: Permission denied
Feb 24 14:19:40 raspberrypi systemd[1]: dump1090-fa.service: Main process exited, code=exited, status=1/FAILURE
Feb 24 14:19:40 raspberrypi systemd[1]: dump1090-fa.service: Failed with result 'exit-code'.
Feb 24 14:20:05 raspberrypi systemd[1]: Stopped dump1090 ADS-B receiver (FlightAware customization).
-- Boot 609aaa05f7e040e69fe88678213b1779 --
Feb 24 14:20:17 raspberrypi systemd[1]: Started dump1090 ADS-B receiver (FlightAware customization).
Feb 24 14:20:18 raspberrypi dump1090-fa[530]: Thu Feb 24 14:20:17 2022 EST  dump1090-fa 7.1 starting up.
Feb 24 14:20:18 raspberrypi dump1090-fa[530]: rtlsdr: using device #0: Generic RTL2832U OEM (Realtek, RTL2838UHIDIR, SN 00001>
Feb 24 14:20:18 raspberrypi dump1090-fa[530]: Detached kernel driver
Feb 24 14:20:18 raspberrypi dump1090-fa[530]: Found Rafael Micro R820T tuner
Feb 24 14:20:18 raspberrypi dump1090-fa[530]: rtlsdr: tuner gain set to 44.5 dB (gain step 26)
Feb 24 14:20:18 raspberrypi dump1090-fa[530]: Allocating 4 zero-copy buffers
Feb 26 22:54:17 raspberrypi dump1090-fa[530]: Sat Feb 26 22:54:17 2022 EST  Caught SIGTERM, shutting down..
Feb 26 22:54:17 raspberrypi systemd[1]: Stopping dump1090 ADS-B receiver (FlightAware customization)...
Feb 26 22:54:17 raspberrypi dump1090-fa[530]: Sat Feb 26 22:54:17 2022 EST  Waiting for receive thread termination
Feb 26 22:54:18 raspberrypi dump1090-fa[530]: Reattaching kernel driver failed!
Feb 26 22:54:18 raspberrypi dump1090-fa[530]: Sat Feb 26 22:54:18 2022 EST  Normal exit.
Feb 26 22:54:18 raspberrypi systemd[1]: dump1090-fa.service: Succeeded.
Feb 26 22:54:18 raspberrypi systemd[1]: Stopped dump1090 ADS-B receiver (FlightAware customization).
Feb 26 22:54:18 raspberrypi systemd[1]: dump1090-fa.service: Consumed 13h 32min 41.623s CPU time.

[dcoffin1]

Is there a convenient command for downloading logs such as piaware.log to a PC other than using cat and cut & paste the screen? scp doesn’t seem to be cooperating for me.

[obj]

@abcd567

What was the issue here?

In your linked issue? Not enough information to tell. If the reboot failed (and since you got some kernel errors), that implies a deeper problem.

[obj]

@dcoffin1

Is there a convenient command for downloading logs such as piaware.log to a PC other than using cat and cut & paste the screen? scp doesn’t seem to be cooperating for me.

You could try sftp as an alternative.