PiAware on Pi3 crashes

FlightAware ADS-B Flight Tracking
1

I have PiAware 3.5.3 installed under Debian on a Pi3 connected via Ethernet and running on a TeckNet Portable Charger.
Antenna is in a poor location as TNT have not delivered my mast yet.

I experience random crashes most days.
From what I can make out the Pi is loosing connection to the server.
Is this correct?
I’m on FTTC at about 7Mb/s upload, so don’t think it could be that.
A new Ethernet cable is now in use, so I will see if that makes any difference.

Jul 13 10:50:17 raspberrypi piaware[455]: 46198 msgs recv’d from dump1090-fa (381 in last 5m); 46042 msgs sent to FlightAware

Jul 13 10:55:16 raspberrypi piaware[455]: 46605 msgs recv’d from dump1090-fa (407 in last 5m); 46449 msgs sent to FlightAware

Jul 13 10:56:25 raspberrypi piaware[455]: mlat-client(1694): Receiver status: connected

Jul 13 10:56:25 raspberrypi piaware[455]: mlat-client(1694): Server status: synchronized with 203 nearby receivers

Jul 13 10:56:25 raspberrypi piaware[455]: mlat-client(1694): Receiver: 108.2 msg/s received 22.0 msg/s processed (20%)

Jul 13 10:56:25 raspberrypi piaware[455]: mlat-client(1694): Server: 0.1 kB/s from server 0.0kB/s TCP to server 0.2kB/s UDP to server

Jul 13 10:56:25 raspberrypi piaware[455]: mlat-client(1694): Results: 25.2 positions/minute

Jul 13 10:56:25 raspberrypi piaware[455]: mlat-client(1694): Aircraft: 4 of 8 Mode S, 13 of 14 ADS-B used

Jul 13 11:00:17 raspberrypi piaware[455]: 47046 msgs recv’d from dump1090-fa (441 in last 5m); 46890 msgs sent to FlightAware

Jul 13 11:05:17 raspberrypi piaware[455]: 47448 msgs recv’d from dump1090-fa (402 in last 5m); 47292 msgs sent to FlightAware

Jul 13 11:10:18 raspberrypi piaware[455]: 47891 msgs recv’d from dump1090-fa (443 in last 5m); 47735 msgs sent to FlightAware

Jul 13 11:11:30 raspberrypi piaware[455]: mlat-client(1694): Receiver status: connected

Jul 13 11:11:30 raspberrypi piaware[455]: mlat-client(1694): Server status: synchronized with 224 nearby receivers

Jul 13 11:11:31 raspberrypi piaware[455]: mlat-client(1694): Receiver: 118.8 msg/s received 25.1 msg/s processed (21%)

Jul 13 11:11:33 raspberrypi piaware[455]: mlat-client(1694): Server: 0.1 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server

Jul 13 11:11:34 raspberrypi piaware[455]: mlat-client(1694): Results: 22.3 positions/minute

Jul 13 11:11:35 raspberrypi piaware[455]: mlat-client(1694): Aircraft: 5 of 8 Mode S, 13 of 14 ADS-B used

Jul 13 11:15:18 raspberrypi piaware[455]: 48246 msgs recv’d from dump1090-fa (355 in last 5m); 48090 msgs sent to FlightAware

Jul 13 11:20:18 raspberrypi piaware[455]: 48673 msgs recv’d from dump1090-fa (427 in last 5m); 48517 msgs sent to FlightAware

Jul 13 11:25:19 raspberrypi piaware[455]: 48883 msgs recv’d from dump1090-fa (210 in last 5m); 48727 msgs sent to FlightAware

Jul 13 11:26:30 raspberrypi piaware[455]: mlat-client(1694): Receiver status: connected

Jul 13 11:26:30 raspberrypi piaware[455]: mlat-client(1694): Server status: synchronized with 73 nearby receivers

Jul 13 11:26:32 raspberrypi piaware[455]: mlat-client(1694): Receiver: 64.2 msg/s received 15.5 msg/s processed (24%)

Jul 13 11:26:33 raspberrypi piaware[455]: mlat-client(1694): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.2kB/s UDP to server

Jul 13 11:26:35 raspberrypi piaware[455]: mlat-client(1694): Results: 11.7 positions/minute

Jul 13 11:26:36 raspberrypi piaware[455]: mlat-client(1694): Aircraft: 7 of 9 Mode S, 6 of 7 ADS-B used

Jul 13 11:30:18 raspberrypi piaware[455]: 49165 msgs recv’d from dump1090-fa (282 in last 5m); 49009 msgs sent to FlightAware

Jul 13 11:35:19 raspberrypi piaware[455]: 49483 msgs recv’d from dump1090-fa (318 in last 5m); 49327 msgs sent to FlightAware

Jul 13 11:35:54 raspberrypi piaware[455]: NOTICE from adept server: 13% of multilateration messages (UDP) are not reaching the server - check your network?

Jul 13 11:40:18 raspberrypi piaware[455]: 49900 msgs recv’d from dump1090-fa (417 in last 5m); 49744 msgs sent to FlightAware

Jul 13 11:41:29 raspberrypi piaware[455]: mlat-client(1694): Receiver status: connected

Jul 13 11:41:32 raspberrypi piaware[455]: mlat-client(1694): Server status: synchronized with 110 nearby receivers

Jul 13 11:41:33 raspberrypi piaware[455]: mlat-client(1694): Receiver: 99.8 msg/s received 20.9 msg/s processed (21%)

Jul 13 11:41:35 raspberrypi piaware[455]: mlat-client(1694): Server: 0.1 kB/s from server 0.0kB/s TCP to server 0.2kB/s UDP to server

Jul 13 11:41:36 raspberrypi piaware[455]: mlat-client(1694): Results: 25.8 positions/minute

Jul 13 11:41:38 raspberrypi piaware[455]: mlat-client(1694): Aircraft: 6 of 12 Mode S, 12 of 13 ADS-B used

Jul 13 11:45:19 raspberrypi piaware[455]: 50353 msgs recv’d from dump1090-fa (453 in last 5m); 50197 msgs sent to FlightAware

Jul 13 11:50:21 raspberrypi piaware[455]: 50810 msgs recv’d from dump1090-fa (457 in last 5m); 50654 msgs sent to FlightAware

Jul 13 11:55:19 raspberrypi piaware[455]: 51223 msgs recv’d from dump1090-fa (413 in last 5m); 51067 msgs sent to FlightAware

Jul 13 11:56:28 raspberrypi piaware[455]: mlat-client(1694): Receiver status: connected

Jul 13 11:56:28 raspberrypi piaware[455]: mlat-client(1694): Server status: synchronized with 223 nearby receivers

Jul 13 11:56:28 raspberrypi piaware[455]: mlat-client(1694): Receiver: 114.1 msg/s received 24.4 msg/s processed (21%)

Jul 13 11:56:28 raspberrypi piaware[455]: mlat-client(1694): Server: 0.1 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server

Jul 13 11:56:28 raspberrypi piaware[455]: mlat-client(1694): Results: 19.9 positions/minute

Jul 13 11:56:28 raspberrypi piaware[455]: mlat-client(1694): Aircraft: 8 of 10 Mode S, 13 of 15 ADS-B used

Jul 13 12:00:20 raspberrypi piaware[455]: 51671 msgs recv’d from dump1090-fa (448 in last 5m); 51515 msgs sent to FlightAware

Jul 13 12:05:21 raspberrypi piaware[455]: 52136 msgs recv’d from dump1090-fa (465 in last 5m); 51980 msgs sent to FlightAware

Jul 13 12:06:00 raspberrypi piaware[455]: NOTICE from adept server: 17% of multilateration messages (UDP) are not reaching the server - check your network?

Jul 13 12:10:24 raspberrypi piaware[455]: 52585 msgs recv’d from dump1090-fa (449 in last 5m); 52429 msgs sent to FlightAware

Jul 13 12:11:35 raspberrypi piaware[455]: mlat-client(1694): Receiver status: connected
Jul 13 12:11:36 raspberrypi piaware[455]: mlat-client(1694): Server status: synchronized with 112 nearby receivers

Jul 13 12:11:36 raspberrypi piaware[455]: mlat-client(1694): Receiver: 103.0 msg/s received 24.4 msg/s processed (24%)
Jul 13 12:11:36 raspberrypi piaware[455]: mlat-client(1694): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.2kB/s UDP to server
Jul 13 12:11:36 raspberrypi piaware[455]: mlat-client(1694): Results: 15.7 positions/minute

Jul 13 12:11:36 raspberrypi piaware[455]: mlat-client(1694): Aircraft: 8 of 11 Mode S, 8 of 10 ADS-B used

Jul 13 12:15:23 raspberrypi piaware[455]: 52909 msgs recv’d from dump1090-fa (324 in last 5m); 52753 msgs sent to FlightAware

Jul 13 12:18:19 raspberrypi piaware[455]: timed out waiting for alive message from FlightAware, reconnecting…

Jul 13 12:18:24 raspberrypi piaware[455]: multilateration data no longer required, disabling mlat client
Jul 13 12:18:41 raspberrypi piaware[455]: fa-mlat-client exited with

Jul 13 12:18:41 raspberrypi piaware[455]: reconnecting in 54 seconds…

Jul 13 12:18:41 raspberrypi piaware[455]: mlat-client(1694): Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data

Jul 13 12:18:41 raspberrypi piaware[455]: mlat-client(1694): Exiting on connection loss

Jul 13 12:19:35 raspberrypi piaware[455]: Connecting to FlightAware adept server at piaware.flightaware.com/1200

Jul 13 12:19:36 raspberrypi piaware[455]: Connection with adept server at piaware.flightaware.com/1200 established

Jul 13 12:19:40 raspberrypi piaware[455]: TLS handshake with adept server at piaware.flightaware.com/1200 completed

Jul 13 12:19:41 raspberrypi piaware[455]: FlightAware server certificate validated

Jul 13 12:19:41 raspberrypi piaware[455]: encrypted session established with FlightAware

Jul 13 12:20:15 raspberrypi piaware[455]: data isn’t making it to FlightAware, reconnecting…

Jul 13 12:20:16 raspberrypi piaware[455]: reconnecting in 72 seconds…

Jul 13 12:20:45 raspberrypi piaware[455]: 53324 msgs recv’d from dump1090-fa (415 in last 5m); 52944 msgs sent to FlightAware

Jul 13 12:21:29 raspberrypi piaware[455]: Connecting to FlightAware adept server at piaware.flightaware.com/1200

Jul 13 12:21:31 raspberrypi piaware[455]: Connection with adept server at piaware.flightaware.com/1200 established

Jul 13 12:21:41 raspberrypi piaware[455]: TLS handshake with adept server at piaware.flightaware.com/1200 completed

Jul 13 12:21:42 raspberrypi piaware[455]: FlightAware server certificate validated

Jul 13 12:21:42 raspberrypi piaware[455]: encrypted session established with FlightAware

Jul 13 12:22:21 raspberrypi piaware[455]: data isn’t making it to FlightAware, reconnecting…

Jul 13 12:22:22 raspberrypi piaware[455]: reconnecting in 62 seconds…

Jul 13 12:23:24 raspberrypi piaware[455]: Connecting to FlightAware adept server at 70.42.6.198/1200

Jul 13 12:23:25 raspberrypi piaware[455]: Connection with adept server at 70.42.6.198/1200 established

Jul 13 12:23:28 raspberrypi piaware[455]: TLS handshake with adept server at 70.42.6.198/1200 completed

Jul 13 12:23:28 raspberrypi piaware[455]: FlightAware server certificate validated

Jul 13 12:23:28 raspberrypi piaware[455]: encrypted session established with FlightAware

Jul 13 12:24:04 raspberrypi piaware[455]: Login attempt with adept server at 70.42.6.198/1200 timed out

Jul 13 12:24:06 raspberrypi piaware[455]: reconnecting in 68 seconds…

Jul 13 12:25:14 raspberrypi piaware[455]: Connecting to FlightAware adept server at 70.42.6.191/1200
Jul 13 12:25:14 raspberrypi piaware[455]: Connection with adept server at 70.42.6.191/1200 established
Jul 13 12:25:20 raspberrypi piaware[455]: TLS handshake with adept server at 70.42.6.191/1200 completed

Jul 13 12:25:20 raspberrypi piaware[455]: FlightAware server certificate validated

Jul 13 12:25:20 raspberrypi piaware[455]: encrypted session established with FlightAware

Jul 13 12:25:58 raspberrypi piaware[455]: data isn’t making it to FlightAware, reconnecting…

Jul 13 12:25:59 raspberrypi piaware[455]: reconnecting in 57 seconds…

2

It worked fine until 12:18. Is it working now? Did you change anything at that time?

If you see it not working can you try to connect to it locally at that time?
That way you can check where exactly the problem is.

3

Hello wiedehopf
It packed up when I was out, which is what it normally does.
I could not connect to it locally by the HDMI and USB. I don’t SSH into it as it is next to the TV at present.
The only way I can get it working again is to power cycle the Pi.

4

So while it’s not working can you check the local Skyview webpage? I’m not sure how you check HDMI and USB i only use my pi via ssh. Do you mean normally the pi is putting out a picture and when the problem happens you don’t even get that? May i ask what this Portable Charger is you were talking about?

Just to see if there is local connectivity. You could also leave your main computer powered while you are away an keep a tool like mtr running to see if there are any lost packages, which there should not be any.

Where do you get the log? Off the flightaware stats page?

5

Thank you
When it crashes next I will check the local page as suggested.
Yes correct, I normally connect the Pi to my TV and when it falls over there is normally no video.
I will find out how to run WinMTR as it reports 100% loss from piaware.flightaware.com
The log is from the Pi after power cycling.

6

|------------------------------------------------------------------------------------------|
| WinMTR statistics |

Host - % Sent Recv Best Avrg Wrst Last
home.gateway.home.gateway - 0 96 96 0 0 2 0
410.xe-2-2-0.central10.pcn-bng04.plus.net - 0 96 96 10 10 15 10
411.be8.pcn-ir01.plus.net - 0 96 96 10 11 14 11
195.99.125.136 - 0 96 96 10 11 19 11
peer6-hu0-6-0-0.telehouse.ukcore.bt.net - 0 96 96 10 11 20 12
t2c3-et-8-1-0-0.uk-lon1.eu.bt.net - 0 96 96 10 12 40 25
hu0-6-0-4.ccr22.lon01.atlas.cogentco.com - 0 96 96 11 11 19 12
be2870.ccr41.lon13.atlas.cogentco.com - 0 96 96 10 11 20 12
be2317.ccr41.jfk02.atlas.cogentco.com - 0 96 96 79 80 84 80
be2806.ccr41.dca01.atlas.cogentco.com - 0 96 96 91 92 96 92
be2112.ccr41.atl01.atlas.cogentco.com - 0 96 96 102 103 109 103
be2687.ccr41.iah01.atlas.cogentco.com - 2 92 91 0 117 122 117
te0-0-0-3.agr13.iah01.atlas.cogentco.com - 2 92 91 113 113 117 113
te0-0-2-1.nr11.b000991-4.iah01.atlas.cogentco.com - 2 92 91 0 117 123 117
38.142.28.186 - 2 92 91 0 118 122 118
border3.ae1-bbnet1.hou.pnap.net - 2 92 91 0 121 134 121
edge1.ae1-edgenet.hou.pnap.net - 2 92 91 112 114 156 112
superconnect-7.edge1.hou.pnap.net - 3 88 86 0 121 125 121
No response from host - 100 19 0 0 0 0 0
piaware.flightaware.com - 0 96 96 112 112 116 113
________________________________________________ ______ ______ ______ ______ ______ ______
WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
7

That’s actually not 100% loss that’s going just fine. The loss is an intermediate hop and it’s perfectly normal some routers on the way don’t respond to such traceroutes :wink:

And i meant you should point it at your RasPi and just leave it running while you are away.
That way you could see if the failure to connect is a problem locally.

If it’s not a local connection issue it becomes even more complicated. Then leaving it running pointed at flightaware might shed some light, but please if you point it towards the internet and leave it running for a longer period, tweak the settings so it only does it’s thing every 5 or 10 seconds :slight_smile:

Generally if the problem only exists when you are away you are gonna need to leave it running as i described while you are away so we can hope to observe the problem.
(Yes such intermittent problems are a pain in the butt!)

1 Like
8

oh now i see you have a log after power cycling?

after power cycling would you mind typing the command “journalctl -b -1”?
That should give you the logs from the previous start until shutdown if they are available which might not be the case.

I’m a little confused as your RPI seems to first not connect to flightaware and then it just dies? The need to powercycle confuses me a bit right now. We’ll see :slight_smile:

9

Err, sorry.
Brain fade moment.
Yes only at intervals, not a bombardment.

10

I had a similar situation with my system at the beginning of the year but it resolved itself and I have no idea what the problem was.

I was able to still ssh into the device but NOT via network, it was using a SUB to serial adapter and I found Pi system still working but network failed. Could not appear to just restart network, had to do a reboot.

Eventually resolved itself.

Try a different sd card.

Geffers

11

Crashed again.
Network still up but live data gives “Problem fetching data from dump1090”
so no data on Flightaware.
Will try setting up SSH.
I’m using an Extreme SD card that is practically new, and don’t have a spare high quality one.
Power cycle time again.

12

try apt-get update and then apt-get dist-upgrade
maybe there is some issue with a driver that has been resolved in newer software.

13

Hello Wiedehopf
Still working when I returned just now.

I have updated again but it was only cups and chromium that were out of date.

SSH now working, so hopefully will learn more if it stops communicating again.

I may have scared it into working properly, time will tell.

14

Not reporting locally or via flightAware.
SSH not working.
responding to WinMTR no packets lost.
Power-fail time

15

so to have it respond to pings but not to ssh is very strange, are you sure you interpreted winmtr correctly?

my only guess would be that it’s running out of memory.

do an apt-get install htop and then run htop to check how your memory is doing.

alternatively you can do “df -h” and check the first column for “tmpfs” and then check it’s use.
but it could also be a program that hogs all the memory, htop or even top should be a better choice.

edit: oh and please post the ssh error message, maybe it’s useful

16

Did it again, no SSH or connection but ping still responded (I forgot to restart Win MTR, so only running a short while for this screenshot).

htop
htop1903×1080 191 KB
WinMTR
WinMTR1920×1050 31 KB

I will try to capture the SSH screen when it goes wrong next time.

17

Now my RasPi is throwing fits. Ping works but ssh times out or is very sporadic. My feeders still seem to work. Maybe there is something wrong with the current RasPi software.

Edit: Nevermind i believe it was some really bad wifi interference. (Doesn’t happen very often)

18

No dropped pings after 24 hours, despite not sending data for the last two hours.

PUTTY
PUTTY999×579 7.33 KB
Skyview
Skyview1916×937 706 KB

Nothing usefulll from PUTTY, just hangs there.

19

Pointing out the obvious here… Is dump1090 running?

20

sudo systemctl status dump1090-fa -l