Raspberri PI randomly stops feed every few days

FlightAware ADS-B Flight Tracking
1

I’ve been having random feed stoppages (site #42787) ever few days or weeks since last September. I’ve replaced the charger (CanaKit) with a new first thinking it might be failing but no joy. Then I replaced the Raspberry PI 3 board with a new Raspberry 4 board thinking it was failing but it still behaves the same way. This evening after about 3 days of use the feed stopped again. while this is all occurring, PiAware Skyware has continuously worked fine.

Each time it happens I shh into it and issue a systemctl restart piaware command to get it going again.

I just bought a replacement Blue stick but I haven’t installed it yet since I’m not sure that will fix it.

Any ideas what to do?

2

So the devices loses the network connection or is it accessible all the time?
Do you operate other feeders on it as well? Are they disconnected too?

Without some logs it’s hard to tell

3

Given that I still see activity in SkyAware I not sure it is a network issue. I do have another Raspberry with Piaware for tracking 978 MHz signals and it does not have this issue. Both are on the same wired network.

When I replaced the Raspberry PI for the site 42787, I had to associate it with the config lines with the feeder site and id.

What kind of logs would help?

4

/var/log/piaware.log when it’s in the failed state.

2 Likes
5

You mean your local installation? That does not give an indication about network. It only shows that it can be reached locally.

you can also utilize these commands and post results here once it failed again:

journalctl -xe|grep piaware
journalctl -xe|grep dump1090-fa

Or simply what @obj mentioned

2 Likes
6

just saw this after I restarted. Will do it again before I do a restart next time.

In the meantime I’m copying the instance where the connection failed below from the piaware.log in case this is of help:

Feb 16 08:07:53 piaware piaware[15433]: 352296 msgs recv’d from dump1090-fa (5720 in last 5m); 352296 msgs sent to FlightAware
Feb 16 08:12:36 piaware piaware[15433]: NOTICE from adept server: 13% of multilateration messages (UDP) are not reaching the server - check your network?
Feb 16 08:12:53 piaware piaware[15433]: 361912 msgs recv’d from dump1090-fa (9616 in last 5m); 361912 msgs sent to FlightAware
Feb 16 08:17:53 piaware piaware[15433]: 368920 msgs recv’d from dump1090-fa (7008 in last 5m); 368920 msgs sent to FlightAware
Feb 16 08:18:21 piaware piaware[15433]: Lost connection to adept server at piaware.flightaware.com/1200: server closed connection
Feb 16 15:38:58 piaware piaware[15375]: creating pidfile /run/piaware/piaware.pid

7

Is it correct to assume that the feed outage was at around 08:18 UTC and you restarted manually at around 15:38 UTC?

(I suspect this is a reoccurance of an old tcl-tls bug that I hoped had been fixed – apparently not)

1 Like
8

Would building and installing tcl-tls from source code linked below overcome this bug?

https://github.com/flightaware/tcltls-rebuild

9

Yes/ That was the time of the outage (8:18) and manual restart (15:38).

10

The idea is right but that specific package needs to be updated to be based on the newer upstream tcl-tls package.

11

[TclTLS: All Tickets]

LAST FEW OPEN TICKETS
Is 2nd last (Server accept invoked before handshake completed) related to Piaware?

# mtime type status subsystem title
0c51e2e5de 2021-01-14 13:03:22 Feature Request Closed tcltls patch to obtain negotiated SSL/TLS protocol version
37bbdb9fb2 2021-01-27 08:17:35 Build Problem Open make install does not work when configure was called without --prefix
520a0fa76b 2021-03-30 21:01:12 Build Problem Open unable to install on CentOS 7 x64
2c7b748796 2021-05-12 10:12:24 Feature Request Open Control TLS socket server errors
0469a61ebc 2021-05-12 10:18:41 Documentation Open handshake failed: certificate verify failed
e1f9a21c67 2021-08-26 12:24:56 Feature Request Open Add support for ALPN (required for HTTP/2)
8de7f5aa07 2021-08-27 09:17:04 Feature Request Open Enable debugging with wireshark
305ee10b86 2021-09-29 08:42:00 Feature Request Open support of openssl options in tls:init
ad8604520e 2021-10-11 04:44:50 Code Defect Open Server accept invoked before handshake completed
316976aff3 2022-01-07 07:17:40 Build Problem Open rules-ext.vc missing from win directory
12

No, it’s this guy: 6dd5588df6

1 Like
14

@obj

Revised procedure.

tcl-tls ver 1.7.22 rebuild

For Bullseye: tcl-tls_1.7.22-1+fa1_armhf.deb
For Buster: tcl-tls_1.7.22-1+fa1~bpo10+1_armhf.deb

 

pi@raspberrypi:~ $ sudo apt install devscripts debhelper libssl-dev tcl-dev chrpath 

pi@raspberrypi:~ $ git clone https://github.com/flightaware/tcltls-rebuild
pi@raspberrypi:~ $ cp -r tcltls-rebuild tcltls-rebuild-orig
pi@raspberrypi:~ $ cd tcltls-rebuild
pi@raspberrypi:~/tcltls-rebuild $ sudo wget https://core.tcl-lang.org/tcltls/uv/tcltls-1.7.22.tar.gz 
pi@raspberrypi:~/tcltls-rebuild $ sudo tar -xvpf tcltls-1.7.22.tar.gz  

pi@raspberrypi:~/tcltls-rebuild $ sudo cp -r tcltls-1.7.16/debian tcltls-1.7.22/

## CHECKED --enable-ssl-fastpath is included in debian/rules:
pi@raspberrypi:~/tcltls-rebuild $ grep -rnw tcltls-1.7.22/debian -e '--enable-ssl-fastpath'
tcltls-1.7.22/debian/rules:16:                       --enable-ssl-fastpath
tcltls-1.7.22/debian/changelog:3:  * Rebuild with --enable-ssl-fastpath workaround

pi@raspberrypi:~/tcltls-rebuild $ sudo rm -rf tcltls-1.7.16
pi@raspberrypi:~/tcltls-rebuild $ sudo rm tcltls_1.7.16.orig.tar.gz
pi@raspberrypi:~/tcltls-rebuild $ sudo rm tcltls_1.7.16-1.debian.tar.xz  
pi@raspberrypi:~/tcltls-rebuild $ sudo rm tcltls_1.7.16-1.dsc

pi@raspberrypi:~/tcltls-rebuild $ ls
Jenkinsfile  prepare-build.sh  README.md  tcltls-1.7.22  tcltls-1.7.22.tar.gz

pi@raspberrypi:~/tcltls-rebuild $ sudo sed -i 's/tcltls (1.7.16-1+fa1) stable/tcltls (1.7.22-1+fa1) stable/' tcltls-1.7.22/debian/changelog
pi@raspberrypi:~/tcltls-rebuild $ sudo sed -i 's/tcltls (1.7.16-1) unstable/tcltls (1.7.22-1) stable/' tcltls-1.7.22/debian/changelog

 

pi@raspberrypi:~/tcltls-rebuild $ sudo sed -i 's/tcltls-1.7.16/tcltls-1.7.22/' prepare-build.sh 
pi@raspberrypi:~/tcltls-rebuild $ sudo sed -i "/buster/a\        dch --local ~bpo10+ --distribution buster-backports --force-distribution \"Automated backport to buster\"\n\        ;;\n\    bullseye)\ " prepare-build.sh       

## PACKAGE FOR BULLSEYE
pi@raspberrypi:~/tcltls-rebuild $ sudo bash prepare-build.sh bullseye
pi@raspberrypi:~/tcltls-rebuild $ cd package-bullseye
pi@raspberrypi:~/tcltls-rebuild/package-bubullseye $ sudo dpkg-buildpackage -b --no-sign

## PACKAGE FOR BUSTER
pi@raspberrypi:~/tcltls-rebuild $ sudo bash prepare-build.sh buster
pi@raspberrypi:~/tcltls-rebuild $ cd package-buster
pi@raspberrypi:~/tcltls-rebuild/package-bubullseye $ sudo dpkg-buildpackage -b --no-sign

pi@raspberrypi:~/tcltls-rebuild/package-bullseye $ cd ../

pi@raspberrypi:~/tcltls-rebuild $ ls
Jenkinsfile       tcltls-1.7.22                                tcltls_1.7.22-1+fa1~bpo10+1_armhf.changes
package-bullseye  tcltls_1.7.22-1+fa1_armhf.buildinfo          tcl-tls_1.7.22-1+fa1~bpo10+1_armhf.deb
package-buster    tcltls_1.7.22-1+fa1_armhf.changes            tcltls-1.7.22.tar.gz
prepare-build.sh  tcl-tls_1.7.22-1+fa1_armhf.deb               tcl-tls-dbgsym_1.7.22-1+fa1_armhf.deb
README.md         tcltls_1.7.22-1+fa1~bpo10+1_armhf.buildinfo  tcl-tls-dbgsym_1.7.22-1+fa1~bpo10+1_armhf.deb

 

 

1 Like
15

@dcoffin1

If you trust me, and are ready to face (remote) chance of re-imaging your microSD card if something breaks down, then you can install the latest version (1.7.22) of tcl-tls package. This package I have rebuilt using Flightaware source-code with workaround for the bug mentioned by @obj, after upgrading it from ver 1.7.16 to 1.7.22,

## First install needed dependency package
sudo apt install libtcl8.6  


## If your OS is Bullseye:
wget https://github.com/abcd567a/tcltls-rebuild/releases/download/v1/tcl-tls_1.7.22-1+fa1_armhf.deb   

sudo dpkg -i tcl-tls_1.7.22-1+fa1_armhf.deb  

sudo apt-mark hold tcl-tls  

sudo reboot   


## if your OS is Buster
wget https://github.com/abcd567a/tcltls-rebuild/releases/download/v1/tcl-tls_1.7.22-1+fa1.bpo10+1_armhf.deb  

sudo dpkg -i tcl-tls_1.7.22-1+fa1.bpo10+1_armhf.deb   

sudo apt-mark hold tcl-tls  

sudo reboot
16

Just stopped feeding again today.
This is the log

Feb 22 17:18:18 piaware piaware[530]: mlat-client(6159): Aircraft: 76 of 122 Mode S, 75 of 111 ADS-B used
Feb 22 17:18:36 piaware piaware[530]: 10490037 msgs recv’d from dump1090-fa (12776 in last 5m); 10487686 msgs sent to FlightAware
Feb 22 17:23:36 piaware piaware[530]: 10502312 msgs recv’d from dump1090-fa (12275 in last 5m); 10499961 msgs sent to FlightAware
Feb 22 17:28:36 piaware piaware[530]: 10514194 msgs recv’d from dump1090-fa (11882 in last 5m); 10511843 msgs sent to FlightAware
Feb 22 17:33:18 piaware piaware[530]: mlat-client(6159): Receiver status: connected
Feb 22 17:33:18 piaware piaware[530]: mlat-client(6159): Server status: synchronized with 451 nearby receivers
Feb 22 17:33:18 piaware piaware[530]: mlat-client(6159): Receiver: 869.8 msg/s received 262.1 msg/s processed (30%)
Feb 22 17:33:18 piaware piaware[530]: mlat-client(6159): Server: 0.1 kB/s from server 0.1kB/s TCP to server 2.6kB/s UDP to server
Feb 22 17:33:18 piaware piaware[530]: mlat-client(6159): Results: 15.4 positions/minute
Feb 22 17:33:18 piaware piaware[530]: mlat-client(6159): Aircraft: 68 of 141 Mode S, 80 of 119 ADS-B used
Feb 22 17:33:36 piaware piaware[530]: 10527058 msgs recv’d from dump1090-fa (12864 in last 5m); 10524707 msgs sent to FlightAware
Feb 22 17:37:27 piaware piaware[530]: Lost connection to adept server at piaware.flightaware.com/1200: server closed connection
pi@piaware:/var/log $`

My current build of piaware is 7.1. Looks like the new Raspberry 4 board came with Buster. I’ll your rebuild it a try tonight.

17

You can confirm your OS name by following command:

lsb_release -sc
18

Network issue? Is your Pi connected to your router via Ethernet cable or wifi? How far away is it from the router?

19

Got this message.

Cannot write to ‘tcl-tls_1.7.22-1+fa1.bpo10+1_armhf.deb’ (Permission denied).

20

This is “normal” in that the internet being the internet, sometimes connections fail. The bug is that piaware should be automatically reconnecting at this point, but instead it just stops doing anything until restarted (probably due to the tcl-tls bug, which manifests as a hang within the tcl-tls code on disconnection in specific cases). Normally, you’d see reconnection attempts logged after the disconnect.

21

(1) What is output of following command?

lsb_release -sc

(2) Give the command starting with sudo like below:

sudo wget https://github.com/abcd567a/tcltls-rebuild/releases/download/v1/tcl-tls_1.7.22-1+fa1.bpo10+1_armhf.deb