All day I’ve been having trouble sending data to Flightaware. Absolutely everything else on my network is working fine, including internet access. Since it’s been all day, and I’ve not seen any other complaints, I’m guessing it’s just me, but I can’t figure out what the issue is. I’ve restart the the Pi, can ssh into it and so forth. At my wits end, so thought I’d see if anyone else has any ideas. Here’s the relevant portion of the piaware.log file:
Dec 6 18:54:35 piaware piaware[5859]: reconnecting in 50 seconds...
Dec 6 18:54:35 piaware piaware[5859]: mlat-client(6777): Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data
Dec 6 18:54:35 piaware piaware[5859]: mlat-client(6777): Exiting on connection loss
Dec 6 18:55:25 piaware piaware[5859]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Dec 6 18:55:25 piaware piaware[5859]: Connection with adept server at piaware.flightaware.com/1200 established
Dec 6 18:55:25 piaware piaware[5859]: TLS handshake with adept server at piaware.flightaware.com/1200 completed
Dec 6 18:55:25 piaware piaware[5859]: FlightAware server certificate validated
Dec 6 18:55:25 piaware piaware[5859]: encrypted session established with FlightAware
Dec 6 18:55:26 piaware piaware[5859]: logged in to FlightAware as user cwesley
Dec 6 18:55:26 piaware piaware[5859]: my feeder ID is **REDACTED**
Dec 6 18:55:26 piaware piaware[5859]: site statistics URL: https://flightaware.com/adsb/stats/user/cwesley#stats-83179
Dec 6 18:55:26 piaware piaware[5859]: multilateration data requested
Dec 6 18:55:26 piaware piaware[5859]: Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type auto --results beast,connect,localhost:30104 --results beast,listen,30105 --results ext_basestation,listen,30106 --udp-transport 70.42.6.156:5972:477411466
Dec 6 18:55:27 piaware piaware[5859]: mlat-client(7794): fa-mlat-client 0.2.10 starting up
Dec 6 18:55:27 piaware piaware[5859]: mlat-client(7794): Using UDP transport to 70.42.6.156 port 5972
Dec 6 18:55:27 piaware piaware[5859]: mlat-client(7794): Listening for Beast-format results connection on port 30105
Dec 6 18:55:27 piaware piaware[5859]: mlat-client(7794): Listening for Extended Basestation-format results connection on port 30106
Dec 6 18:55:27 piaware piaware[5859]: mlat-client(7794): Input connected to localhost:30005
Dec 6 18:55:27 piaware piaware[5859]: mlat-client(7794): Detected BEAST format input
Dec 6 18:55:27 piaware piaware[5859]: mlat-client(7794): Input format changed to BEAST, 12MHz clock
Dec 6 18:55:27 piaware piaware[5859]: mlat-client(7794): Beast-format results connection with 127.0.0.1:30104: connection established
Dec 6 18:56:05 piaware piaware[5859]: data isn't making it to FlightAware, reconnecting...
Dec 6 18:56:05 piaware piaware[5859]: multilateration data no longer required, disabling mlat client
Dec 6 18:56:06 piaware piaware[5859]: fa-mlat-client exited normally
Dec 6 18:56:06 piaware piaware[5859]: reconnecting in 63 seconds...
Dec 6 18:56:06 piaware piaware[5859]: mlat-client(7794): Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data
Dec 6 18:56:06 piaware piaware[5859]: mlat-client(7794): Exiting on connection loss
I can ping it reliably. Running mtr for a few minutes doesn’t show any packet losses to FlightAware, although there is an intermediate host routinely not responding to pings… don’t think that means anything though.
Look at the latency difference between hops 2 and 4 - 100ms. That could mean that the link is overloaded. I get 40ms coast to coast in my company. Even Europe is only 75ms from NYC.
Is the device connected via wifi? The latency to hop 1 looks like wifi timings.
ICMP loss is not significant as it is treated as low priority traffic in most networks.
Hop 3 to 4 is my internet connection… it is fairly high latency. I live in the middle of nowhere (according to the ISPs, that is), and have very limited selection of providers. What I’m using now is something I’ve hobbled together using a VPN over an LTE link. I really hesitate to mention that because it tends to be the scapegoat for anything internet related that goes wrong in my household. But, it has been reliably reporting data to flightaware for over a year, until today, and I’ve changed nothing about it recently. Every other internet connected IoT device in my home is still working just fine, including Amazon Echos, Netflix on the TV, etc.
This is the primary problem and it means the TCP connection’s write buffer has filled up. Either the outgoing TCP segments aren’t making it to the FlightAware server, or the acks aren’t making it back.
The mlat stuff can be ignored as it is piggy-backing on the main TCP connection and so if the main connection goes down, so does mlat.
You can check netstat -nto, look for a non-zero send-q on the piaware connection to port 1200.
As to the cause - no idea, you might need to dig around with wireshark to work out what’s breaking. If I had to guess, maybe a path MTU problem that’s making the TCP connection wedge?
