FlightAware Discussions

Piaware UDP error messages

I know, there’s 100 other posts about this on here, but none of them really have a solution for me. I’m getting the “X% of multilateration messages (UDP) are not reaching the server 0 check your network?” But the numbers are generally very high. I’m running piaware on Debian on a laptop

Static hostname: arthas
...
Operating System: Debian GNU/Linux 10 (buster)
Kernel: Linux 4.19.0-14-amd64
Architecture: x86-64

It’s hard-wired to ethernet

enx5c857e31e986: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
inet 10.0.0.143  netmask 255.255.255.0  broadcast 10.0.0.255
inet6 2601:380:4103:ead0:f4f7:cafa:eac1:a2f2  prefixlen 64  scopeid 0x0<global>
inet6 2601:380:4103:ead0::1909  prefixlen 128  scopeid 0x0<global>
inet6 fe80::f968:b348:3863:2d7c  prefixlen 64  scopeid 0x20<link>
ether 5c:85:7e:31:e9:86  txqueuelen 1000  (Ethernet)
RX packets 9942  bytes 1551106 (1.4 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 9301  bytes 2738757 (2.6 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The ethernet goes through 3 unmanaged switches to my comcast router. I turned the firewall in my router completely off for both IPv4 and IPv6 and it didn’t change anything.

I’m running piaware and dump1090-fa version 5

root@arthas:/home/deadgilder# dump1090-fa --version
-----------------------------------------------------------------------------
| dump1090 ModeS Receiver                                   dump1090-fa 5.0 |
| build options: ENABLE_RTLSDR ENABLE_BLADERF ENABLE_HACKRF ENABLE_LIMESDR  |
-----------------------------------------------------------------------------
detected runtime CPU features: AVX AVX2
selected DSP implementations:
magnitude_uc8                            lookup_unroll_4_x86_avx2
magnitude_uc8_aligned                    lookup_unroll_4_x86_avx2
magnitude_power_uc8                      twopass_x86_avx2
magnitude_power_uc8_aligned              twopass_x86_avx2_aligned
magnitude_sc16                           exact_float_x86_avx2
magnitude_sc16_aligned                   exact_float_x86_avx2_aligned
magnitude_sc16q11                        exact_float_x86_avx2
magnitude_sc16q11_aligned                exact_float_x86_avx2_aligned
mean_power_u16                           u32_x86_avx2
mean_power_u16_aligned                   u32_x86_avx2_aligned

root@arthas:/home/deadgilder# piaware -v
5.0

But this has been happening even when I was on 4, it wasn’t until I upgraded yesterday that I actually took notice of it. I’m getting random disconnects, 80%+ message failures… I don’t know what else to troubleshoot. ICMP pings are fine, I can’t find anything to send a udp ping to the IPv6 address it lists below (Using UDP transport to 2600:c13:1002:4::232 port 9848). I put my antenna on the roof yesterday and I’m getting TONS of data which I’m storing in Vertica on my machine, but they’re all dropping while going out to flightaware.

Apr 12 22:18:18 arthas piaware[1132]: 227914 msgs recv'd from dump1090-fa (671 in last 5m); 227914 msgs sent to FlightAware
Apr 12 22:19:15 arthas piaware[1132]: NOTICE from adept server: 98% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:20:00 arthas piaware[1132]: NOTICE adept server is shutting down.  reason: reconnection request received
Apr 12 22:20:02 arthas piaware[1132]: Lost connection to adept server at piaware.flightaware.com/1200: server closed connection
Apr 12 22:20:02 arthas piaware[1132]: multilateration data no longer required, disabling mlat client
Apr 12 22:20:03 arthas piaware[1132]: fa-mlat-client exited normally
Apr 12 22:20:03 arthas piaware[1132]: reconnecting in 5 seconds...
Apr 12 22:20:03 arthas piaware[1132]: mlat-client(1186): Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data
Apr 12 22:20:03 arthas piaware[1132]: mlat-client(1186): Exiting on connection loss
Apr 12 22:20:08 arthas piaware[1132]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Apr 12 22:20:08 arthas piaware[1132]: Connection with adept server at piaware.flightaware.com/1200 established
Apr 12 22:20:09 arthas piaware[1132]: TLS handshake with adept server at piaware.flightaware.com/1200 completed
Apr 12 22:20:09 arthas piaware[1132]: FlightAware server certificate validated
Apr 12 22:20:09 arthas piaware[1132]: encrypted session established with FlightAware
Apr 12 22:20:09 arthas piaware[1132]: logged in to FlightAware as user DeadGilder
Apr 12 22:20:09 arthas piaware[1132]: my feeder ID is 16d6f776-19d6-4020-8dcb-9d0a394c47f4
Apr 12 22:20:09 arthas piaware[1132]: site statistics URL: https://flightaware.com/adsb/stats/user/DeadGilder#stats-145781
Apr 12 22:20:09 arthas piaware[1132]: multilateration data requested
Apr 12 22:20:09 arthas piaware[1132]: Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,30105 --$
Apr 12 22:20:09 arthas piaware[1132]: mlat-client(514378): fa-mlat-client 0.2.11 starting up
Apr 12 22:20:09 arthas piaware[1132]: mlat-client(514378): Using UDP transport to 2600:c13:1002:4::232 port 9848
Apr 12 22:20:09 arthas piaware[1132]: mlat-client(514378): Listening for Beast-format results connection on port 30105
Apr 12 22:20:09 arthas piaware[1132]: mlat-client(514378): Listening for Extended Basestation-format results connection on port 30106
Apr 12 22:20:09 arthas piaware[1132]: mlat-client(514378): Route MTU changed to 1500
Apr 12 22:20:09 arthas piaware[1132]: mlat-client(514378): Input connected to localhost:30005
Apr 12 22:20:09 arthas piaware[1132]: mlat-client(514378): Input format changed to BEAST, 12MHz clock
Apr 12 22:20:10 arthas piaware[1132]: mlat-client(514378): Beast-format results connection with ::1:30104: connection established
Apr 12 22:21:10 arthas piaware[1132]: NOTICE from adept server: 43% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:22:10 arthas piaware[1132]: NOTICE from adept server: 43% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:23:11 arthas piaware[1132]: NOTICE from adept server: 86% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:23:18 arthas piaware[1132]: 228476 msgs recv'd from dump1090-fa (562 in last 5m); 228461 msgs sent to FlightAware
Apr 12 22:24:11 arthas piaware[1132]: NOTICE from adept server: 56% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:25:12 arthas piaware[1132]: NOTICE from adept server: 60% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:26:12 arthas piaware[1132]: NOTICE from adept server: 83% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:27:12 arthas piaware[1132]: NOTICE from adept server: 43% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:28:13 arthas piaware[1132]: NOTICE from adept server: 100% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:28:18 arthas piaware[1132]: 228979 msgs recv'd from dump1090-fa (503 in last 5m); 228964 msgs sent to FlightAware
Apr 12 22:29:13 arthas piaware[1132]: NOTICE from adept server: 62% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:30:13 arthas piaware[1132]: NOTICE from adept server: 81% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:31:14 arthas piaware[1132]: NOTICE from adept server: 44% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:32:14 arthas piaware[1132]: NOTICE from adept server: 100% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:33:14 arthas piaware[1132]: NOTICE from adept server: 43% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:33:18 arthas piaware[1132]: 229504 msgs recv'd from dump1090-fa (525 in last 5m); 229489 msgs sent to FlightAware
Apr 12 22:34:15 arthas piaware[1132]: NOTICE from adept server: 100% of multilateration messages (UDP) are not reaching the server - check your network?
Apr 12 22:35:09 arthas piaware[1132]: mlat-client(514378): Receiver status: connected
Apr 12 22:35:09 arthas piaware[1132]: mlat-client(514378): Server status:   synchronized with 10 nearby receivers
Apr 12 22:35:09 arthas piaware[1132]: mlat-client(514378): Receiver:  123.2 msg/s received       72.2 msg/s processed (59%)
Apr 12 22:35:09 arthas piaware[1132]: mlat-client(514378): Server:      0.1 kB/s from server    0.0kB/s TCP to server     0.7kB/s UDP to server
Apr 12 22:35:09 arthas piaware[1132]: mlat-client(514378): Results:  43.5 positions/minute
Apr 12 22:35:09 arthas piaware[1132]: mlat-client(514378): Aircraft: 9 of 11 Mode S, 16 of 25 ADS-B used

If you suspect it’s IPv6 specific, you could try disabling ipv6 and see if that helps; add

net.ipv6.conf.all.disable_ipv6=1

to /etc/sysctl.conf and reboot

We added IPv6 addresses to the DNS name that piaware uses to locate servers a couple of months ago, but I’ve seen a few reports of problems like this so I’m considering backing that change out.

I do believe that fixed it! Thank you! As soon as I turned off ipv6 the errors stopped and it’s gone 15 minutes with nothing in the log but msg rec’d messages. The longest it had gone before that was 1 minute 1 second.

Apr 13 21:04:07 arthas piaware[1153]: 1612 msgs recv'd from dump1090-fa (693 in last 5m); 1602 msgs sent to FlightAware
Apr 13 21:09:07 arthas piaware[1153]: 2350 msgs recv'd from dump1090-fa (738 in last 5m); 2340 msgs sent to FlightAware
Apr 13 21:14:07 arthas piaware[1153]: 3107 msgs recv'd from dump1090-fa (757 in last 5m); 3097 msgs sent to FlightAware

It’s not quite in place yet, but in a few days there will be an alternative workaround which doesn’t require disabling IPv6 everywhere:

piaware-config adept-serverhosts piaware-ipv4.flightaware.com

Checked this morning and still no errors. I’m already triple what I was all day Monday which was the first full day the antenna was on the roof. Much appreciated

Has this workaround been implemented yet?

Yes. (20 characters)

1 Like