MLAT Unreliable Timing

FlightAware ADS-B Flight Tracking
1

(Edit - the MLAT flag changed to red again Sun/3/21 so it stands unresolved. Updated antenna location/elevation so will see if that has anything to do with it. If that doesn’t fix it I’ll just let it do the red/green flip flop. ADS-B works fine and that’s most of the traffic I see.)

We just updated our internet connection with CenturyLink from 7Mbps single connection to a 60Mbps VDSL bonded connection. Now my MLAT banner has turned red and I see this message:

“This feeder is not being used for multilateration because its timing information appears to be unreliable. This can be caused by the site location being incorrect, or because your Pi is running out of free CPU.”

We’ve also seen video streams on Roku defocus and lose detail. I think there is possibly an issue with out of sequence packets due to the two paths and error correction on the two paths (we see RS FEC corrections on both links - not too many but 55 on one connection and 15281 on the other right now since a reboot this morning. This varies and initially the counts were swapped. No CRC errors at all so far.

Is the issue with MLAT something that just happens with these dual path connections? Is this an issue with the FEC corrections? And is there any workaround?

We don’t have fiber here yet but apparently it is available if I don’t mind paying for trenching to get fiber to the house. Presumably that would be a single connection without the possibility to have data arrive out of order like this trunked connection now does.

Thanks for any guidance. Still learning the down sides to this kind of connection. I didn’t know there were any but out of sequence packets seem to impact a lot of things to varying degrees. Sigh.

2

I remember having relatively high packet loss reported by piaware when i was using a bonded connection.

Maybe the server doesn’t handle out of order UDP packets well?
But … i wouldn’t expect UDP packet loss to cause loss of sync, interesting.

1 Like
3

mlat shouldn’t really care about network latency (within reason) or mis-sequencing or duplication, in theory at least. The timing may just be coincidence?

The packet loss calculation is a pretty dumb “number of UDP packets heard by server vs. number of UDP packets sent as reported via the TCP connection” so it’s quite possible something is getting confused there.

(or possibly you were getting packet fragmentation/reassembly problems? reassembly really doesn’t like out of order fragments and some setups will quite aggressively drop data in that case)

1 Like
4

How long is the typical interval between the UDP packets?
For reordering to take place you’d need a considerable delay.

Maybe that connection was just dropping UDP packets i really didn’t trust the software setup, seemed unstable overall.
Seems bonding tunnels aren’t easy to set up and ISPs like to cut corners.

I tried lowering the MTU, not sure anymore if that helped.
Once i tried disabling that bonding and only using the DSL link the complete internet connection felt like much less random hiccups … so i switched it off.

But in my case the DSL connection was 20 MBit/s already i believe or more … so the mobile wouldn’t add huge amounts.

1 Like
5

If you want to try changing the MTU maybe that helps.
These commands will only change it until the next reboot:

sudo ifconfig wlan0 mtu 1280
sudo ifconfig eth0 mtu 1280
sudo systemctl restart piaware

Thats wifi and ethernet.

Oh and you also want to check the piaware logs for “out of order” messages.

sudo journalctl -e -u piaware
1 Like
6

Any partial packet is flushed every 0.5s, but they’re also sent when they get close to full (near MTU) so you might see a burst. But yeah, I would not expect much scope for reordering unless you’re really unlucky.

1 Like
7

Reordering would lead to packet loss correct?
I suppose it depends on how the server is programmed that’s receiving the UDP.

And yes i’m aware the packet loss doesn’t mean you’re gonna lose sync.
(i never had issues with that, packet loss was around 30% at most)

1 Like
8

The server doesn’t really care about the order, it just counts packets for the packet-loss stats.
Packets are self-contained so an out of order packet can still be processed.

(I think you’re familiar with the jsonclient protocol in mlat-client, the UDP processing on the server side basically just reconstitutes that from the packed data)

2 Likes
9

Thanks for the replies, all!

I ran that command @wiedehopf suggested and see things like this below. There are occasional messages about “clock unstable” in there. As an aside, my MLAT flag is green again so whatever it is comes and goes. It’s the blue FlightAware SDR dongle if that matters.

If I am understanding, the trunking shouldn’t affect the MLAT messages then? I haven’t changed MTU yet. Will give that a try. The Pi is connected via WiFi but to a router this side of the modem - not connected directly to the modem. I disabled the modem WiFi as I like to control my own security destiny. That way when we got the new modem, I just plugged the WAN port of my own router into a LAN port on the new modem which is how everything was configured before. The router WAN was plugged into the old modem LAN. Old modem WiFi was similarly disabled.

I’ll give the MTU settings a try and keep checking the MLAT status flag on my Feeder Statistics page.

Thanks again!

Mar 19 11:32:23 raspberrypi piaware[544]: mlat-client(24104): Receiver status: connected
Mar 19 11:32:23 raspberrypi piaware[544]: mlat-client(24104): Server status: synchronized with 93 nearby receivers
Mar 19 11:32:23 raspberrypi piaware[544]: mlat-client(24104): Receiver: 770.2 msg/s received 352.4 msg/s processed (46%)
Mar 19 11:32:23 raspberrypi piaware[544]: mlat-client(24104): Server: 0.1 kB/s from server 0.0kB/s TCP to server 2.2kB/s
Mar 19 11:32:23 raspberrypi piaware[544]: mlat-client(24104): Results: 29.0 positions/minute
Mar 19 11:32:23 raspberrypi piaware[544]: mlat-client(24104): Aircraft: 7 of 16 Mode S, 49 of 105 ADS-B used
Mar 19 11:36:43 raspberrypi piaware[544]: 1230601 msgs recv’d from dump1090-fa (1983 in last 5m); 1200182 msgs sent to FlightAware
Mar 19 11:41:43 raspberrypi piaware[544]: 1232724 msgs recv’d from dump1090-fa (2123 in last 5m); 1202305 msgs sent to FlightAware
Mar 19 11:46:43 raspberrypi piaware[544]: 1234766 msgs recv’d from dump1090-fa (2042 in last 5m); 1204347 msgs sent to FlightAware
Mar 19 11:47:23 raspberrypi piaware[544]: mlat-client(24104): Receiver status: connected
Mar 19 11:47:23 raspberrypi piaware[544]: mlat-client(24104): Server status: clock unstable
Mar 19 11:47:23 raspberrypi piaware[544]: mlat-client(24104): Receiver: 849.5 msg/s received 403.9 msg/s processed (48%)
Mar 19 11:47:23 raspberrypi piaware[544]: mlat-client(24104): Server: 0.1 kB/s from server 0.0kB/s TCP to server 2.6kB/s
Mar 19 11:47:23 raspberrypi piaware[544]: mlat-client(24104): Results: 39.5 positions/minute
Mar 19 11:47:23 raspberrypi piaware[544]: mlat-client(24104): Aircraft: 9 of 11 Mode S, 46 of 118 ADS-B used

10

@wiedehopf - I dropped my WLAN MTU, restarted PiAware, the MLAT flag went back to green… And now it’s red again. Just checked.

11

Did you try setting the location on your stats page again? Even if you enter the same values and save it, maybe it can help.

And also check that the Raspberry is getting the correct date/time from a timeserver.
This broke my setup recently

Also my airsquitter remain “red” until the date/time is set via GPS after a reboot

12

Yeah it was unlikely … still worth a try :wink:

It’s unlikely that your internet connection is the issue honestly.
No output for this command means good news:

sudo dmesg --ctime | grep voltage

But even if there is no undervoltage message there there can still be voltage issues if you have the SDR connected via an USB extension.

The typical issue: for w/e reason the SDR isn’t getting sufficient voltage.
Try a different power supply.

Not so typical issue: SDR dying

Not so typical issue: Combining data from two receivers.
(but then it should say out of order timestamps i think)

13

No voltage messages and the receiver is directly connected. I checked the Pi logs and no voltage messages there either but good idea to check it. Also, looking at the error rates, there’s not been much change but the MLAT flag is red again now. It’s definitely coming and going for some reason.

I checked the Pi time/date and it’s correct to the second. There were some time server messages in the logs but just from during the time the network was down. But in grepping for time messages, the logs had dump978-fa timeouts every 30 seconds but I don’t have a 2nd SDR for 978. So I shut the 978 daemons down. MLAT flag is still red, though.

Had a thought, though. The Pi was up during and since the time the network was down so am throwing a reboot on it now just in case. The flag turned green again so will keep an eye on it.

I haven’t re-entered coordinates yet but will keep that in pocket if the problem persists.

I thought I had turned off the 978 daemons but just did the upgrade to 5.0 five days ago. I just stopped them for now as I may add a second SDR for 978 just to see how it goes. I also have an orange FlightAware dongle and another Pi and power supply so can even change out hardware.

Thanks!

14

I edited 1st post that things seem fixed after an embarrassingly simple reboot of the Pi. For whatever reason it seems it wasn’t tolerant of losing contact with the FlightAware servers for the approximate hour while modems were swapped and wiring got done.

Apologies (again) for wasting time. I had a somewhat similar issue but different behavior after upgrading to PiAware 5.0. Rebooted once. Should have rebooted twice. In this case just a single reboot would have fixed it.

No idea why losing network access for an hour would make MLAT timing think it was unreliable or why it would periodically decide it was reliable and then go back to unreliable. I could refresh my site web page and the MLAT flag would sometimes flip state.

While the issue did seem to appear with the network outage and upgrade, it seems to have had nothing to do with the trunked connection and @obj was right that out of order packets (if there even were out of order packets) wasn’t the issue. Best guess is some part of PiAware just got confused during the network outage and stayed confused after it had access again.

Thanks again for the help. In the future, I’ll reboot a few times (if necessary) before posting. :wink:

15

Well, not resolved, apparently. The MLAT flag is back red again. The popup explainer for the anomaly says timing information appears unreliable because the location being incorrect or running out of free CPU. I checked the load with top and dump1090-fa is running at 28-29%. A reboot fixes the red flag as previously noted.

So maybe this is something with the 5.0 upgrade and increased CPU load? This is a 3B Pi that isn’t overclocked. I would expect it to hit the Pi Zero folks harder since I think their CPU clocks a little slower.

I’m going to halt and copy the SD card in prep to run some commands to also send data to adsbexchange so if it is a failing SD card, I might trap it there. Will also see if my location could be the issue per fox above.

Ultimately, it seems to be a fairly rare thing and MLAT doesn’t seem that useful around here anyway. Just not many MLAT aircraft. So worst case is MLAT goes unused since everything else seems to work fine.

Mysteries.

16

FWIW, I am running on a Pi2 and the CPU is at 50% but I have not seen any problem with MLAT timing.

1 Like
17

Yeah, at 30% CPU you would think it could still get messages out.

I just reset my antenna location and elevation on the stats page so will see if that might have been the issue. The flag was (apparently) green until this morning when I checked it. It seems to reset itself so sometimes a refresh of the stats page shows a change of state.

It also hadn’t been happening until now, or at least I never caught it since it can come and go. Two things that changed from when I wasn’t seeing it before to seeing it now are switching to trunked VDSL from single connection (great speed improvement and lower bill), and the recent upgrade to PiAware 5.0. Unfortunately I didn’t notice the red flags until after both changes were done so no idea which is the culprit - if either. With obj saying the packets are complete and stand alone, I tend to doubt it’s the twin routes and packets out of sequence. It could possibly still be related to the VDSL connection, though, if something else is going on.

But if the antenna location/elevation bit doesn’t shake it out, it’s not a big deal. I don’t see that many MLAT aircraft. The vast majority are ADS-B. So will probably just leave this unresolved, take the hit, and carry on. Perfection is the enemy of good enough. :wink:

18

The usual culprits for timing errors, assuming your location etc is correct, are bad USB cabling/connection or a bad power supply. Maybe your hardware went bad (or got rearranged when you did the VDSL install, provoking the problem?)

19

Hey obj, it would have to be hardware going bad between those choices - the Pi is located in a box in the garage and connected in over WiFi with a strong signal. The modem is used strictly as a modem and my router/AP just plugs into the modem and also wasn’t touched. I’ll keep an eye on it since if it’s hardware dying, that almost always gets worse over time instead of better or staying the same.

Do your logs record if MLAT timing goes unreliable for our stations? It’s obviously part of the station stats page. Are any others seeing the MLAT flag get set and cleared?

20

I have a similar problem with two of my sites.

They both report This feeder is not being used for multilateration at the same time and clear the problem at the same time.

The sites are located about 1 KM apart and have nothing in common except they are attached to the same account.

Site 81639 is running Ubuntu 20_04 in a VMware VM on a PC
It is connected by Ethernet and a 50 Mb/s cable service.
It is co-located with 4 other Pis which do not go out in sympathy.

Feeder Type: PiAware (Debian Package Add-on) 3.8.1
Feeder Mode: Mode-S (1090 MHz)
Multilateration (MLAT): Supported / Enabled

Site 93830 is a Pi3B+ running Raspian with Dump1090 and Piaware installed following Flightaware instructions.
It is connected by Ethernet to a D-Link LTE modem.

Feeder Type: PiAware (Debian Package Add-on) 4.0
Feeder Mode: Mode-S (1090 MHz)
Multilateration (MLAT): Supported / Not Enabled

Note that Multilateration is NOT ENABLED on site 93830

image

This occurs frequently enough with these two Sites that I believe it is a Flightaware failure.

It would be pleasing if the Anomalies list wasn’t cluttered with notices about upgrading to Piaware version 5.0

S

1 Like