MLAT unreliable after moving to 8.2

I’ve been running various versions of PiAware since December 2018. Raspberry Pi 3 B+. Pro Stick Plus receiver, FlightAware 1090MHz antenna, power over ethernet hat. MLAT was reliable until recently but now alternates between green and red on my status page. The only thing that’s changed is I installed version 8.2 (new SD card, new image). Modest CPU load shown when I ssh to the Pi and run top(1); dump1090-fa uses about 30% of a CPU and there are no other processes using substantial CPU time.

2 Likes

I upgraded to 8.2 lately and notice the same exact problem with MLAT. Status is quite often yellow or red.
Also a RPI 3b+ and FA Pro stick plus.

2 Likes

this is my problem. I have 2 different setups and 2 different sticks and antennas, and 1 works fine, the other keeps saying there is a sy stem time problem. It’s not my setuip. I even swapped computers and the same thing happens.

If I run piaware from the ubuntu CLI, I can get it to say the MLAT is working and it syncs to 1 receiver.

I’ve come to terms that it’s a piaware funky error, and not anything that i’m doing. I tried to fix this for hours and hours, reloaded the system at least 5 times, etc. etc. etc. I probably spent 40 hours trying to fix it. It’s a piaware thing…

1 Like

I had a simillar MLAT problem with a development system. Seems I had the location offset too far from the real location. Had to use Google Earth to get LAT/LON/Height correct. Worked reliably after that. I figured I was about 50 ft in error.

1 Like

My lat/lon is perfect.

If I swap my systems and change the feeder id and the lat lon for the other setup the same thing happens.

I’ve wiped the drive clean and reinstalled many times. Does this from the outset. I’ve used premade Ubuntu packages and also built the packages myself.

It’s piaware, not my setup.

When I run piaware via cli instead of GUI it supposedly syncs fine to mlat but then can only find one other antenna. Nothing is different except for how I load piaware.

To isolate the problem to hardware or the software on the SD card, can you swap the cards between your systems and see if the problems stay with the SD card or if the problems stay with the hardware. Just one way to rule out hardware and software issues. If it stays with the hardware, perhaps swap other pieces of the systems. learned that while an Air Force electronics technician. Sounds like you may have done some of this already. Just a thought.
You could also remove anything else connected to the USB/Ethernet ports as the USB and Ethernet ports share the same controller and can seriously affect MLAT timing status.

FlightAware mlat doesn’t use system time and won’t complain about system time (it may say “clock unstable”, but that’s not the system time it’s complaining about).

Can you please provide a screenshot of what you’re doing here because this doesn’t sound like piaware.

My suspicion is that it’s some other mlat system, not piaware, that you are having problems with.

Sorry. Yes. Clock unstable is the error.

I run ubuntu. Piaware starts at startup. But if I go into the cli and run “piaware” the clock unstable error goes away but I can only connect to one other mlat station.

Again, I’ve done the switcheroo with known working equipment. I’ve reinstalled zillions of times. I have a 2nd piaware system that is working well with mlat.

It’s some weird error. It’s either piaware or dump1090-fa but it certainly isn’t my setup.

On the system where MLAT doesn’t work it actually worked for a few hours and I have like 4 mlat planes in a normal sea of zeros. I can’t explain it.

@rickined1

You are running Ubuntu on which machine? Just saying “I am running ubuntu cli” does not give enough info.

Is Ubuntu installed on microSD card of RPi, installed on HD of a PC, or installed on Windows computer through a Vertual Machine, or something else

here’s part of a log from the piaware stats website for when the piaware from the cli isn’t running – just the regular piaware setup where the linux box boots up and runs piaware automatically. It says “system clock unstable” on the outer screen but MLAT.

[2023-03-04 05:18 EST] 825780 msgs recv’d from dump1090-fa (193 in last 5m); 825773 msgs sent to FlightAware
[2023-03-04 05:23 EST] 826008 msgs recv’d from dump1090-fa (228 in last 5m); 826001 msgs sent to FlightAware
[2023-03-04 05:26 EST] mlat-client(313998): Receiver status: connected
[2023-03-04 05:26 EST] mlat-client(313998): Receiver: 76.4 msg/s received 34.4 msg/s processed (45%)
[2023-03-04 05:26 EST] mlat-client(313998): Aircraft: 3 of 4 Mode S, 13 of 15 ADS-B used
[2023-03-04 05:26 EST] mlat-client(313998): Server status: not synchronized with any nearby receivers
[2023-03-04 05:28 EST] 826221 msgs recv’d from dump1090-fa (213 in last 5m); 826214 msgs sent to FlightAware
[2023-03-04 05:33 EST] 826502 msgs recv’d from dump1090-fa (281 in last 5m); 826495 msgs sent to FlightAware
[2023-03-04 05:41 EST] mlat-client(313998): Aircraft: 2 of 2 Mode S, 17 of 25 ADS-B used
[2023-03-04 05:41 EST] mlat-client(313998): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.5kB/s UDP to server
[2023-03-04 05:41 EST] mlat-client(313998): Server status: not synchronized with any nearby receivers
[2023-03-04 05:41 EST] mlat-client(313998): Receiver: 99.6 msg/s received 44.4 msg/s processed (45%)
[2023-03-04 05:41 EST] mlat-client(313998): Receiver status: connected
[2023-03-04 05:43 EST] 827175 msgs recv’d from dump1090-fa (369 in last 5m); 827168 msgs sent to FlightAware
[2023-03-04 05:48 EST] 827592 msgs recv’d from dump1090-fa (417 in last 5m); 827585 msgs sent to FlightAware
[2023-03-04 05:53 EST] 828043 msgs recv’d from dump1090-fa (451 in last 5m); 828036 msgs sent to FlightAware
[2023-03-04 05:56 EST] mlat-client(313998): Receiver: 144.0 msg/s received 62.6 msg/s processed (43%)
[2023-03-04 05:56 EST] mlat-client(313998): Server status: not synchronized with any nearby receivers
[2023-03-04 05:56 EST] mlat-client(313998): Aircraft: 1 of 1 Mode S, 21 of 26 ADS-B used
[2023-03-04 05:56 EST] mlat-client(313998): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.7kB/s UDP to server
[2023-03-04 05:58 EST] 828579 msgs recv’d from dump1090-fa (536 in last 5m); 828572 msgs sent to FlightAware
[2023-03-04 06:03 EST] 829029 msgs recv’d from dump1090-fa (450 in last 5m); 829022 msgs sent to FlightAware
[2023-03-04 06:08 EST] 829423 msgs recv’d from dump1090-fa (394 in last 5m); 829416 msgs sent to FlightAware
[2023-03-04 06:11 EST] mlat-client(313998): Receiver: 114.0 msg/s received 50.9 msg/s processed (45%)
[2023-03-04 06:11 EST] mlat-client(313998): Aircraft: 0 of 0 Mode S, 12 of 14 ADS-B used
[2023-03-04 06:11 EST] mlat-client(313998): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.6kB/s UDP to server
[2023-03-04 06:11 EST] mlat-client(313998): Server status: not synchronized with any nearby receivers

Now I’m going to go into the CLI and run “piaware” and let it run for a few minutes – it’s a intel celeron x64 system with 16 gb memory, 256 gb ssd, ubuntu pro 22.04. (note, it throws some errors but I get the exact same errors on my 2nd system with the exact same setup in which MLAT works)

(well, I’m impatient, because I don’t have time this morning, but this is the initial log when I run piaware from the CLI. eventually it gets to the exact same MLAT output as the log above but I can get it to connect to 1 other MLAT station… Which is probably the other identical system… and I have the same exact setup on both systems – flightaware pro stick. flight aware 1090 filter. same bias tee. same LNA. same antenna. I even swapped the flightaware pro sticks and it didnt’ change anything.

2023-03-04 11:15:08Z failed to reopen /var/log/piaware.log: couldn’t open “/var/log/piaware.log”: permission denied
2023-03-04 11:15:08Z ****************************************************
2023-03-04 11:15:08Z piaware version 8.2 is running, process ID 824258
2023-03-04 11:15:08Z your system info is: Linux pihole1 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
2023-03-04 11:15:10Z Connecting to FlightAware adept server at piaware.flightaware.com/1200
2023-03-04 11:15:10Z Connection with adept server at piaware.flightaware.com/1200 established
2023-03-04 11:15:10Z TLS handshake with adept server at piaware.flightaware.com/1200 completed
2023-03-04 11:15:10Z FlightAware server certificate validated
2023-03-04 11:15:10Z encrypted session established with FlightAware
2023-03-04 11:15:11Z adept reported location: 40.49283, -79.89401, 900ft AMSL
2023-03-04 11:15:11Z logged in to FlightAware as user rickined1
2023-03-04 11:15:11Z my feeder ID is 7f00805d-0621-418c-926d-882be2370c79
2023-03-04 11:15:11Z Failed to update feeder ID file at /var/cache/piaware/feeder_id: couldn’t open “/var/cache/piaware/feeder_id.new”: permission denied
2023-03-04 11:15:11Z site statistics URL: Eric Rickin ADS-B Feeder Statistics - FlightAware
2023-03-04 11:15:11Z multilateration data requested
2023-03-04 11:15:11Z Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,30105 --results ext_basestation,listen,30106 --udp-transport 206.253.80.200:15230:243944826
2023-03-04 11:15:11Z mlat-client(824274): fa-mlat-client 0.2.12 starting up
2023-03-04 11:15:11Z mlat-client(824274): Using UDP transport to 206.253.80.200 port 15230
2023-03-04 11:15:11Z mlat-client(824274): Warning: Could not create results output ‘beast,listen,30105’: [Errno 98] Address already in use
2023-03-04 11:15:11Z mlat-client(824274): Warning: Could not create results output ‘ext_basestation,listen,30106’: [Errno 98] Address already in use
2023-03-04 11:15:11Z mlat-client(824274): Route MTU changed to 1500
2023-03-04 11:15:11Z mlat-client(824274): Input connected to localhost:30005
2023-03-04 11:15:11Z mlat-client(824274): Input format changed to BEAST, 12MHz clock
2023-03-04 11:15:11Z ADS-B data program ‘dump1090-fa’ is listening on port 30005, so far so good
2023-03-04 11:15:11Z Starting faup1090: /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat 40.493 --lon -79.894
2023-03-04 11:15:11Z Started faup1090 (pid 824278) to connect to dump1090-fa
2023-03-04 11:15:11Z UAT support disabled by local configuration setting: uat-receiver-type
2023-03-04 11:15:11Z mlat-client(824274): Beast-format results connection with ::1:30104: connection established
2023-03-04 11:15:12Z piaware received a message from dump1090-fa!
2023-03-04 11:15:13Z piaware has successfully sent several msgs to FlightAware!
2023-03-04 11:15:43Z 41 msgs recv’d from dump1090-fa; 41 msgs sent to FlightAware
2023-03-04 11:20:43Z 363 msgs recv’d from dump1090-fa (322 in last 5m); 363 msgs sent to FlightAware

hi – linux box – it’s a celeron nuc running ubuntu pro 22.04. 256 gb ssd. 16 gb ram. only using about 6% of memory, hard drive is basically empty besides my flight software and ubuntu and pihole. pihole shows it using 0.25 of 4 cores…

dump1090-fa logs just shows the usual up and down gain changes.
Mar 04 02:04:57 pihole1 dump1090-fa[743]: rtlsdr: tuner gain set to 43.4 dB (gain step 24)
Mar 04 02:05:07 pihole1 dump1090-fa[743]: adaptive: available dynamic range (30.2dB) >= required dynamic range (30.0dB), stopping downwards scan here
Mar 04 03:08:35 pihole1 dump1090-fa[743]: adaptive: start periodic scan for acceptable dynamic range at increased gain
Mar 04 03:08:35 pihole1 dump1090-fa[743]: adaptive: changing gain from 43.4dB (step 24) to 43.9dB (step 25) because: periodic re-probing of dynamic range gain upper bound
Mar 04 03:08:35 pihole1 dump1090-fa[743]: rtlsdr: tuner gain set to 43.9 dB (gain step 25)
Mar 04 03:08:46 pihole1 dump1090-fa[743]: adaptive: available dynamic range (29.4dB) < required dynamic range (30.0dB), switching to downward scan
Mar 04 03:08:46 pihole1 dump1090-fa[743]: adaptive: changing gain from 43.9dB (step 25) to 43.4dB (step 24) because: probing dynamic range gain lower bound
Mar 04 03:08:46 pihole1 dump1090-fa[743]: rtlsdr: tuner gain set to 43.4 dB (gain step 24)
Mar 04 03:08:56 pihole1 dump1090-fa[743]: adaptive: available dynamic range (30.4dB) >= required dynamic range (30.0dB), stopping downwards scan here
Mar 04 04:11:27 pihole1 dump1090-fa[743]: adaptive: available dynamic range (28.6dB) + half gain step down (30.0dB) < required dynamic range (1.3dB), starting downward scan
Mar 04 04:11:27 pihole1 dump1090-fa[743]: adaptive: changing gain from 43.4dB (step 24) to 42.1dB (step 23) because: dynamic range fell below target value
Mar 04 04:11:27 pihole1 dump1090-fa[743]: rtlsdr: tuner gain set to 42.1 dB (gain step 23)
Mar 04 04:11:38 pihole1 dump1090-fa[743]: adaptive: available dynamic range (31.1dB) >= required dynamic range (30.0dB), stopping downwards scan here
Mar 04 05:15:12 pihole1 dump1090-fa[743]: adaptive: start periodic scan for acceptable dynamic range at increased gain
Mar 04 05:15:12 pihole1 dump1090-fa[743]: adaptive: changing gain from 42.1dB (step 23) to 43.4dB (step 24) because: periodic re-probing of dynamic range gain upper bound
Mar 04 05:15:12 pihole1 dump1090-fa[743]: rtlsdr: tuner gain set to 43.4 dB (gain step 24)
Mar 04 05:15:22 pihole1 dump1090-fa[743]: adaptive: available dynamic range (30.3dB) >= required dynamic range (30.0dB), continuing upward scan
Mar 04 05:15:22 pihole1 dump1090-fa[743]: adaptive: changing gain from 43.4dB (step 24) to 43.9dB (step 25) because: probing dynamic range gain upper bound
Mar 04 05:15:23 pihole1 dump1090-fa[743]: rtlsdr: tuner gain set to 43.9 dB (gain step 25)
Mar 04 05:15:33 pihole1 dump1090-fa[743]: adaptive: available dynamic range (29.3dB) < required dynamic range (30.0dB), switching to downward scan
Mar 04 05:15:33 pihole1 dump1090-fa[743]: adaptive: changing gain from 43.9dB (step 25) to 43.4dB (step 24) because: probing dynamic range gain lower bound
Mar 04 05:15:33 pihole1 dump1090-fa[743]: rtlsdr: tuner gain set to 43.4 dB (gain step 24)
Mar 04 05:15:44 pihole1 dump1090-fa[743]: adaptive: available dynamic range (30.2dB) >= required dynamic range (30.0dB), stopping downwards scan here
Mar 04 05:42:29 pihole1 dump1090-fa[743]: adaptive: available dynamic range (29.2dB) + half gain step down (30.0dB) < required dynamic range (1.3dB), starting downward scan
Mar 04 05:42:29 pihole1 dump1090-fa[743]: adaptive: changing gain from 43.4dB (step 24) to 42.1dB (step 23) because: dynamic range fell below target value
Mar 04 05:42:30 pihole1 dump1090-fa[743]: rtlsdr: tuner gain set to 42.1 dB (gain step 23)
Mar 04 05:42:40 pihole1 dump1090-fa[743]: adaptive: available dynamic range (31.1dB) >= required dynamic range (30.0dB), stopping downwards scan here

edit –
after letting piaware run from the command line for about 10 minutes I get this when I go into my flightaware stats –
Multilateration (MLAT): Supported / Enabled (synchronized with 1 nearby receivers)
and the mlat box is now green saying it’s synced.
2023-03-04 11:43:49Z mlat-client(825825): Receiver status: connected
2023-03-04 11:43:49Z mlat-client(825825): Server status: synchronized with 1 nearby receivers
2023-03-04 11:43:49Z mlat-client(825825): Receiver: 117.8 msg/s received 56.8 msg/s processed (48%)
2023-03-04 11:43:49Z mlat-client(825825): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.7kB/s UDP to server
2023-03-04 11:43:49Z mlat-client(825825): Aircraft: 4 of 5 Mode S, 20 of 28 ADS-B used
2023-03-04 11:44:21Z 1640 msgs recv’d from dump1090-fa (600 in last 5m); 1640 msgs sent to FlightAware

It’s receiving mlat but something happens at the server end so it doesn’t send the data to FA.

again, this is a piaware or dump1090-fa problem. I have 2 setups, exactly the same, I’ve swapped everything I can swap, I’ve wiped and reinstalled. and other people are having the same problem. There’s an error somewhere.

You shouldn’t run piaware by hand like that. Now you have two copies running, and one is running as the wrong user.

I don’t see that in the logs that you provide; the logs show no sync at all. I’ll take your word for it that sometimes you see “clock unstable”.

Assuming there are other feeders to sync with nearby, and your location is correct, and there are ADS-B aircraft available to sync with, then the most likely explanation is that the effective sample rate that your dongle is producing is wildly off. mlat will tolerate up to around 100ppm relative error; beyond that you won’t sync with other receivers. So the probable explanation is that there’s only approx one other receiver within +/- 100ppm of yours.

that plus clock unstable (which means that the apparent frequency of the dongle sampling rate is jumping around) leads me to think that you have a USB problem on that hardware. If USB is frequently dropping data, that looks exactly like a too-fast-and-unstable sample rate. (Most commonly I’ve seen this on VMs where the VM USB passthrough was not working well, but it looks like you’re on bare metal in this case)

(I’d also note that the mlat server is fairly generous about giving new connections some time to settle down before deciding that they have an unstable clock. So getting sync on a started-by-hand-[but-you-shouldn’t-be-doing-that] copy may simply be that it’s a new connection)

so, with respect (and given that mlat in general is working fine for everyone else) …

… it’s your setup.

Do you have different host hardware (not dongle/antenna) you can try?

Who are the other people?

1 Like

I actually swapped the computers (took the hard drive out and put it in the other computer) and the same thing happened. And I swapped dongles and the same thing happened. So I don’t think it’s the hardware.

So thousands of other users don’t have this issue, and obj from flightaware has reviewed and suggests its related to your set up.
But you still maintain that what you are experiencing is a flightaware issue…

Are your two NUC setups identical in all respects?

You can get MLAT issues when there is low voltage at the dongle due to a weak power supply or a long/higher resistance usb cable between the computer and the dongle.

It could be that one of your SSDs is more power hungry than the other causing an undervoltage at the dongle. If you have a USB voltage tester, this would be something to check.

So to be clear –

  1. you see the issue on machine A but not on machine B
  2. you swap the drives of A and B and make no other changes
  3. now you see the issue on machine B but not on machine A?

(please ensure you run each setup for at least a couple of hours without a restart when testing)

If that’s correct, can you try swapping only the feeder IDs of A and B and see if the problem moves?

also, please record timestamps for each of your tests so that I can correlate them with what I see on the server side.

Yes. When I just swap the hard drives the problem follows the hard drive. Not the hardware. I’ve swapped everything I can as A/B tests. I’ve tested the USB ports with a voltage tester. I’ve wiped the system and reinstalled Ubuntu and also piaware multiple times.

I’ve triple checked the GPS coordinates for the antenna. But when I swap systems and change antenna coordinates the “good” hard drive picks up mlat no matter which antenna I use, and the “bad” one doesn’t.

It’s very strange.

Do you think I could just set it up again de novo and get a new feeder id and see if that works? I haven’t tried that yet…

I’m away but I’ll have time this week and let you know. I appreciate your help. I think there’s just a strange bug somewhere since I can get my other system working without a problem.

I even had mlat working for the “bad” hard drive when I had it set up initially, but when I moved the antenna to outside and changed the GPS it stopped finding MLAT (unstable clock…) So I changed it back to the original (now not correct) gps coordinates and mlat worked again. I was using that for a while but I ended up reformatting the ssd when I messed something up in Ubuntu and didn’t have a time shift image saved, so when I reinstalled I stated getting “clock unstable” no matter what gos (new/old) I used even though the antenna didn’t move. I then moved the antenna back to the original position inside and it still didn’t work. Oh well.

Seems the hard drive with which problem follows is power hungry, draws lot of current, and causes low voltage in the NUC to which it is connected.

What is the output of following command on both the NUC?

sudo dmesg --ctime | grep voltage

 

I’ll check and get back to you .I’m wondering if you nailed down the problem.

I also haven’t tried a powered USB hub. Maybe that would solve things too if it is a weird intermittent power problem.

Edit: that command doesn’t return any output.
Is there another command I can use? When I use it without the pipe it returns a lot of stuff.