Lost MLAT?

FlightAware ADS-B Flight Tracking
1

Hello,

One week ago I installed my first Piaware with a Prostick Plus dongle on a PiZero. Everything worked fine for a week.

Yesterday I noticed (without making any change in my system, that MLAT was no longer working and reporting “Clock Unstable” or “MLAT disabled/not synked/Lat Long not configured”.

My coordinates are configured and precise (with a GPS) and were working for the first week. MLAT is enabled.

I just disconnected and reconnected everything and it worked for 2-3 minutes, even reporting a few positions, but went down again. ADSB is working fine, so I assume it’s not a USB/Dongle problem.

My location (Mexico City) does not have a lot of other stations.(9 within 10 miles according to my stats page, 6 of which show the green check under the MLAT column). Sometimes (very few) I’ve seen a message in the “not synchronized with any nearby receivers” in the log.

Any suggestions?

Thanks

PD Here is my last hor log:

Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Receiver status: connected
Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Server status: clock unstable
Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Receiver: 39.9 msg/s received 28.7 msg/s processed (72%)
Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server
Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Aircraft: 0 of 3 Mode S, 6 of 8 ADS-B used
Sep 4 07:05:09 piaware piaware[525]: 49464 msgs recv’d from dump1090-fa (332 in last 5m); 49464 msgs sent to FlightAware
Sep 4 07:10:09 piaware piaware[525]: 49866 msgs recv’d from dump1090-fa (402 in last 5m); 49866 msgs sent to FlightAware
Sep 4 07:15:09 piaware piaware[525]: 50198 msgs recv’d from dump1090-fa (332 in last 5m); 50198 msgs sent to FlightAware
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Receiver status: connected
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Server status: clock unstable
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Receiver: 45.5 msg/s received 32.8 msg/s processed (72%)
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Aircraft: 1 of 1 Mode S, 4 of 12 ADS-B used
Sep 4 07:20:09 piaware piaware[525]: 50545 msgs recv’d from dump1090-fa (347 in last 5m); 50545 msgs sent to FlightAware
Sep 4 07:25:09 piaware piaware[525]: 50884 msgs recv’d from dump1090-fa (339 in last 5m); 50884 msgs sent to FlightAware
Sep 4 07:30:09 piaware piaware[525]: 51179 msgs recv’d from dump1090-fa (295 in last 5m); 51179 msgs sent to FlightAware
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Receiver status: connected
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Server status: clock unstable
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Receiver: 41.3 msg/s received 29.0 msg/s processed (70%)
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Aircraft: 1 of 4 Mode S, 9 of 10 ADS-B used
Sep 4 07:35:09 piaware piaware[525]: 51461 msgs recv’d from dump1090-fa (282 in last 5m); 51461 msgs sent to FlightAware
Sep 4 07:40:09 piaware piaware[525]: 51731 msgs recv’d from dump1090-fa (270 in last 5m); 51731 msgs sent to FlightAware
Sep 4 07:42:47 piaware piaware[525]: NOTICE adept server is shutting down. reason: reconnection request received
Sep 4 07:42:49 piaware piaware[525]: Lost connection to adept server at piaware.flightaware.com/1200: server closed connection
Sep 4 07:42:49 piaware piaware[525]: multilateration data no longer required, disabling mlat client
Sep 4 07:42:50 piaware piaware[525]: fa-mlat-client exited normally
Sep 4 07:42:50 piaware piaware[525]: reconnecting in 5 seconds…
Sep 4 07:42:50 piaware piaware[525]: mlat-client(32356): Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data
Sep 4 07:42:50 piaware piaware[525]: mlat-client(32356): Exiting on connection loss
Sep 4 07:42:55 piaware piaware[525]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Sep 4 07:42:55 piaware piaware[525]: Connection with adept server at piaware.flightaware.com/1200 established
Sep 4 07:42:55 piaware piaware[525]: TLS handshake with adept server at piaware.flightaware.com/1200 completed
Sep 4 07:42:55 piaware piaware[525]: FlightAware server certificate validated
Sep 4 07:42:55 piaware piaware[525]: encrypted session established with FlightAware
Sep 4 07:42:58 piaware piaware[525]: logged in to FlightAware as user nospamprl
Sep 4 07:42:58 piaware piaware[525]: my feeder ID is 168b5d52-eacc-4f55-a9cf-a11261fb8094
Sep 4 07:42:58 piaware piaware[525]: site statistics URL: https://flightaware.com/adsb/stats/user/nospamprl#stats-134282
Sep 4 07:42:58 piaware piaware[525]: multilateration data requested
Sep 4 07:42:59 piaware piaware[525]: Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,30105 --results ext_basestation,listen,30106 --udp-transport 70.42.6.198:18011:3988778853
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): fa-mlat-client 0.2.11 starting up
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Using UDP transport to 70.42.6.198 port 18011
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Listening for Beast-format results connection on port 30105
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Listening for Extended Basestation-format results connection on port 30106
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Route MTU changed to 1500
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Input connected to localhost:30005
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Input format changed to BEAST, 12MHz clock
Sep 4 07:43:03 piaware piaware[525]: mlat-client(24314): Beast-format results connection with ::1:30104: connection established
Sep 4 07:45:09 piaware piaware[525]: 52012 msgs recv’d from dump1090-fa (281 in last 5m); 52006 msgs sent to FlightAware
Sep 4 07:50:09 piaware piaware[525]: 52329 msgs recv’d from dump1090-fa (317 in last 5m); 52323 msgs sent to FlightAware
Sep 4 07:55:09 piaware piaware[525]: 52697 msgs recv’d from dump1090-fa (368 in last 5m); 52691 msgs sent to FlightAware
Sep 4 08:04:18 piaware piaware[557]: creating pidfile /run/piaware/piaware.pid
Sep 4 08:04:18 piaware piaware[557]: ****************************************************
Sep 4 08:04:18 piaware piaware[557]: piaware version 3.8.1 is running, process ID 557
Sep 4 08:04:18 piaware piaware[557]: your system info is: Linux piaware 5.4.51+ #1333 Mon Aug 10 16:38:02 BST 2020 armv6l GNU/Linux
Sep 4 08:04:20 piaware piaware[557]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Sep 4 08:04:20 piaware piaware[557]: Connection with adept server at piaware.flightaware.com/1200 established
Sep 4 08:04:20 piaware piaware[557]: TLS handshake with adept server at piaware.flightaware.com/1200 completed
Sep 4 08:04:23 piaware piaware[557]: FlightAware server certificate validated
Sep 4 08:04:23 piaware piaware[557]: encrypted session established with FlightAware
Sep 4 08:04:27 piaware piaware[557]: ADS-B data program ‘dump1090-fa’ is listening on port 30005, so far so good
Sep 4 08:04:27 piaware piaware[557]: Starting faup1090: /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat 19.421 --lon -99.234
Sep 4 08:04:27 piaware piaware[557]: Started faup1090 (pid 732) to connect to dump1090-fa
Sep 4 08:04:27 piaware piaware[557]: UAT support disabled by local configuration setting: uat-receiver-type
Sep 4 08:04:28 piaware piaware[557]: adept reported location: 19.42072, -99.23367, 7764ft AMSL
Sep 4 08:04:28 piaware piaware[557]: logged in to FlightAware as user nospamprl
Sep 4 08:04:28 piaware piaware[557]: my feeder ID is 168b5d52-eacc-4f55-a9cf-a11261fb8094
Sep 4 08:04:28 piaware piaware[557]: site statistics URL: https://flightaware.com/adsb/stats/user/nospamprl#stats-134282
Sep 4 08:04:28 piaware piaware[557]: multilateration data requested
Sep 4 08:04:28 piaware piaware[557]: Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,30105 --results ext_basestation,listen,30106 --udp-transport 70.42.6.232:10006:4229442807
Sep 4 08:04:29 piaware piaware[557]: piaware received a message from dump1090-fa!
Sep 4 08:04:29 piaware piaware[557]: piaware has successfully sent several msgs to FlightAware!
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): fa-mlat-client 0.2.11 starting up
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Using UDP transport to 70.42.6.232 port 10006
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Listening for Beast-format results connection on port 30105
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Listening for Extended Basestation-format results connection on port 30106
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Route MTU changed to 1500
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Input connected to localhost:30005
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Input format changed to BEAST, 12MHz clock
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Beast-format results connection with ::1:30104: connection established
Sep 4 08:04:57 piaware piaware[557]: 57 msgs recv’d from dump1090-fa; 57 msgs sent to FlightAware
Sep 4 08:09:57 piaware piaware[557]: 472 msgs recv’d from dump1090-fa (415 in last 5m); 472 msgs sent to FlightAware
Sep 4 08:14:57 piaware piaware[557]: 825 msgs recv’d from dump1090-fa (353 in last 5m); 825 msgs sent to FlightAware
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Receiver status: connected
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Server status: clock unstable
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Receiver: 48.9 msg/s received 33.3 msg/s processed (68%)
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Results: 0.7 positions/minute
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Aircraft: 0 of 5 Mode S, 8 of 13 ADS-B used
Sep 4 08:19:57 piaware piaware[557]: 1228 msgs recv’d from dump1090-fa (403 in last 5m); 1228 msgs sent to FlightAware
Sep 4 08:24:57 piaware piaware[557]: 1706 msgs recv’d from dump1090-fa (478 in last 5m); 1706 msgs sent to FlightAware
Sep 4 08:29:57 piaware piaware[557]: 2173 msgs recv’d from dump1090-fa (467 in last 5m); 2173 msgs sent to FlightAware

2

You most likely have power issues. Are you using the official RPi power supply?

3

Yes, and my syslog doesn’t report any under voltage as I have seen with other raspberries in the past.

4

Just to be on the safeside, I changed the power supply for another official one that works 24/7 on a Pi 3 and the MLAT problem is still there.

5

check the cpu usage, i have the same problem & the cpu usage was maxed out

6

50%

I’m doing some tests and seems that feeding to ADSBEXCHANGE (which is receiving MLAT data) is preventinf FA from using MLAT.

I disabled sudo systemctl stop adsbexchange-feed and sudo systemctl stop adsbexchange-mlat and FA is now receiving MLAT again (at least for the last 10 minutes).

Loking now for a method to permanently uninstall ADBS excchange.

1 Like
7

It may just be a case of trying to do too much with a Zero. The USB system can get flaky under CPU load, and that will directly affect mlat clock stability.

1 Like
8

Or he is running a Kernel with the known problem regarding CPU usage which is already discussed here.

I would give the solution provided by wiedehopf a try:

9

My CPU usage stays around 45-50%.

My kermel is:

pi@piaware:/var/www/html $ hostnamectl
Static hostname: piaware
Icon name: computer
Machine ID: 3f51d129d8084753a7bfb67d19e014b7
Boot ID: c2dad6a2b0154c07801c784502aafcf4
Operating System: Raspbian GNU/Linux 10 (buster)
Kernel: Linux 5.4.51+
Architecture: arm

10

That’s your insufficient CPU or some other hardware limitation.

The adsbexchange feed scripts work fine alongside FA for more than enough forum members here.

It’s on the github page:
https://github.com/adsbxchange/adsb-exchange#removal--disabling-the-services

11

Sorry, I didn’t mean to blame the program, I’m just stating what is happening in my setup. There should be some hardware limitation, as you say. I’m running a Pi Zero W which I know is the basic Pi.

Just FYI, I re-enabled ADSBExchange and the problem appeared again immediately after about six hours since I disabled it. Disabling it corrected the problem again.

CPU usage should not be the problem. According to TOP it’s 35% idle and the PIDs belonging to ADSBExchange only add about 3% usage. :

top - 17:47:23 up 4:25, 1 user, load average: 1.26, 1.08, 1.02
Tasks: 106 total, 1 running, 105 sleeping, 0 stopped, 0 zombie
%Cpu(s): 44.9 us, 12.1 sy, 3.6 ni, 35.7 id, 0.0 wa, 0.0 hi, 3.6 si, 0.0 st
MiB Mem : 432.2 total, 153.0 free, 95.4 used, 183.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 281.4 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
516 dump1090 15 -5 29832 8128 2576 S 44.4 1.8 118:41.51 dump1090-fa
701 piaware 20 0 15864 9152 5544 S 1.6 2.1 4:49.56 fa-mlat-client
23262 adsbexc+ 19 -1 16360 9136 5348 S 1.6 2.1 0:04.34 mlat-client
530 piaware 20 0 25824 11608 6196 S 1.3 2.6 4:01.67 piaware
22665 root 20 0 0 0 0 I 1.3 0.0 0:03.13 kworker/u2:1-brcmf_wq/mmc1:0001:1
23634 pi 20 0 10176 3076 2572 R 1.3 0.7 0:00.56 top
23230 adsbexc+ 19 -1 34296 2948 2244 S 1.0 0.7 0:02.40 feed-adsbx
521 tar1090 39 19 7872 2232 1920 S 0.6 0.5 0:35.85 tar1090.sh
596 fr24 20 0 147232 14816 5908 S 0.6 3.3 3:39.68 fr24feed
23591 root 20 0 0 0 0 I 0.6 0.0 0:00.31 kworker/0:0-events
7 root 20 0 0 0 0 S 0.3 0.0 0:45.44 ksoftirqd/0
167 root 20 0 0 0 0 S 0.3 0.0 0:27.11 brcmf_wdog/mmc1
222 root 20 0 27640 1444 1304 S 0.3 0.3 1:01.86 rngd
625 www-data 20 0 9700 5744 3912 S 0.3 1.3 1:06.97 lighttpd
691 piaware 20 0 2980 2100 1800 S 0.3 0.5 0:53.19 faup1090
23670 tar1090 39 19 6440 1384 1320 S 0.3 0.3 0:00.01 sleep
1 root 20 0 32816 8184 6488 S 0.0 1.8 0:10.83 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kthreadd
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
8 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kdevtmpfs
9 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns
11 root 20 0 0 0 0 S 0.0 0.0 0:00.01 khungtaskd
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
13 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 writeback
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
28 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kblockd
29 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 blkcg_punt_bio
30 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdogd
31 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rpciod

Best regards,

12

Could easily be that the extra processing causes the voltage to drop enough so that the SDR starts losing samples.

As mentioned by foxhunter you should patch the system so dump1090-fa uses less CPU.

CPU usage isn’t constant, the fluctuations it can reach limits and that’s all that’s needed for MLAT instability.
Also you should check your coordinates on the FA page.

You might check the logs for undervoltage and also check the dump1090-fa logs:
Debug commands · wiedehopf/adsb-wiki Wiki · GitHub

13

I tried to patch the library as described by compiling it but unfortunately had to revert to the previous one because I wasnt able to even start FA and dump 1090.

14

Something wasn’t properly executed in this case somewhere. Don’t give up - there are several methods spelled out in the pertinent thread to fix the excess CPU use (which in my opinion matches the others’ conclusions).

If all else fails, start off with an older kernel and talk yourself out of performing an rpi-update or dist-upgrade since there is absolutely zero need to do so for the tasks you are contemplating with your Pi Zero. There is usually ZERO reason to keep updating anyhow - Microsoft has everyone brainwashed I swear it. If bleeding edge is required for some reason, then you’re likely advanced enough to work through it on the flip-side, but it’s not needed here and I doubt you are running any governments on the Pi Zero anyhow.

Keep plugging away mate, you’re on the right track.

1 Like
15

Thanks for the Hype!

I went thru the precompiled route to replace the library and my TOP idle is about 50% now up from 35%. dump1090-fa consumes around 42% now (I expected it would be much lower), but the good news is I enabled Adsbexchange again and it doesn’t seem to bring the clock unstability.

The only glitch now is that enabling STATS turned the clock unstable. Maybe there is th CPU threshold for my PiZero.

Thanks again.

16

Just an update.

Last week I updated to the new 4.0 PiAware and no more clock unstability issues have been logged for 6 days in a row now.

2 Likes
17

I too found that I was having “clock unstable” problems and therefore mlat was not working until I shutdown adsbexchange-{feed,mlat}, as you suggested. Now I’m sync’d with nearby receivers. I’m running piaware 3.8.1 on 32-bit Linux on VMware Player on an anemic ASUS EeePC 1005PEB (circa 2009) with an Intel Atom N450 CPU. Apparently, running the adsbexchange feeder software on this setup was consuming too much CPU. Thanks for the tip.

18

VMs have historically had problems with dropping some USB traffic from the dongle, which mlat really doesn’t like (it relies on receiving data at a constant rate for timing purposes); that may be contributing to your problem.

2 Likes
19

I have often experienced this MLAT issue when running Piaware on Debian / Ubuntu / Raspberry Pi OS for Mac & PC through Oracle VM. The USB devices, particularly DVB-T do not properly “passthrough” to VM.

1 Like
20

Thanks @obj and @abcd567. I am well aware of the challenges of running piaware on a VM. However, I’ve been running this setup for a long time without chronic mlat “clock unstable” problems (prior to adding adbsexchange services). It was happening occasionally prior to adding the adsbexchange feeder scripts. Only recently had I noticed that the problem had become chronic. Thanks to the suggestion of @nospamprl, after turning off the adsbexchange-feed and adsbexchange-mlat services, I noticed that the problem with “clock unstable” went back to happening only occasionally. I also noticed on my graphs1090 graphs that the overall CPU utilization went down slightly and the ADS-B CPU utilization went up slightly after I turned off the adsbexchange services. Apparently, these differences were enough to cause a reduction in dropped USB traffic from the dongle, such that the “clock unstable” problems went back to being occasional. I have removed the adsbexchange services. To those who claim that the adsbexchange services incur only “negligible” overhead, I beg to differ. One might argue that it doesn’t make sense to run piaware on a VM (some people find that antithetical), but I beg to differ on that also. It works OK. Not great, but it works well enough for me. Mlats are not always happening reliably, but they are happening at least some of the time.
What amazes me is the resilience of the piaware software. It can run even on a VM on a very anemic platform (an ASUS Eee PC Netbook from 2009 with an Intel Atom N450 CPU and 2GB memory running Windows 10 32-bit, aka “Starter Edition”). I’m running piaware 3.8.1 on 32-bit Linux with 512MB of memory allocated to the VM. For those of you who want to try this at home, note that VM Player 6 is the last version to support a 32-bit OS (and CPU). It is free for non-commercial use. VMware Player is much better than VirtualBox regarding its implementation of USB virtualization. I could never get mlats to work with VirtualBox.
Hey @abcd567, have you figured out the recipe for installing piaware 4.0 on 32-bit Linux? Is it worth the upgrade, i.e. do you think it might not work so well on an anemic platform? Does it require more or less compute resources compared to piaware 3.8.1?