FlightAware Discussions

Lost MLAT?

Hello,

One week ago I installed my first Piaware with a Prostick Plus dongle on a PiZero. Everything worked fine for a week.

Yesterday I noticed (without making any change in my system, that MLAT was no longer working and reporting “Clock Unstable” or “MLAT disabled/not synked/Lat Long not configured”.

My coordinates are configured and precise (with a GPS) and were working for the first week. MLAT is enabled.

I just disconnected and reconnected everything and it worked for 2-3 minutes, even reporting a few positions, but went down again. ADSB is working fine, so I assume it’s not a USB/Dongle problem.

My location (Mexico City) does not have a lot of other stations.(9 within 10 miles according to my stats page, 6 of which show the green check under the MLAT column). Sometimes (very few) I’ve seen a message in the “not synchronized with any nearby receivers” in the log.

Any suggestions?

Thanks

PD Here is my last hor log:

Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Receiver status: connected
Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Server status: clock unstable
Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Receiver: 39.9 msg/s received 28.7 msg/s processed (72%)
Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server
Sep 4 07:01:45 piaware piaware[525]: mlat-client(32356): Aircraft: 0 of 3 Mode S, 6 of 8 ADS-B used
Sep 4 07:05:09 piaware piaware[525]: 49464 msgs recv’d from dump1090-fa (332 in last 5m); 49464 msgs sent to FlightAware
Sep 4 07:10:09 piaware piaware[525]: 49866 msgs recv’d from dump1090-fa (402 in last 5m); 49866 msgs sent to FlightAware
Sep 4 07:15:09 piaware piaware[525]: 50198 msgs recv’d from dump1090-fa (332 in last 5m); 50198 msgs sent to FlightAware
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Receiver status: connected
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Server status: clock unstable
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Receiver: 45.5 msg/s received 32.8 msg/s processed (72%)
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server
Sep 4 07:16:46 piaware piaware[525]: mlat-client(32356): Aircraft: 1 of 1 Mode S, 4 of 12 ADS-B used
Sep 4 07:20:09 piaware piaware[525]: 50545 msgs recv’d from dump1090-fa (347 in last 5m); 50545 msgs sent to FlightAware
Sep 4 07:25:09 piaware piaware[525]: 50884 msgs recv’d from dump1090-fa (339 in last 5m); 50884 msgs sent to FlightAware
Sep 4 07:30:09 piaware piaware[525]: 51179 msgs recv’d from dump1090-fa (295 in last 5m); 51179 msgs sent to FlightAware
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Receiver status: connected
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Server status: clock unstable
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Receiver: 41.3 msg/s received 29.0 msg/s processed (70%)
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server
Sep 4 07:31:46 piaware piaware[525]: mlat-client(32356): Aircraft: 1 of 4 Mode S, 9 of 10 ADS-B used
Sep 4 07:35:09 piaware piaware[525]: 51461 msgs recv’d from dump1090-fa (282 in last 5m); 51461 msgs sent to FlightAware
Sep 4 07:40:09 piaware piaware[525]: 51731 msgs recv’d from dump1090-fa (270 in last 5m); 51731 msgs sent to FlightAware
Sep 4 07:42:47 piaware piaware[525]: NOTICE adept server is shutting down. reason: reconnection request received
Sep 4 07:42:49 piaware piaware[525]: Lost connection to adept server at piaware.flightaware.com/1200: server closed connection
Sep 4 07:42:49 piaware piaware[525]: multilateration data no longer required, disabling mlat client
Sep 4 07:42:50 piaware piaware[525]: fa-mlat-client exited normally
Sep 4 07:42:50 piaware piaware[525]: reconnecting in 5 seconds…
Sep 4 07:42:50 piaware piaware[525]: mlat-client(32356): Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data
Sep 4 07:42:50 piaware piaware[525]: mlat-client(32356): Exiting on connection loss
Sep 4 07:42:55 piaware piaware[525]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Sep 4 07:42:55 piaware piaware[525]: Connection with adept server at piaware.flightaware.com/1200 established
Sep 4 07:42:55 piaware piaware[525]: TLS handshake with adept server at piaware.flightaware.com/1200 completed
Sep 4 07:42:55 piaware piaware[525]: FlightAware server certificate validated
Sep 4 07:42:55 piaware piaware[525]: encrypted session established with FlightAware
Sep 4 07:42:58 piaware piaware[525]: logged in to FlightAware as user nospamprl
Sep 4 07:42:58 piaware piaware[525]: my feeder ID is 168b5d52-eacc-4f55-a9cf-a11261fb8094
Sep 4 07:42:58 piaware piaware[525]: site statistics URL: https://flightaware.com/adsb/stats/user/nospamprl#stats-134282
Sep 4 07:42:58 piaware piaware[525]: multilateration data requested
Sep 4 07:42:59 piaware piaware[525]: Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,30105 --results ext_basestation,listen,30106 --udp-transport 70.42.6.198:18011:3988778853
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): fa-mlat-client 0.2.11 starting up
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Using UDP transport to 70.42.6.198 port 18011
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Listening for Beast-format results connection on port 30105
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Listening for Extended Basestation-format results connection on port 30106
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Route MTU changed to 1500
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Input connected to localhost:30005
Sep 4 07:43:02 piaware piaware[525]: mlat-client(24314): Input format changed to BEAST, 12MHz clock
Sep 4 07:43:03 piaware piaware[525]: mlat-client(24314): Beast-format results connection with ::1:30104: connection established
Sep 4 07:45:09 piaware piaware[525]: 52012 msgs recv’d from dump1090-fa (281 in last 5m); 52006 msgs sent to FlightAware
Sep 4 07:50:09 piaware piaware[525]: 52329 msgs recv’d from dump1090-fa (317 in last 5m); 52323 msgs sent to FlightAware
Sep 4 07:55:09 piaware piaware[525]: 52697 msgs recv’d from dump1090-fa (368 in last 5m); 52691 msgs sent to FlightAware
Sep 4 08:04:18 piaware piaware[557]: creating pidfile /run/piaware/piaware.pid
Sep 4 08:04:18 piaware piaware[557]: ****************************************************
Sep 4 08:04:18 piaware piaware[557]: piaware version 3.8.1 is running, process ID 557
Sep 4 08:04:18 piaware piaware[557]: your system info is: Linux piaware 5.4.51+ #1333 Mon Aug 10 16:38:02 BST 2020 armv6l GNU/Linux
Sep 4 08:04:20 piaware piaware[557]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Sep 4 08:04:20 piaware piaware[557]: Connection with adept server at piaware.flightaware.com/1200 established
Sep 4 08:04:20 piaware piaware[557]: TLS handshake with adept server at piaware.flightaware.com/1200 completed
Sep 4 08:04:23 piaware piaware[557]: FlightAware server certificate validated
Sep 4 08:04:23 piaware piaware[557]: encrypted session established with FlightAware
Sep 4 08:04:27 piaware piaware[557]: ADS-B data program ‘dump1090-fa’ is listening on port 30005, so far so good
Sep 4 08:04:27 piaware piaware[557]: Starting faup1090: /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat 19.421 --lon -99.234
Sep 4 08:04:27 piaware piaware[557]: Started faup1090 (pid 732) to connect to dump1090-fa
Sep 4 08:04:27 piaware piaware[557]: UAT support disabled by local configuration setting: uat-receiver-type
Sep 4 08:04:28 piaware piaware[557]: adept reported location: 19.42072, -99.23367, 7764ft AMSL
Sep 4 08:04:28 piaware piaware[557]: logged in to FlightAware as user nospamprl
Sep 4 08:04:28 piaware piaware[557]: my feeder ID is 168b5d52-eacc-4f55-a9cf-a11261fb8094
Sep 4 08:04:28 piaware piaware[557]: site statistics URL: https://flightaware.com/adsb/stats/user/nospamprl#stats-134282
Sep 4 08:04:28 piaware piaware[557]: multilateration data requested
Sep 4 08:04:28 piaware piaware[557]: Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --input-type dump1090 --results beast,connect,localhost:30104 --results beast,listen,30105 --results ext_basestation,listen,30106 --udp-transport 70.42.6.232:10006:4229442807
Sep 4 08:04:29 piaware piaware[557]: piaware received a message from dump1090-fa!
Sep 4 08:04:29 piaware piaware[557]: piaware has successfully sent several msgs to FlightAware!
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): fa-mlat-client 0.2.11 starting up
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Using UDP transport to 70.42.6.232 port 10006
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Listening for Beast-format results connection on port 30105
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Listening for Extended Basestation-format results connection on port 30106
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Route MTU changed to 1500
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Input connected to localhost:30005
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Input format changed to BEAST, 12MHz clock
Sep 4 08:04:32 piaware piaware[557]: mlat-client(742): Beast-format results connection with ::1:30104: connection established
Sep 4 08:04:57 piaware piaware[557]: 57 msgs recv’d from dump1090-fa; 57 msgs sent to FlightAware
Sep 4 08:09:57 piaware piaware[557]: 472 msgs recv’d from dump1090-fa (415 in last 5m); 472 msgs sent to FlightAware
Sep 4 08:14:57 piaware piaware[557]: 825 msgs recv’d from dump1090-fa (353 in last 5m); 825 msgs sent to FlightAware
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Receiver status: connected
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Server status: clock unstable
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Receiver: 48.9 msg/s received 33.3 msg/s processed (68%)
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.3kB/s UDP to server
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Results: 0.7 positions/minute
Sep 4 08:19:32 piaware piaware[557]: mlat-client(742): Aircraft: 0 of 5 Mode S, 8 of 13 ADS-B used
Sep 4 08:19:57 piaware piaware[557]: 1228 msgs recv’d from dump1090-fa (403 in last 5m); 1228 msgs sent to FlightAware
Sep 4 08:24:57 piaware piaware[557]: 1706 msgs recv’d from dump1090-fa (478 in last 5m); 1706 msgs sent to FlightAware
Sep 4 08:29:57 piaware piaware[557]: 2173 msgs recv’d from dump1090-fa (467 in last 5m); 2173 msgs sent to FlightAware

You most likely have power issues. Are you using the official RPi power supply?

Yes, and my syslog doesn’t report any under voltage as I have seen with other raspberries in the past.

Just to be on the safeside, I changed the power supply for another official one that works 24/7 on a Pi 3 and the MLAT problem is still there.

check the cpu usage, i have the same problem & the cpu usage was maxed out

50%

I’m doing some tests and seems that feeding to ADSBEXCHANGE (which is receiving MLAT data) is preventinf FA from using MLAT.

I disabled sudo systemctl stop adsbexchange-feed and sudo systemctl stop adsbexchange-mlat and FA is now receiving MLAT again (at least for the last 10 minutes).

Loking now for a method to permanently uninstall ADBS excchange.

It may just be a case of trying to do too much with a Zero. The USB system can get flaky under CPU load, and that will directly affect mlat clock stability.

Or he is running a Kernel with the known problem regarding CPU usage which is already discussed here.

I would give the solution provided by wiedehopf a try:

My CPU usage stays around 45-50%.

My kermel is:

pi@piaware:/var/www/html $ hostnamectl
Static hostname: piaware
Icon name: computer
Machine ID: 3f51d129d8084753a7bfb67d19e014b7
Boot ID: c2dad6a2b0154c07801c784502aafcf4
Operating System: Raspbian GNU/Linux 10 (buster)
Kernel: Linux 5.4.51+
Architecture: arm

That’s your insufficient CPU or some other hardware limitation.

The adsbexchange feed scripts work fine alongside FA for more than enough forum members here.

It’s on the github page:
https://github.com/adsbxchange/adsb-exchange#removal--disabling-the-services

Sorry, I didn’t mean to blame the program, I’m just stating what is happening in my setup. There should be some hardware limitation, as you say. I’m running a Pi Zero W which I know is the basic Pi.

Just FYI, I re-enabled ADSBExchange and the problem appeared again immediately after about six hours since I disabled it. Disabling it corrected the problem again.

CPU usage should not be the problem. According to TOP it’s 35% idle and the PIDs belonging to ADSBExchange only add about 3% usage. :

top - 17:47:23 up 4:25, 1 user, load average: 1.26, 1.08, 1.02
Tasks: 106 total, 1 running, 105 sleeping, 0 stopped, 0 zombie
%Cpu(s): 44.9 us, 12.1 sy, 3.6 ni, 35.7 id, 0.0 wa, 0.0 hi, 3.6 si, 0.0 st
MiB Mem : 432.2 total, 153.0 free, 95.4 used, 183.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 281.4 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
516 dump1090 15 -5 29832 8128 2576 S 44.4 1.8 118:41.51 dump1090-fa
701 piaware 20 0 15864 9152 5544 S 1.6 2.1 4:49.56 fa-mlat-client
23262 adsbexc+ 19 -1 16360 9136 5348 S 1.6 2.1 0:04.34 mlat-client
530 piaware 20 0 25824 11608 6196 S 1.3 2.6 4:01.67 piaware
22665 root 20 0 0 0 0 I 1.3 0.0 0:03.13 kworker/u2:1-brcmf_wq/mmc1:0001:1
23634 pi 20 0 10176 3076 2572 R 1.3 0.7 0:00.56 top
23230 adsbexc+ 19 -1 34296 2948 2244 S 1.0 0.7 0:02.40 feed-adsbx
521 tar1090 39 19 7872 2232 1920 S 0.6 0.5 0:35.85 tar1090.sh
596 fr24 20 0 147232 14816 5908 S 0.6 3.3 3:39.68 fr24feed
23591 root 20 0 0 0 0 I 0.6 0.0 0:00.31 kworker/0:0-events
7 root 20 0 0 0 0 S 0.3 0.0 0:45.44 ksoftirqd/0
167 root 20 0 0 0 0 S 0.3 0.0 0:27.11 brcmf_wdog/mmc1
222 root 20 0 27640 1444 1304 S 0.3 0.3 1:01.86 rngd
625 www-data 20 0 9700 5744 3912 S 0.3 1.3 1:06.97 lighttpd
691 piaware 20 0 2980 2100 1800 S 0.3 0.5 0:53.19 faup1090
23670 tar1090 39 19 6440 1384 1320 S 0.3 0.3 0:00.01 sleep
1 root 20 0 32816 8184 6488 S 0.0 1.8 0:10.83 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.03 kthreadd
6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 mm_percpu_wq
8 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kdevtmpfs
9 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 netns
11 root 20 0 0 0 0 S 0.0 0.0 0:00.01 khungtaskd
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
13 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 writeback
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
28 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kblockd
29 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 blkcg_punt_bio
30 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdogd
31 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rpciod

Best regards,

Could easily be that the extra processing causes the voltage to drop enough so that the SDR starts losing samples.

As mentioned by foxhunter you should patch the system so dump1090-fa uses less CPU.

CPU usage isn’t constant, the fluctuations it can reach limits and that’s all that’s needed for MLAT instability.
Also you should check your coordinates on the FA page.

You might check the logs for undervoltage and also check the dump1090-fa logs:
https://github.com/wiedehopf/adsb-wiki/wiki/Debug-commands#undervoltage

I tried to patch the library as described by compiling it but unfortunately had to revert to the previous one because I wasnt able to even start FA and dump 1090.

Something wasn’t properly executed in this case somewhere. Don’t give up - there are several methods spelled out in the pertinent thread to fix the excess CPU use (which in my opinion matches the others’ conclusions).

If all else fails, start off with an older kernel and talk yourself out of performing an rpi-update or dist-upgrade since there is absolutely zero need to do so for the tasks you are contemplating with your Pi Zero. There is usually ZERO reason to keep updating anyhow - Microsoft has everyone brainwashed I swear it. If bleeding edge is required for some reason, then you’re likely advanced enough to work through it on the flip-side, but it’s not needed here and I doubt you are running any governments on the Pi Zero anyhow.

Keep plugging away mate, you’re on the right track.

1 Like

Thanks for the Hype!

I went thru the precompiled route to replace the library and my TOP idle is about 50% now up from 35%. dump1090-fa consumes around 42% now (I expected it would be much lower), but the good news is I enabled Adsbexchange again and it doesn’t seem to bring the clock unstability.

The only glitch now is that enabling STATS turned the clock unstable. Maybe there is th CPU threshold for my PiZero.

Thanks again.