MLAT Anomalies

I had the same issue Matthew, so I’ve done that :

sudo rpi-update

sudo rpi-update 52241088c1da59a359110d39c1875cda56496764

No need to reboot between the two rpi-update.

1 Like

I had the same issue too and carried out the above commands, very annoying as I didn’t initiate a firmware update. Sometimes it must come through with the normal packages…

In the process of getting my antenna back up and running, I also ran into MLAT timing errors. Downgrading the kernel resolved the issue. It’s a Pi 2 version 1.1, iirc.
Thanks for posting the fix!

As outlined in another post, I am having MLAT dropout issues with a Pi 2 and firmware/kernel, But I can not get rpi-update to downgrade the kernel. It writes the firmware, but not the kernel.

Using: sudo rpi-update 52241088c1da59a359110d39c1875cda56496764 [after removing .firmware_revision in /boot]

The message I get is:

*** Updating firmware
** *** As requested, not updating kernel
*** As requested, not updating kernel modules**
*** Updating VideoCore libraries
*** Using HardFP libraries
*** Updating SDK
*** Running ldconfig
*** Storing current firmware revision
*** Deleting downloaded files
*** Syncing changes to disk
*** If no errors appeared, your firmware was successfully updated to 52241088c1da59a359110d39c1875cda56496764
*** A reboot is needed to activate the new firmware

I don’t see a way to override the not updating kernel part. Ideas?

[Well, I hard coded rpi-update to ignore the customized kernel detection, SKIP_KERNEL=1 that was also hard coded in the bash script. It was ignoring SKIP_KERNEL=0 at the command line-----which is also default, anyway, if the customized kernel logic does not override it.]

pi@piaware:~$ uname -a
Linux piaware 4.4.50-v7+ #970 SMP Mon Feb 20 19:18:29 GMT 2017 armv7l GNU/Linux
pi@piaware:~$
restarting, seeing if problem goes away.

pi@piaware:~$ uname -a
Linux piaware 4.4.50-v7+ #970 SMP Mon Feb 20 19:18:29 GMT 2017 armv7l GNU/Linux

uname -a shows the currently running kernel, so you are already on the older 4.4 kernel there.

Sorry I was not clear. I am on the older kernel because I back dated it. To get that to work, I had to hard code the rpi-update command as indicated, then run it with the UPDATE_SELF option 0, so it did not overwrite my hard coding of SKIP_Kernel=0 (so it would overwrite a custom kernel). There might have been a better way, but that is how I did it.

Here is the problem, as seen on a Pi 2 with



Linux piaware 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
raspberrypi-kernel 1.20170703-1



This plots the difference in receive time for ADS-B messages seen by the Pi 2 compared to the same messages received by a beast-gps (running off a splitter from the same antenna)
The shallow slope is normal - this is the frequency difference between the clocks of the two receivers, which is somewhere around 0.7ppm in this case. The mlat server expects this and accounts for it.
The large jumps are not normal; they mean the Pi is silently dropping USB data and this is what causes mlat synchronization issues if it happens too often.

Trying to narrow down the cause now…

Does the same problem exist with RPI 3? I see sometimes MLAT anomalies… Most of time it works. I have 2.5A power supply, and working usb extension between pi and official dongle. Checked dmesg, no problems. I have stable wlan connection (-62dBm)

The problem appears to be specific to the Pi 2, it does not affect the 0W, 1B+, or 3B in my testing.

I raised a bug report with the raspberry pi kernel project: github.com/raspberrypi/linux/issues/2134

I have the same result; a pi2 with 4.9.36 (I think) was handling mlat for only about 3-5 minutes after a restart. Doing a backdate to the hash given above resulted in a 4.4 kernel that works fine. If the problem is in the kernel.org sources, one can tediously find it by doing a binary search between the versions in question, then carefully inspecting the changelogs. I don’t know if the pi usb drivers are in the central source tree or not, though; knowing Broadcom, they may well not be. (I actually have done this for other drivers, back in the 2.6.24 to 2.6.30 time… A major bother.) Eventually I’ll fix it by putting the adsb stuff on a pi3 and using the 2 for something slower like a weather station.

A related question for obj or someone is - if the clocking problem arises from the station position being off, (1) how much off matters for your solution method and accuracy, and (2) the time errors should affect results more in some directions than others giving an elliptical offset polar (or maybe off-center circle? the acceptable region should look like a figure 8). Is it possible to get this kind of information out of the mlat servers (probably lots more trouble than it’s worth, but knowing which direction and roughly how far ones location is off can help to fix it). In my case, the building picture shadows in the “choose your location” map let me pick out the part of my house to about 10 ft, (plenty of precision), and if the map is accurate enough to local geodetics position should not be a source of problems here. (12mhz sample clock means about 81 feet for each integer step. Enough averaging with moving targets (which we have) can refine the solution much finer; don’t know if you use that, though (thank Prof Kalman if so…))

Just for reference, the pi2 has plenty of cpu to run dump1090 and dump978 simultaneously (about 30% on one core for each), and there is enough power available from a 2.4 amp source (poe) to handle the pi with 2 (nooelec) sdr’s. (One could probably squeeze more cpu out by running rtprio and core affinity for the phase loop in each receiver, but at these levels it doesn’t seem necessary).

– Pete

The trouble is that the Pi kernel has a bunch of additional patches on top of the kernel.org sources; and they jumped from 4.4 to 4.9 which is where the problem surfaced. So it’s hard to bisect over that change. The Pi kernel developers seem interested in fixing the problem though which is promising!

A related question for obj or someone is - if the clocking problem arises from the station position being off, (1) how much off matters for your solution method and accuracy, and (2) the time errors should affect results more in some directions than others giving an elliptical offset polar (or maybe off-center circle? the acceptable region should look like a figure 8). Is it possible to get this kind of information out of the mlat servers (probably lots more trouble than it’s worth, but knowing which direction and roughly how far ones location is off can help to fix it). In my case, the building picture shadows in the “choose your location” map let me pick out the part of my house to about 10 ft, (plenty of precision), and if the map is accurate enough to local geodetics position should not be a source of problems here. (12mhz sample clock means about 81 feet for each integer step. Enough averaging with moving targets (which we have) can refine the solution much finer; don’t know if you use that, though (thank Prof Kalman if so…))

The underlying clock is more like 2.4MHz rather than 12, though the phase stuff lets you get a little more resolution there.

There are two things that the station position affects: the quality of clock sync and the quality of the mlat solutions themselves. I haven’t really stared at the problem too much but the clock sync effects are very dependent on which aircraft are chosen for sync. The sync error is maximized when those aircraft are travelling exactly along the line of the position error and minimized when they are travelling perpendicular; it’s something like a Doppler effect. In the mlat solution itself, errors in station position directly contribute to errors in the solution, so we just try to weed out those errors early on by looking at the quality of the clock sync (which is why mlat disables itself when there are sync problems)

The solver does use a Kalman filter with a CV model which works pretty well for anything that’s not maneuvering (and not very well for anything that is maneuvering - improving that is somewhere on my todo list…)

same setup on my R2

That’s the trick, my R2 is back to

  • Kernel version: Linux 4.4.50-v7+ armv7l
  • Firmware: #970

thanks … :smiley:

I’ve just put the older 4.4 kernel packages in the FlightAware apt repository. If you have this configured (piaware sdcards have this by default) then you can prevent upgrades past 4.4 by creating a file /etc/apt/preferences.d/50-pin-kernel.pref containing this:



Package: raspberrypi-kernel raspberrypi-bootloader libraspberrypi0 libraspberrypi-bin
Pin: origin "flightaware.com"
Pin-Priority: 995


You can check it’s correct via apt-cache policy:



$ sudo apt-get update
$ apt-cache policy raspberrypi-kernel
raspberrypi-kernel:
  Installed: 1.20170405-1
  Candidate: 1.20170405-1
  Package pin: 1.20170405-1
  Version table:
     1.20170703-1 995
        500 http://archive.raspberrypi.org/debian/ jessie/main armhf Packages
 *** 1.20170405-1 995
        500 http://flightaware.com/adsb/piaware/files/packages/ jessie/piaware armhf Packages
        100 /var/lib/dpkg/status


This is probably what I’ll do for the next piaware sdcard release.

I am having the issue with mlat suddenly not working. I recently moved to a Raspberry Pi 3 and hoped that would solve the problem, but it didn’t. I’ve found that if I change the setting for the antenna height and reboot, all is well for a while. So, I’ve been alternating between 9 feet and 10 feet each time I see the anomaly warning that mlat has stopped. I don’t have other software running on my Pi.
Ken

It sounds there are a couple things you can eliminate. 1. The kernel issue is only with the older models, so it won’t be your issue. 2. The MLAT calculations are not sensitive enough for 1 ft to cause a difference. I suspect every time you’re doing that, it’s simply resetting and giving your receiver another chance. It’s not really fixing the underlying problem. Are you using anything in-between the Pi and the USB dongle? Does the red LED stay solid red?

I’ve got a single USB 3 extension cord from Amazon that I use between the Pi and the dongle. It’s the extension cord that’s listed on the Piaware required/recommended equipment page. The red LED is on all the time as far as I know.

May be a coincidence, but seems that MLAT is much more stable with 4.9.43 kerenel (the latest from rpi-update).

Good news and bad news!

The good news is that the latest kernel available via rpi-update (4.9.46 at the time of posting) does indeed seem to work better. I’ve been running it on a Pi 2 for 45 mins with no timestamp jumps. 4.9.35 on the same hardware would usually break within 15 minutes.

The bad news is that 4.9.35 is still the most recent packaged kernel, so the piaware sdcard image will need to stick with the old 4.4 kernel for now until a more recent version is packaged.

1 Like

Upgrade to kernel 4.9.50 on raspbery 2 and working fine now…

Linux adsb 4.9.50-v7+ #1035 SMP Wed Sep 13 23:16:24 BST 2017 armv7l GNU/Linux
1 Like

Thanks for this its been doing my head in for weeks… I was on 4.9.35 and rebooting mlat would work for 10 to 15 minutes then come up with the timing errors. All the posts said it was location wrong or not enough receivers. I ran sudo apt-get install rpi-update then sudo rpi-update followed by a reboot and mlat has been fine since. Im now on 4.9.79