Required packages for running piaware, and more about mlats

FlightAware ADS-B Flight Tracking
105

Great :+1:
.

Good point.
I will try different USB ports on my desktop and see if it helps.

1 Like
106

Some USB ports go through a USB hub, and you don’t know which ones unless you look at the circuit board.

107

No problem, I will find out same way as a toddler finds “which button on remote makes the tv on” by hitting all buttons on remote one by one :smile:

1 Like
108

I declared victory prematurely. I am back to having the unstable clock problem.

I’m now using the 3rd USB port. Back in sync. Initially things look good.
…
Multilateration (MLAT): Supported / Enabled (synchronized with 309 nearby receivers)
…
What I notice is that after a reboot, there is a short spurt of being in sync with “nearby receivers”, and then the unstable clock problem recurs and persists. So it’s not the USB ports that are causing the problem. It’s the latencies and jitter inherent with this setup. Latency matters for MLATs.

1 Like
109

You may be interested in this thread where that very topic was discussed just a few months ago.

I suspect the support issues would be insurmountable.

Spend $US35 and buy a Pi, download the image and it will just work.

S.

2 Likes
110

This is very affordable and the best solution if feeding Flightaware and getting mlat feed-back is the sole objective. In this case RPi is the winner.

However if the objective is exploration and experimentation, then there are lot of other options to try.

3 Likes
111

Join Starfleet.

Worked out really well for Marie Curie.

Which is why we have hobbies to enjoy.

S

1 Like
112

Some PC’s have more USB ports, connected to different chipsets internally. Some connected directly to the SouthBridge, some connected to a interposer chip, connected to one of the PCIe 1x lanes.
But on the laptops with limited number of ports, that is not an issue usually.

1 Like
113

Indeed, that doesn’t seem to have been the issue. I had been changing USB ports and rebooting after each change, and at first I (incorrectly) assumed that it was the different port that was allowing mlats to succeed. This turned out to be a false assumption. The issue is that when you run piaware in a VM on a host with a very slow CPU (e.g., Atom), there are lots of latencies and jitter (variable latencies) that quickly lead to the “unstable clock” problem. I wonder if there is some way to correct for this in software. What I noticed is that right after rebooting the VM, initially it is in sync with nearby receivers. But then it quickly (within a short matter of minutes) goes out of sync. Would it be possible for the software to “re-sync” with other nearby receivers periodically in order to correct for this? That is, if a node starts getting “unstable clock” conditions, it calls a “re-sync” routine that is less disruptive than a reboot. Is this doable in software? Since I built from source, I could try this out if someone tells me what to change.

114

Mlat isn’t hugely dependent on CPU speed (you need enough to actually run the software, but I’d have thought an Atom would be OK). It counts samples from the dongle, so as long as data is not dropped, latency per se doesn’t matter (as long as it’s under a second or so)

The “works for a bit then goes unstable” is most likely because it takes a while for the server to notice that the timing is regularly unreliable. Occasional dropped data (e.g. maybe once an hour) is fairly normal for these dongles and the server will just deal with it by resetting all the clock synchronization info and starting over on the synchronization process. Once it’s done that a few times in quick succession, though, it starts to distrust the data entirely and will report “clock unstable” to tell you about it. It takes a longer period of good data to exit that state and if you’re constantly dropping data it’ll never get there.

2 Likes
115

Thanks, that’s interesting. Don’t forget that this case isn’t an Atom running piaware directly. It is an Atom running a VM running piaware.

So if it isn’t the slow CPU, what is causing data to be dropped? Is there any way to periodically clear “history” so as to quickly recover from getting into this state? I understand that the root cause is dropped data, so this would be a workaround to that underlying problem. Or is this just a bad idea?

116

USB bus noise (there’s no mechanism for the dongle to retransmit damaged messages, they’re just dropped)

USB stack problems (e.g. there were some specific kernel versions that would tend to drop lots of USB data, subsequently fixed). VM USB problems are similar.

Active USB repeater cables sometimes cause problems.

On a Pi, marginal power can cause USB problems.

Or sometimes you just get a dongle that’s misbehaving.

If data is being dropped regularly, the timing info isn’t usable for mlat, so there’s no real workaround here.

2 Likes
117

Thanks very much. That answers that.

Interesting discussion, guys. You are all great. This is an excellent community of smart folks.

I think it’s interesting to test the limits of hardware/software to accomplish a particular task, including taking things to extremes in order to further understanding.

118

Very few will be interested in this.
Some may even hate this endeavor of yours, and tell you “Join Starfleet” and “Worked out really well for Marie Curie” :slight_smile:

1 Like
119

I’m not sure about the “very few” part, but I do realize that not everyone shares this interest that you and I seem to have. That’s OK!!

I find it interesting that my “insane” setup is good enough to plot a good number of planes on my skyview, but not good enough to make mlats work. From @obj’s message, and without seeing the server code (which is not open-source, I presume), I believe what he’s saying is that if the server doesn’t see a steady, uninterrupted, timely (not too stale, i.e. there isn’t a big time lag in the data with respect to other nearby reporting stations) stream of data for a flight without ADS-B data, then that data stream is considered untrustworthy and is marked as having an “unstable clock” (which seems like a bit of a misnomer, but perhaps not). So “dropped” USB data, whether it is really dropped, garbled, or overly delayed – is the issue here. Mlats seem to be the only part of this system that involve real-time constraints on the server.

Isn’t it the case that all aircraft are supposed to have ADS-B starting in 2020, and if so – I’m assuming that mlats will not be as important (though still necessary as there will not likely be 100% compliance for some time).

120

The FA server code is not open-source but it is conceptually similar to the open-source prototype I built here: GitHub - mutability/mlat-server: Mode S multilateration server

Without going into a ton of detail, the basic thing that the server is doing for synchronization is the code here: mlat-server/mlat/server/clocktrack.py at master · mutability/mlat-server · GitHub
which takes as input a pair of ADS-B position messages seen by the same two receivers, and uses the arrival times as measured on each receiver to establish a correlation between the two receiver clocks (absolute offset and relative clock speed). Later, that correlation can be used to compare the arrival times of non-ADS-B messages for multilateration.

The critical thing to note here is that the “arrival time” clock on each (dump1090-based) receiver is just a scaled free-running count of received samples (which should be arriving at a steady rate of approximately 2.4MHz), not anything anchored to real time. So any dropped sample data makes it look like that receiver’s clock suddenly stopped for a while which throws off the synchronization of the receiver pair, since the synchronization assumes that the clock characteristics are stable in the short term and don’t jump around a lot.

That is a US-specific deadline, and it’s not all aircraft. The ADS-B mandate will definitely help but I expect multilateration to continue to be necessary for a good while yet.

4 Likes
121

Thanks, @obj!! This is very interesting indeed. You certainly are very good at explaining things. It’s also a good thing how you engage with the piaware community, so people don’t try silly things like trying to “fake out” the server with bad data (my earlier thought of resetting the USB stream periodically to try to get back in “good graces” with the server). Now that you’ve made it clear how the mlat algorithm works on the server, it is clear that would have been a very bad idea. I had mistakenly assumed that my faulty feed started out in sync and then drifted out of sync. Now I understand that it really never was in sync to begin with, but was initially trusted by the server until proven otherwise.

122

Piaware on VMware on Windows 8.1

About an hour ago finished installing dump1090-fa + piaware on VMware.
No mlat :frowning:

abcd@ubuntu:~$ journalctl -u piaware | grep 'Receiver status\|Server status'

Mar 06 09:29:44 ubuntu piaware[11989]: mlat-client(12011): Receiver status: connected
Mar 06 09:29:44 ubuntu piaware[11989]: mlat-client(12011): Server status:   clock unstable
Mar 06 09:44:44 ubuntu piaware[11989]: mlat-client(12011): Receiver status: connected
Mar 06 09:44:44 ubuntu piaware[11989]: mlat-client(12011): Server status:   clock unstable
Mar 06 09:59:44 ubuntu piaware[11989]: mlat-client(12011): Receiver status: connected
Mar 06 09:59:44 ubuntu piaware[11989]: mlat-client(12011): Server status:   clock unstable
Mar 06 10:14:45 ubuntu piaware[11989]: mlat-client(12011): Receiver status: connected
Mar 06 10:14:45 ubuntu piaware[11989]: mlat-client(12011): Server status:   clock unstable

.
image

1 Like
123

Same as my experience. I think it’s clear now what’s going on now. Dropped USB data as seen by the server. It occurs to me this could be a networking issue also. BTW, this didn’t happen with the latest VMware Player (version 15) on a decent laptop. So either the older version (VMP version 6) or your old laptop, or both – are dropping some USB data. From @obj’s excellent explanations, I’m guessing it may be the USB stack in VMP version 6 (a rather old version). But it could be your laptop, or some combination of the two.

124

As the CPU of my Desktop is not so bad, I think it is VM pass through which is causing problem.

I have now done following:

  1. Detached dongle from VMWare so that the dongle is now with host (Windows 8.1).

  2. Downloaded dump1090-win.1.10.3010.14 (which is included in dump1090-master.zip at dump1090 Malcolm Robb fork), on Windows Desktop and running it by its batch file.

  3. Edited file /etc/default/dump1090-fa and in line starting with RECEIVER_OPTIONS, replaced --device-index 0 by --net-only and running it in --net-only mode, fetching data from dump1090-win.1.10.3010.14

  4. Created a new file net-feed.sh to establish a data pipe between dump1090 on Windows and dump1090-fa on VMware Ubuntu

    sudo apt install netcat
sudo touch net-feed.sh
sudo chmod +x net-feed.sh
sudo nano net-feed.sh

    Copy-pasted following in file net-feed.sh

    #!/bin/sh
while true
  do
    sleep 30
    /bin/nc 192.168.0.10 30005 | /bin/nc 127.0.0.1 30004
  done

  5. Added entry in crontab to start the file net-feed.sh at boot

    sudo crontab -e

    Chose editor option 1 (nano )
    Added this line at bottom

    @reboot /bin/bash /home/abcd/net-feed.sh

    Saved (Ctrl+o) & closed (Ctrl+x)

  6. Rebooted Ubuntu

sudo reboot

.

.

abcd@ubuntu:~$ journalctl -u piaware | grep 'Receiver status\|Server status'

Mar 06 11:27:43 ubuntu piaware[722]: mlat-client(968): Receiver status: connected
Mar 06 11:27:43 ubuntu piaware[722]: mlat-client(968): Server status:   synchronized with 35 nearby receivers

.

image
image758×402 17.9 KB

.
.
image

3 Likes