modesmixer2 and MLAT data on port30004

FlightAware ADS-B Flight Tracking
1

Piaware has recently added the capability of doing MLAT processing, I decided to see if I could get this working with my modesmixer2 setup that is receiving serial over USB data from a Mode-S Beast receiver rather than than a dongle using dump1090. That setup is working very well, feeding data to the PlanePlotter servers via modesmixer2 and ppup1090 on port 30005.

The basic flow of the new Piaware MLAT system is here:

http://flightaware.com/adsb/piaware/about

My current modesmixer2 invocation is with:


modesmixer2 --inSerial /dev/ttyUSB0:3000000:hardware --outServer beast:30005 --web 8080 --location yy.yyyyyy:-xx.xxxxxx

This simply receives serial data (already decoded) in Beast binary format on USB serial port ttyUSB0 and translates it to an output on port 30005. I also start the web server and provide my location for modesmixer web page plotting of my coverage and stats and for more efficient position decoding.

Piaware used to have problems receiving data for normal position reporting on port 30005 from modesmixer2, even though that was the same port used by the expected dump1090 decoder application. However, that issue seems to have been resolved with the latest MLAT-capable version of Piaware. piaware_2.1-2_armhf.deb.

Here are the relevant piaware log messages showing things are starting up just fine with modesmixer2:


07/27/2015 01:36:41 ADS-B data program 'modesmixer2' is listening on port 30005, so far so good
07/27/2015 01:36:41 Starting faup1090: /usr/bin/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --lat 41.911 --lon -88.357
07/27/2015 01:36:41 Started faup1090 (pid 2311) to connect to modesmixer2
07/27/2015 01:36:41 logged in to FlightAware as user donf99
07/27/2015 01:36:41 multilateration support enabled (use piaware-config to disable)
07/27/2015 01:36:41 multilateration data requested, enabling mlat client
07/27/2015 01:36:41 Starting multilateration client: /usr/lib/fa-mlat-client/fa-mlat-client --input-connect localhost:30005 --udp-transport 70.42.6.203:6288:4111205693
07/27/2015 01:36:45 piaware received a message from modesmixer2!
07/27/2015 01:36:45 piaware has successfully sent several msgs to FlightAware!
07/27/2015 01:36:46 mlat(2313): fa-mlat-client 0.2.3 starting up
07/27/2015 01:36:46 mlat(2313): Using UDP transport to 70.42.6.203:6288
07/27/2015 01:36:46 mlat(2313): Input connected to localhost:30005
07/27/2015 01:37:11 141 msgs recv'd from modesmixer2; 141 msgs sent to FlightAware
07/27/2015 01:37:59 server is sending alive messages; we will expect them
07/27/2015 01:42:11 1972 msgs recv'd from modesmixer2 (1831 in last 5m); 1972 msgs sent to FlightAware
07/27/2015 01:47:11 3874 msgs recv'd from modesmixer2 (1902 in last 5m); 3874 msgs sent to FlightAware
07/27/2015 01:51:46 mlat(2313): Receiver connection: ready
07/27/2015 01:51:46 mlat(2313): Server connection:   ready
07/27/2015 01:51:46 mlat(2313): Receiver:  942.3 msg/s received     15.7kB/s from receiver
07/27/2015 01:51:46 mlat(2313): Server:      2.7 kB/s from server    0.0kB/s TCP to server  10.0kB/s UDP to server
07/27/2015 01:51:46 mlat(2313): Results:  1259.9 positions/minute
07/27/2015 01:51:46 mlat(2313): Aircraft: 140 known, 132 requested by server
07/27/2015 01:52:11 5746 msgs recv'd from modesmixer2 (1872 in last 5m); 5746 msgs sent to FlightAware

If you examine the FlightAware flow disgram, you will see that normal sharing data is received from modesmixer2 on port 30005, the same port used by PlanePlotter and ppup1090 for autonomous uploading to PlanePlotter servers. However, Piaware also has a back-channel on port 30004 that Piaware normally uses to send MLAT positions back to input port 30004 on dump1090 in Beast binary format. Dump1090 can display the MLAT positions on its own web server with distinct color coding.

MLAT on Piaware can be enabled with or with out the backchannel for reporting Piawares own MLAT positions with the configuration command:

piaware-config -mlat 0/1 (enables/disables MLAT processing. Starts the MLAT process if request received from Flightaware servers.)

piaware-config -mlatResults 0/1 (enables/disables sendin MLAT position data on port 30004.)

piaware-config -mlatResultsFormat formatlist (Poorly documented. Seems to allow afjusting MLAT data on 30004 in other than Beat binary.)

I tried enabling reception of the MLAT data in modesmixer 2 by defining an additional input with this invocation:


modesmixer2 --inSerial /dev/ttyUSB0:3000000:hardware --inConnect 127.0.0.1:30004 --outServer beast:30005 --web 8080 --location yy.yyyyyy:-xx.xxxxxx

However, when I enable the MLAT back channel on Piaware, I receive port 30004 “Connction Refused” errors in both the modes2mixer web page log and the Piaware log.

It seems modesmixer2 should be able to detect the secondary position reports on port 30004, as they are in the common Beast binary format, according to documentation on Piaware.

Has anyone come up with a way of combining the Piaware MLAT positions in the output stream of modesmixer2?

Thanks and regards,

Don
WD9DMP

2

Nice summary! And that’s an impressive result rate…

I would guess that you want “–inServer” not “–inConnect”.

Be aware that the synthetic mlat results are likely to be forwarded back out on port 30005 (this doesn’t affect mlat or piaware, they know to ignore them, but it might affect anything else you have connected)

3

Well, the “–inServer 30004” did the trick! “–inConnect” was the incorrect modesmixer2 switch.

However, as you warned, there were some undesirable side-effects.

After setting up the reception of MLAT positions from Piaware on modesmixer2, port 30004, the data was being merged with the serial input from my Mode-S Beast receiver. I viewed this data on PlanePlotter via TCP port 30005 and saw an explosion of additional planes from Piaware MLAT! Apparently, the MLAT system is working quite well. Interestingly, the new planes actually show a signal level in PlanePlotter, even though the positions are not being received by an actual receiver. I have no idea why. MLAT planes were identifiable by the somewhat irregular flight path as plotted by PlanePlotter.

What I am concerned about is that the intermingling of MLAT data with Beast receiver data on port 30005 (which is also feeding ppup1090 to the PlanePlotter servers and providing PlanePlotter MLAT services) will likely make the combined data stream unusable for PlanePlotter MLAT fixes, and perhaps normal position reports as well. I suspect the time tags from the Beast are being completely mangled by the MLAT position reports from Piaware. Although Piaware can filter out its own MLAT data from 30005, ppup1090 has no way of doing so.

So, I returned to my original settings, which provide MLAT service to FlightAware and Planeplotter without corrupting the raw datastream from the Mode-S Beast receiver.

An interesting experiment!

Best regards,

Don
WD9DMP

4

I’d guess that planeplotter is getting the signal level from the (positionless) Mode S messages.

The synthetic mlat messages use a single, fixed, timestamp (this is how you can identify them), which does indeed break the timestamp sequencing and is likely to confuse things…

If you don’t mind a more complex setup you could do something like this and run a couple of modesmixers:

modesmixer2 #1: --inSerial /dev/ttyUSB0:3000000:hardware --outServer beast:30005
modesmixer2 #2: --inConnect localhost:30005 --inServer 30004 --web 8080

i.e. #1 processes only real on-the-air data and can forward to planeplotter etc
#2 gets both real data (from #1) and mlat data (from the port 30004 backchannel) and provides the web display
data never flows from #2 to #1
you could add another --outServer on a different port on #2 if you wanted to be able to selectively connect some things to the combined feed

Or you could get in touch with the modesmixer2 author and ask for an option to disable mlat forwarding :slight_smile:
You can refer him here for the details of how to identify the mlat messages: github.com/mutability/mlat-clie … ut.py#L282

5

I believe you are right that the signal level on MLAT aircraft was coming from the positionless Mode-S data from my receiver. When I disabled the output of MLAT data on 30004 and restarted Piaware, the MLAT positions stopped updating on the PlanePlotter screen. However, before they timed out and vanished from the display, altitude and signal strength were still updating on the chart display.

I have posted this on the modesmixer2 board on radarspotting.com, so perhaps this will make it on the list of enhancements.

I may try the multiple instances of modesmixer2. Good idea. I am somewhat worried about running out of CPU capacity on my Pi v2. The Piaware MLAT process takes up quite a bit of CPU resources. I needed to disable Mode-A/C decoding on the Beast receiver to avoid overloading the CPU cores when running Piaware MLAT, ppup1090, PlanePlotter, and Piaware.

Best,

Don
WD9DMP

6

This combination allows feeding the PlanePlotter servers via ppup1090 with normal sharing, as well as servicing MLAT requests, while allowing PlanePlotter to display local receiver positions, PlanePlotter normal sharing positions, PlanePlotter MLAT positions, and Piaware MLAT positions, without corrupting the data stream to ppup1090.

You MUST put PlanePlotter in download-only mode!!! You then configure the PlanePlotter TCP connection to point to the output of the second modesmixer2 instance on port 30015. To see normal PlanePlotter only plots, point to port 30005 instead of 30015.

ppup1090 is fed from port 30005 with the stream only from the Beast receiver via the first modesmixer2 instance.

The modesmixer2 web interface displays the feed from the Beast receiver only. If you try to run the web server on the second modesmixer2 instance, it will not display the Piaware MLAT fixes, for some reason.

Quite an amazing display of normally invisible Mode-S planes and traffic on PlanePlotter!



modesmixer2 #1: modesmixer2 --inSerial /dev/ttyUSB0:3000000:hardware --web 8080  --outServer beast:30005
modesmixer2 #2: modesmixer2 --inConnect localhost:30005 --inServer 30004  --outServer beast:30015

This should not break anything, but strains the Pi v2 to the limit at 1300 Mode-S messages/sec. - near 95% CPU. Seems to work OK, though.

Regards,

Don
WD9DMP

7

So I would be interested to know if in the coming days you have any issues with modesmixer crashing. I’ve been doing this same thing testing the same setup exactly for the last week and after about 24 hours running modesmixer starts to eat RAM until it eventually crashes.

1 Like
8

I’ll keep an eye on memory utilization and post any issues. I’m running the latest test build of modesmixer2 (modesmixer2_rpi2_20150715) on both instances.

I modified my /etc/init.d startup script to launch modesmixer2, then ppup1090, then modesmixer2 again for the merged streams. Piaware has its own startup script that is probably running at the same priority at startup, so things aren’t sequenced perfectly. However, Piaware seems robust enough to keep trying connections until everything is running properly.

I’m able to start and reboot with the dual-modesmixer2 setup with no intervention in putty. CPU % is high at peak traffic periods, but acceptable at my location and message rates (1300/sec, max).

Amazing how many US aircraft have Mode-S transponders without ADS-B. My locatable aircraft doubles when PlanePlotter connects to the merged feed. The number of aircraft without locations in the PlanePlotter aircraft list is only 4 or 5 out of 180 or so. Impressive!

Best,

Don
WD9DMP

9

Are you running PP on the Pi via Wine? I don’t know how you’re getting that kind of CPU utilization. I can’t break 50% with all the feeders running on a Pi 2. Looking at htop the biggest user is dump1090-mutability.

10

I’m not running dump1090 or any other dongle decoder program. I have a Mode-S Beast that does all decoding on the internal FPGA and outputs via serial over USB to modesmixer2. Modesmixer2 then outputs on port 30005. Port 30005 feeds several destinations: the modesmixer2 web server, ppup1090 for MLAT and uploading to Planeplotter servers, a connection to a second instance of modesmixer2, a connection to faup1090 (Piaware), and a connection to fa-mlat-client.

The second modesmixer2 instance takes input from the Mode-S Beast output on port 30005 and the fa-mlat-client output on port 30004 and feeds the combined data to PlanePlotter on my PC on port 30015.

Even though there is no I/Q dongle decoder running, it does chew up CPU. The fa-mlat-client is the highest user of CPU resources.

Best,

Don
WD9DMP

11

Just for comparison, here is a shot of my received aircraft this morning, directly from my Mode-S Beast receiver only:

http://projectmf.homelinux.com/station_pics/planeplotter_07_28_2015_09_06_Beast_Only.jpg

…and from Piaware MLAT only…

http://projectmf.homelinux.com/station_pics/planeplotter_07_28_2015_09_06_MLAT_Only.jpg

…and from both combined…

http://projectmf.homelinux.com/station_pics/planeplotter_07_28_2015_09_06_MLAT_Beast.jpg

12

I need to revise my estimate of CPU resources with this rather complicated setup with multiple modesmixer2 instances, Piaware with full bi-directional MLAT and ppup1090 running. I’m afraid I was not reading the results returned by the “top” command correctly.

With all running and active feeds to PlanePlotter and the modesmixer2 web interface map display active, total CPU utilization is only about 25%, top. That is great and more in line with my expectations.

System as described here has been quite stable with no obvious memory leaks.

Regards,

Don
WD9DMP

13

15 minute load average?

14

1.08 with aggregate CPU core system idle percentage of 75%.

Don
WD9DMP

15

I would suggest looking more at your load average and ignoring CPU %. If your load average is above 1.00, then the running threads are queuing up and waiting on the CPU. This is what’s important and can cause MLAT data to be stale because the CPU is overloaded. So, a load average of 1.08 is 8% overloaded. In your case, that’s not extreme, but is not ideal.

16

I think he’s running on a Model 2 with the quad core, does the 1.0 rule still apply or does it jump to 4.0 for the four cores?

17

A loadavg of 1 is fine on a quadcore. loadavg is not a hugely useful metric anyway, free CPU and busiest process are more interesting here.

18

Yes. A load average of 4.0 on any system regardless of the number of CPUs would indicate the CPUs are 300% overloaded.
A 1.0 means they are fully loaded. A 0.50 means they are 50% loaded, etc.

To avoid playing the game of Citation Needed…
See: https://en.wikipedia.org/wiki/Load_%28computing%29

OBJ, I disagree. A load average of 1.00 is ok, but not good. Load average is a great metric. CPU % is foolish way to monitor system load.

19

Please give your argument for this. Free CPU just misleads people thinking that everything is a-ok, as their system is running bogged down. Free CPU is the bad metric, not load average.

20

http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages