Aggregation of dump1090-mutability outputs

[dstreufe]

Does anyone know the limit of how may receivers could be consolidated on a single dump1090 target? How many planes can dump1090 handle at once?

Could it be compiled on something more robust than a Raspberry Pi and handle traffic from hundreds (or thousands) of receivers?

[obj]

There’s no hardcoded limit, but I haven’t tested with large numbers of receivers so there’s surely bottlenecks to discover. (notably, the tracking code will probably start going slow with large numbers of aircraft)

[dstreufe]

I want to take output from Dump1090 and feed it into my own PostgreSQL database.

What’s the best way to get readable data out? Port 30003, “Kinetic SBS” format? That seems to be the most readable, although it does not include MLAT data.

Does the “RAW” port 30002 send the MLAT data as well? Obviously, it would be more work to parse, but maybe that’s the “cleaner” way?

–Dan

[obj]

@dstreufe

I want to take output from Dump1090 and feed it into my own PostgreSQL database.

What’s the best way to get readable data out? Port 30003, “Kinetic SBS” format? That seems to be the most readable, although it does not include MLAT data.

Does the “RAW” port 30002 send the MLAT data as well? Obviously, it would be more work to parse, but maybe that’s the “cleaner” way?

By default mlat data is not forwarded on any port.

If you turn on --forward-mlat on dump1090 then mlat positions will be forwarded out 30002, 30003, 30005. If you do this please be careful that the mlat data doesn’t go to places that aren’t expecting it. An alternative is to tell the mlat client to also send data somewhere else via piaware-config -mlatResultsFormat, and then connect to that second feed as well as dump1090.

If you don’t want to do your own mode S message decoder / position decoder etc (it is not entirely trivial - you need a pair of messages to get an unambiguous position, so you need to track state per aircraft) then the port 30003 (“basestation”) format is your best bet.

[antthomas]

The socat method of aggregating is working very nicely for me (far better than netcat). Just passed 3 sets of data to a central server in prep for me adding a feeder at another location, mlat also. Not feeding to FA, just using it as a display of all feeders.

Message rate is the correct total and there’s no duplicate planes.

[antthomas]

Just thought I’d add a little bash script that ensures that the socat command restarts if there’s an issue or drop in connection for some reason

#!/bin/bash
while true
do
socat -u TCP:feederIP:30005 TCP:aggregatorIP:30004
sleep 5
done

The sleep isn’t essential but in case the feeder is down for whatever reason it stops the script going into a resource hogging loop, only retrying every 5 seconds.

[abcd567]

I just wanted to experiment with aggrigating the feed.

First I gave following command to the aggregator RPi, checked gmap of aggregator RPi, it seemed to work.

 nc -d <ip address of receiver pi> 30005 | nc 127.0.0.1 30004 & disown

I then rebooted the aggregator RPi (to stop the aggregating process started by above command) and then gave following command, checked gmap of aggregator RPi, it seemed to work.

 sudo apt-get install socat  ##only one time command, as Raspbian Wheezy does not have socat installed by default
 sudo socat -u TCP:<ip address of receiver pi>:30005 TCP:<ip address of aggrigator pi>:30004 & disown

I have added “&” and “disown” at the end of commands to keep aggregating alive even if that console is used for some other command, or console window is closed.

Now I have a problem: When any of the two RPi is rebooted, the aggregation breaks. Is there a way to restart the process automatically at reboot?
.

[obj]

@abcd567

Now I have a problem: When any of the two RPi is rebooted, the aggregation breaks. Is there a way to restart the process automatically at reboot?
.

To restart it on reboot, you can write an init.d script or add something to /etc/rc.local

You probably want a loop around it as antthomas said so that it restarts the forwarding if either end disconnects for some reason.

[abcd567]

Thanks Oliver. Will try to write a script to restart/reconnect this weekend.

[TomMuc]

@DinoM

ok, thanks for the info. I haven’t successfully run VRS yet… I’m on iMac and have installed mono and I also can run a windows virtual machine via Parallels… not sure I’d want to run it 24/7 but once I do try and see how it works perhaps I can find an older PC kicking around.

hi dino, virtualradarserver is phantastic and runs stable on my macmini mediacenter server 24/7. with mono and x-server the server control panel is disgusting to use but beside that anything is stable and energy-efficient. possibilities are way better than with dump1090 webinterface.

regards
tom

[antthomas]

@abcd567

I just wanted to experiment with aggrigating the feed.

First I gave following command to the aggregator RPi, checked gmap of aggregator RPi, it seemed to work.

 nc -d <ip address of receiver pi> 30005 | nc 127.0.0.1 30004 & disown

I then rebooted the aggregator RPi (to stop the aggregating process started by above command) and then gave following command, checked gmap of aggregator RPi, it seemed to work.

 sudo apt-get install socat  ##only one time command, as Raspbian Wheezy does not have socat installed by default
 sudo socat -u TCP:<ip address of receiver pi>:30005 TCP:<ip address of aggrigator pi>:30004 & disown

I have added “&” and “disown” at the end of commands to keep aggregating alive even if that console is used for some other command, or console window is closed.

Now I have a problem: When any of the two RPi is rebooted, the aggregation breaks. Is there a way to restart the process automatically at reboot?
.

As obj mentioned, the script I posted above will do the job. I could be wrong but using nc you have to do something on both the feeder and aggregator? But with socat you don’t, that’s how I found things, which means socat is an easier method and you only need a script on the aggregator.

You can create a file like:

sudo nano /usr/local/bin/dump1090-aggregator

use the script above accordingly

sudo chmod +x /usr/local/bin/dump1090-aggregator

If you don’t want to play with init.d or rc.local you can also add a line to your root crontab

@reboot /usr/local/bin/dump1090-aggregator

[abcd567]

Thanks antthomas. Your method seems good, will give it a try.

[abcd567]

@DinoM

ok, thanks for the info. I haven’t successfully run VRS yet… I’m on iMac and have installed mono and I also can run a windows virtual machine via Parallels… not sure I’d want to run it 24/7 but once I do try and see how it works perhaps I can find an older PC kicking around.

@TomMuc

hi dino, virtualradarserver is phantastic and runs stable on my macmini mediacenter server 24/7. with mono and x-server the server control panel is disgusting to use but beside that anything is stable and energy-efficient. possibilities are way better than with dump1090 webinterface.
regards
tom

Hi dino & tom
I have been aggregating feeds from my 2 sets of (RPi+DVB-T+Antenna) on my Windows Desktop using VRS and used this aggregate to feed RadarBox24. The feed is intermittent as I do not run my Desktop 24/7. Still I use Windows Desktop + VRS to feed RadarBox24 because:
(1) They do not have a Linux based data feeder client
(2) They have given me only one registration (EXTSHA000184)

The first screenshot below shows one of the planes on map for which data is fed by my station EXTSHA000184.
Other images show the various settings of VRS to aggregate & feed.

Please click on images to see larger size

https://farm1.staticflickr.com/742/21993206516_666a2e884d_m.jpg . https://farm1.staticflickr.com/651/21835078169_8df45a3026_m.jpg . https://farm1.staticflickr.com/612/21993204026_f1562d77ce_m.jpg .

https://farm6.staticflickr.com/5677/21831554538_845c2316fc_m.jpg . https://farm6.staticflickr.com/5762/22019414165_c1db0f6b42_m.jpg . https://farm1.staticflickr.com/668/22019648145_de83e7c919_m.jpg
.

[TomMuc]

cool

but i would still want to encourage you using a mac mini. these units a really great because they are enduring, quiet and have really low energy costs. and if windows is your favorite you can run this os instead of osx. mine is a 4 core i7 but just because it’s my mediacenter where mp4 files for use iphone/ipad are generated and four different channels from my sat dish can be recorded (or streamed to computers or i-devices) simultaneous. if using as a server for ads-b data and virtualradarserver a used (150$) one from ebay is easily adequate.

[ppavey]

So would these methods work to aggregate the feed from two co-located RPI’s to supply one feed to flightaware? Could the aggregation occur on one of the two RPI’s?

Thanks - phil

[obj]

@ppavey

So would these methods work to aggregate the feed from two co-located RPI’s to supply one feed to flightaware? Could the aggregation occur on one of the two RPI’s?

Not if you want mlat; that does not work if you feed from an aggregated source. The recommended setup if you have two dongles is to have two piawares.

[TomMuc]

… but is there no way to differentiate between several input streams on the same machine in future releases by tagging them before mlat?

[obj]

@TomMuc

… but is there no way to differentiate between several input streams on the same machine in future releases by tagging them before mlat?

No, not without inventing a new protocol for transporting the data.

[TomMuc]

ok - that’s not worth the thing

[ppavey]

Thanks for the advise, will run as separate systems