FlightAware Discussions

Saturnus' topic on getting a better Dump1090 etc. understanding, toward the ultimate setup in Docker

As said in Distance column gone compared to dump1090-ma | and more tech talk, Docker, etc I am going to try to make a collection of Docker images without the limitations of current available Docker images for ADS-B that often have unlogical designs, outdated software, missing manuals and don’t use version tags and changelogs, posing a limitation to what Docker is meant to be. My goal is to create images (which becomes containers when running) of the main ADS-B components/players, creating a highly modular setup.

Super summary of benefits related to Docker:

  • Easy deployment
  • Easy and safe testing/experimenting.
  • Safer and quicker upgrading.
  • User specific files in a shared, yet save location.
  • Modularity. Use what software you want, leave out others.
  • Flexibility and cross-platform. Run x on Pi, run y on Linux x86.
  • Roughly no performance penality.

I have to admit for me this is a very ambitious project as, although 10+ years experience with running Linux servers, networking etc. my coding/scripting skills are rather basic. But I want to learn, so this can be great.

In order to not make fundamental design flaws in the beginning I thought to have this topic. Knowledge shared here can be applied to any setup thus, in case I fail, it will still be useful to others.

My first questions:

  1. Is Dump1090 (standalone) showing everything it finds unfiltered (military, etc.)?

  2. I’ve seen websites provide more detailed information about flights. Is there any way to pipe that data back from a server (e.g. ADSB Exchange) to your own webserver interface or so? I mean is there software to enrich the default webinterface with more options? For example the old dump1090-mutability has links to Airframes, FA, FR24, FlightStats and PlaneFinder, while Dump1090-fa only links to FA.

  3. What is the best Dump1090 network mode (Raw, beast, etc.)? Upsides/downsides?

  4. How important is BladeRF support when building Dump1090? Many people using this or super niche?

  5. In https://flightaware.com/adsb/piaware/about MLAT-data is send to Dump1090 from fa-mlat-client. Terms of use say that this data cannot be shared with other providers. And you may not feed MLAT results back to FlightAware as if it were native ADS-B data.
    5a. How to prevent sharing with non-FA parties when you are running Dump1090 (beast/raw) that is providing data to also FR24, ADSB Exchange, etc?
    5b. How to prevent sending this data back as native ADS-B data?

  6. How to combine multiple 1090MHz receivers into one output? I found http://sonicgoose.com/using-dump1090-in-windows/ (see integration paragraph). ModeSMixer2: http://xdeco.org/?page_id=48. Does Dump1090 have this mixer capabilities too, or maybe another package has? How to render a (webinterface) from the mixed data?

Thanks!

2 Likes

Yes. MLAT results may be filtered though.
If they are running a ModeS transponder you will see them in the list without a position.
Not all military planes are running ModeS or ADS-B transponders though.

The beast protocol provides relative receiver timestamps and uses less traffic because it’s binary instead of ASCII like AVR.

dump1090-fa per default does not forward MLAT results, they are marked somehow.
(mutability 1.15 also doesn’t forward those but 1.14 does because it doesn’t understand MLAT results yet)

https://github.com/wiedehopf/combine1090#combine1090

Thank you very much for all the help here and the topic before!

  1. I guess it is not possible to start dump1090 with a config file? Like putting --net --forward-mlat in /etc/dump1090.conf as
net=1
forward-mlat=1

and then execute dump1090 /etc/dump1090.conf?

Did not find anything like this and https://github.com/flightaware/dump1090/tree/master/debian seems to put the command line options in a file and then call for them as arguments.

The program itself can’t handle a configuration file.
You’ll need to give the options on the command line or use a wrapper script that uses the options from a configuration file.
I’m sure you can copy and adapt /usr/share/dump1090-fa/start-dump1090-fa

And why would you enable MLAT forwarding?
That should generally not be necessary and if that same dump1090-fa is used to feed data would be BAD.

You can get the flightaware MLAT results at port 30105 per default.
You can also push those data to extra programs:

Multilateration Settings

The “mlat-results-format” configuration setting controls how multilateration results are provided. It is a space-separated list, where each item is something that can be passed to the “–results” option of mlat-client:

  • ,connect,host:port - makes an outgoing connection to host:port
  • ,listen,port - listens for incoming connections on the given port

Supported protocols are:

  • basestation: the “port 30003” BaseStation format
  • ext_basestation: an extended version of the BaseStation format that includes mlat-specific data
  • beast: results encoded as DF18 ADS-B messages, Beast binary format

The default settings set up three items:

  • beast,connect,localhost:30104 - this feeds multilateration results back to the local dump1090 for display on the map
  • beast,listen,30105 - this provides a Beast binary format feed of multilateration results that can be used by external systems, e.g., VRS
  • ext_basestation,listen,30106 - like 30105, but in a different format

It should not be necessary to change the multilateration settings in most cases.

https://flightaware.com/adsb/piaware/advanced_configuration

What exactly are you trying to do?
Is the combine1090 script not working for you or do you need additional functionality?

Some more about the ports:
https://flightaware.com/adsb/piaware/about

Don’t use forward-mlat is the best solution to not feeding MLAT data.
So i’m confused why you want to use that option?

Ok thanks. Did not know about that MLAT.

I was just experimenting to find out if dump1090 would accept a config file.
What would be the place to ask for a config file feature?

Progress on the Docker project is going well. Got an <11MB container for standalone Dump1090 now and finding out what would be the best way to push the configuration to Dump1090 from a file (volume). That would at this moment be a script setup then, yes.

You can raise an issue on github but i highly doubt that obj will implement it.
After all it ships with a script that can use a config file.

You could always implement your own config file wrapper.
People for whom the existing config file is too complicated shouldn’t be touching the options anyway …

What is the problem with modifying the script i mentioned to your liking so you have a wrapper that takes a config file (/usr/share/dump1090-fa/start-dump1090-fa)

You are using dump1090-fa correct?
I’m not even sure what dump1090 debian package produced when building the package is.

Anyway would be helpful to write dump1090-fa, writing dump1090 confuses me :wink:

Having a config file, in my opinion, is easier to read than all the arguments.

I looked at https://github.com/flightaware/dump1090 but the issue tab is not there for the repo.

Ah yes, dump1090-fa. Will do.
Dump1090-mutability is a bit outdated concerning airplane data right?

Based on Debian a container would be ~90-120MB. I use Alpine.

Indeed it’s not.
Curious.

Anyway you can make a simple wrapper that is not error tolerant very simply:
Modify /usr/share/dump1090-fa/start-dump1090-fa like this:

source /etc/default/dump1090-fa.conf
exec /usr/bin/dump1090-fa \
  --gain $GAIN \
  --lat $LAT \
  --lon $LON \

If you are not using the systemd unit file you need to source the config file so you can access the variable values.
You could also specify default values for the case the config file doesn’t exist or a line is missing from the config file by defining default values before the source command.

And have a config file like this:

GAIN=25.4
LAT=50.05
LON=10.01

It’s not tolerant against people introducing spaces though.

GAIN= 25.4

This does not work.
I suppose you could just remove all spaces before sourcing it (doesn’t modify the file)

source <(tr -d ' ' </etc/default/dump1090-fa)

That way you can have something like this:

GAIN = 25.4

And it will still work :wink:

It’s some per-repo setting; I’ll see if I can hunt down someone with the right access to modify it.

Pretty much this. I’d consider a pull request, but there’s already mechanisms to read config from a file in the service definition etc.

@wiedehopf
Thanks, that is a nice idea.

@obj
Acknowledged!

Now I am onto the Piaware part.

  1. After building, when running Piaware No faup1090 found at , 1090ES support disabled I get this error. Looked up what other people are doing and here https://hub.docker.com/r/mikenye/piaware/dockerfile someone builds faup1090 from Dump1090-fa and then copies it in place. I have the feeling faup1090 actually should have been working from the Piaware package (makes more sense?) and not from Dump1090. Any hints?

https://github.com/flightaware/piaware_builder

Yeah I checked that out, and tried it too but it’s for Debian. Not working on Alpine.

Or if you mean something else maybe… Downloaded as zip just now and scanned the files for ‘faup’ and found https://github.com/flightaware/piaware_builder/blob/master/stretch/rules#L46
That is the way to obtain faup?

EDIT: Tried it out and that indeed seems to be the case. faup1090 is build from the dump1090-fa package. :slight_smile: :+1:

I mean yeah you’ll have to figure out how to adapt the piaware_builder to Alpine.

Seems you found the appropriate build files.

./sensible-build.sh stretch

Try that and then check the package directory.
You can also do that on debian and copy over the package-stretch to your Alpine.

Then make the compilation work.

Yeah to be honest i don’t really have a clue how to compile that stuff on a non-debian system.
Don’t think it’s really meant to be compiled on a non-debian system.

Compiling works as long as the required programs and libraries are available for the distro. Open source helps a lot. I think FR24Feed is closed source thus compiling isn’t even possible. Have to replace the busybox base of Alpine via chroot or something, will see about that next week when time is available again. Also going to set up MLAT inside the same image as the main Piaware stuff then.
So the progress is quite good. It’s like getting it to work first and then smashing around with a hammer to make the package smaller then drilling a little hole/channel for the userdata to come in and finally polishing. :crazy_face:
Thanks for all the help, back at it next week.

Quick question while I am on the go for a few days…
Got fa-mlat-client working as a program but no MLAT results in stats. It is started by Piaware with these options:

allow-mlat yes
mlat-results yes
mlat-results-format beast,connect,172.19.0.19:30104 beast,listen,30105 ext_basestation,listen,30106

Log:

2019-07-13 12:22:51Z mlat-client(251): fa-mlat-client 0.2.10 starting up
2019-07-13 12:22:51Z mlat-client(251): Using UDP transport to 70.42.6.156 port 8559
2019-07-13 12:22:51Z mlat-client(251): Listening for Beast-format results connection on port 30105
2019-07-13 12:22:51Z mlat-client(251): Listening for Extended Basestation-format results connection on port 30106
2019-07-13 12:22:51Z mlat-client(251): Input connected to 192.168.1.19:30005
2019-07-13 12:22:51Z mlat-client(251): Detected BEAST format input
2019-07-13 12:22:51Z mlat-client(251): Input format changed to BEAST, 12MHz clock
2019-07-13 12:22:51Z mlat-client(251): Beast-format results connection with 172.19.0.19:30104: connection established
2019-07-13 12:37:51Z mlat-client(251): Receiver status: connected
2019-07-13 12:37:51Z mlat-client(251): Server status:   not synchronized with any nearby receivers
2019-07-13 12:37:51Z mlat-client(251): Receiver:   68.0 msg/s received       14.8 msg/s processed (22%)
2019-07-13 12:37:51Z mlat-client(251): Server:      0.0 kB/s from server    0.0kB/s TCP to server     0.2kB/s UDP to server
2019-07-13 12:37:51Z mlat-client(251): Aircraft: 8 of 8 Mode S, 5 of 6 ADS-B used

Does not change after a few hours.

Is this normal? Maybe no results because my non-development setup with much better antenna is working from the same lat/lon location? Because I saw no MLAT hits anymore since the day of starting the pro setup.

Probalby your clock is not stable.

dump1090-fa likes to run on real hardware so no USB data is lost.

Run rtl_test after stopping dump1090:

rtl_test -s 2400000

Until a second after startup some lost samples are acceptable.
After that, any lost samples will be a problem for MLAT.

How much time offset is acceptable? At this moment it’s ~3 seconds ahead.

The Ali stick is fit in a Windows computer, directly passed through in Virtualbox to Ubuntu Server, then fed to a Docker container. No emulation/simulation.

Maybe the stick itself broke already. Will do rtl_test later today.

The system time being somewhat accurate doesn’t hurt.
One normally runs some sort of time synchronization.
systemd-timesyncd or ntp is both acceptable.

But that’s not the clock i’m talking about.
I was talking about the MLAT pseudo clock which is basically just a sample count.
Very simplified:
Every amplitude reading from the SDR is called a sample.
You should get 2.4 Million samples per second. (2.4 Megasamples/s)

You count those samples.
If some samples are lost due to the USB bus not being capable of transmitting all of them, then you lose your clock.
It’s unstable.

Virtualization can be enough of a problem for USB latency and dump1090-fa working sufficiently for MLAT.

Understood, thanks!

I was unable to test on the holiday, but continued yesterday at home. Seems you are right about the forwarding of USB to a VM. It surprises me because I expected an almost ‘native forward’ since the device becomes unavailable in the host after forwarding, I think it even disappears.

Both the Docker container and the base OS (=VM) reported data loss. I plugged the stick into a RPi 3B+ and there was no loss any more in the Docker container. Used that opportunity (x86_64 --> arm) to test out my Docker build code and that went pretty well.

mlat-client(16): Receiver status: connected
mlat-client(16): Server status:   synchronized with 107 nearby receivers
mlat-client(16): Receiver:   12.9 msg/s received        4.2 msg/s processed (33%)
mlat-client(16): Server:      0.0 kB/s from server    0.0kB/s TCP to server     0.1kB/s UDP to server
mlat-client(16): Results:  1.5 positions/minute
mlat-client(16): Aircraft: 2 of 2 Mode S, 0 of 0 ADS-B used