I’ve been looking at the useful adsbcompare scripts created by @caius, with data collection supported by graphs1090. In particular the compare.sh script looks useful because I’m currently comparing a pair of LNAs.
There’s no dedicated topic for the compare tool (there is one for heatmap), and that optimizing gain topic is already huge, so I thought I’d create this topic.
I’m not completely sure how to interpret the scatter plots produced by compare.sh, for example the example given on the github page.
The main plot seems to be showing that blue is receiving more messages per aircraft. Does that mean it literally is doing per aircraft, or does it mean it’s receiving more messages for a given number of aircraft and so the average is higher? And it also seems to show that as well as more messages, this translates to more aircraft overall received, compared with how red did.
The smaller plot seems to be showing that greater number of aircraft detected by blue are further away in general.
There are no legends shown for the candlestick indicators and I’m not sure how to interpret the tails, the points outside the tails, the upper and lower edges of the blocks and the horizontal line shown in the blocks.
This was an example comparing two LNAs. Can you talk though how to interpret this image in that context? It seems like the LNA from the blue period “did better”, if better equates to more messages and range.
What would someone be looking for to improve on blue? More points bunched up to the upper-right on the main plot? More aircraft for a given number of messages, perhaps implying clearer reception of said messages?
I’m just not quite clear on how to interpret the image produced, and what to immediately look for when assessing relative performance, and welcome ideas.
On a sidenote, in graphs1090 can enable_scatter=yes be left enabled, or should this only be enabled intentionally for a comparison and then turned off again? Perhaps it is heavy on SD card writes or massively increases the database sizes or something like that?
Thanks for a really useful tool.
I’ll be happy to do a PR summarising the interpretation and a link to this thread, as a guide for future users.
It is the latter. The data is taken from the files generated when the scatter option is turned on. These are the highest resolution data from the rrd database used for graphs1090 which disappears after a day or so.
Each data point is a snapshot of a single time point so that for example when the receiver is seeing 150 aircraft, and the message rate at that time is 1948, then that is the point plotted. it isn’t a plot of the number of messages from each individual aircraft as that is not tracked in the data available.
For this particular example, the comparison was of two different LNAs (I think one was a satellite TV LNA and the other was something like an LNA4all, but can’t remember exactly). Blue was the better performing one.
The main plot shows the message rate at a particular time plotted against the number of visible aircraft at the same time. In the example, it shows that for any given number of aircraft being received the LNA during the blue period received more messages than the LNA during the red period. This indicates that the blue LNA is performing better - more messages are successfully being decoded.
Generally the behaviour of an ads-b receiver is that the number of messages per second received is roughly proportional to the number of aircraft within range, up to the point where messages start to overlap due to volume of traffic and become garbled. This is shown in the plots where there is an almost linear increase at lower traffic levels and a larger cluster of points at a similar message rate once there are more than about 150 aircraft visible.
This pattern is characteristic of ads-b reception, since it is a non-cooperative protocol and transponders just transmit continuously without any regard for whether anyone else is transmitting. The more aircraft transmitting at once, the higher the likelihood of overlapping messages.
The message rate will depend heavily on your location. Someone in Europe will see a much higher message rate for a given number of aircraft than someone in the US because there is a higher density of interrogating radars, and they tend to trigger transmissions like mode-s all call replies far more than radars in the US.
The aircraft vs range plot was really to see what the relationship between maximum range received and number of aircraft received was. In the example it allows you to see that there is a slight negative correlation between maximum range and number of aircraft, but that the blue LNA is seeing aircraft further overall.
The candlestick plots are just the distribution of each statistic. The horizontal line is the mean, the box is the interquartile range, so between 25-75% of values and the tails are the minimum/maximum. The dots are outliers.
So to assess performance, really it’s as simple as higher is better, with the caveat that you have to be careful to identify what your variables are. Using at least several days to allow short term variability in traffic to even out is necessary, but also you have to be aware of things like holidays, schedule changes etc skewing results if you are trying to see differences in receiver configuration.
More messages per second per aircraft means the receiver is doing better at discriminating signal from noise - in the example above, the blue LNA is seeing more aircraft over all, more messages on average from each aircraft and is receiving aircraft from further away. That means that the blue LNA probably has a significantly better signal to noise performance than the red one.
This is a comparison of the same two weeks of the year in 2019, when I’d first installed an antenna on the roof and 2024, using exactly the same hardware.
The obvious difference is the very large increase in message rate, which is entirely down to the significant reworking of the airspy decoder by prog and wiedehopf a few years ago. It is now much better at discriminating overlapping messages.
You can see that the number of messages per aircraft is higher throughout the range, and that the overall average message rate is higher despite air traffic still not having fully returned to pre-pandemic levels.
It also shows that range performance is largely unchanged, which is to be expected since the maximum range is largely limited by geography.
Yes it can be left enabled, and it’s better to do that because it means you can arbitrarily select data to compare against later on. It might help to keep notes when you make changes to your system as you likely won’t remember specifics later on. It doesn’t use a huge amount of space - i’ve had it enabled continuously for the last 5 years or so and it’s only about 120MB of data in total and it’s only one text file generated per day. To keep data that long requires a slight change to the script that produces the scatter plots though since by default it will delete them after a set period.
If you update graphs1090 at any point it also resets that option back to default so you could inadvertenly lose data. I use a script to update graphs1090 which puts the change back after the update, and also increases the resolution from 3m intervals to 1m intervals:
#!/bin/bash
sudo bash -c "$(curl -L -o - https://github.com/wiedehopf/graphs1090/raw/master/install.sh)"
sudo sed -i -e 's/rm -f $(find $data_dir -type f | sort | head -n-450)/#rm -f $(find $data_dir -type f | sort | head -n-450)/' /usr/share/graphs1090/scatter.sh
sudo sed -i -e 's/3m/1m/' /usr/share/graphs1090/scatter.sh
One thing about this script and the data it uses is that it’s not particularly robust - if there’s any corruption to the data from bad sd cards or system crashes which result in malformed files, the graphing script won’t handle it particularly well and you have to hunt through the data files by hand to delete the bad lines.
The y-axis is labelled Messages, so what this really means is Messages/s? I did wonder that, because in older posts (eg here, here, here) the axis is labelled as either Messages/s or Message Rate. This made me wonder if this version was explicitly just some absolute number instead of a rate. Logically it looks like a rate, but then I wondered why it was explicitly labelled as such on older posts but not on this more evolved version. I think I’ve been over-thinking it
I guess this is essentially capturing the numbers from the top-right of Skyview which shows Total Aircraft as a number and Messages as a number/sec. This updates every second, and the high resolution rrd capture is recording those each second.
At first glance it seems like an alternative interpretation could be that for any given message rate, the LNA during the red period saw more aircraft than the LNA during the blue period. This is because for a given value up the y-axis, the red plot is further along the x-axis.
Could this not lead to the conclusion that the LNA in the red period was performing better – it was able to see more aircraft for a given message rate, perhaps able to present cleaner decodable messages? The fallacy of interpreting it this way though isn’t immediately clear to me, which bothers me. It doesn’t work higher up the x-axis because there are no plots from the red period showing there.
Thanks for the second worked example. It’s very striking and really brings home the differences sliced across all these different metrics.
I have a build document here so that future me, and a mate who also feeds, is able to do a fresh install of PiAware from scratch and get it all set up. I’ll add this note to the graphs part so that the max res data is being saved all the time.
Thanks for the commands, I currently back up the graphs manually, and update following the instructions on its github page. I’ll work through your script and the 3m → 1m tweak so I’m clear on the best approach to save not just the graph data backups but the scatter data but allow graphs to update and be happy. Is the 3m → 1m tweak going to affect graphs1090’s graphing?
Sort of - it’s those which are stored in the rrd database by collectd. This is a resolution of once per minute, so that figure is effectively an average over that minute rather than every instantaneous reading being stored. The scatter script pulls out data on a 3m interval by default, but I have it set to pull the 1m data.
Without context you could draw that conclusion, but I think the fact that the message rate is the dependent variable means it is unlikely. Apart from receiver performance, the number of messages transmitted (whether they overlap or not) depends on the number of aircraft present. ADS-B transmits at a fixed rate, and mode-s replies will vary depending on the number of ground interrogations. In all cases, the more aircraft present, the more messages are transmitted.
For the receiver to see more aircraft with fewer messages would require there to be fewer transmissions, as a better performing receiver logically should be able to decode more messages. You do occasionally see sudden jumps in message rate without a corresponding change in the number of visible aircraft, which I can only assume results from one or more ground interrogators switching modes, or going on or off air.
No it won’t affect graphs1090. graphs1090 does not use those files, it uses the database directly. The data used to plot these graphs is extracted once per day from the database used by graphs1090 in order to preserve the higher resolution. The round robin database only stores that for a day or so, so extracting it later means you get fewer data points for any given day and also those data points are more of an average for that period.
The data for the scatter graphs is stored in individual text files stored in /var/lib/graphs1090/scatter. You should back those files up in addition to the graphs1090 databases if you are interested in keeping them longer term.
You might also find the other script (daily.sh) which uses those files useful. It produces these two graphs:
I ran compare when setting up my feeders years ago and completely forgot about it until prompted by this thread. I just used it to compare my feeders with their new antennae to the same week last year when they had home made spiders.
edit:
This is the spider with an RTL-SDR triple filtered LNA (red) vs the jetvision A3 and the same RTL-SDR triple filtered LNR (blue):
