Signal Strength Heatmap

FlightAware ADS-B Flight Tracking
472

You might find the problem resolves itself with more data. I’ve not seen that error occur before.

If you have timelapse1090 then that will accumulate data for the period set. The script will use all the data that is available. The interval and duration for data collection is set in /etc/default/timelapse

I have mine set for 24 hours collecting every 3 seconds. That produces quite dense plots, but takes correspondingly longer to process. Not so bad on a pi 4 or PC but can take a while on pi 3. If you set it that high though, the actual timelapse web page will probably crash when you load it under normal traffic levels. If you only use it for making these plots then it’s not really a problem.

473

So i’ve ran ./polar.sh -1 and the COLUMN output was;

  • COLUMN:
    Mean: -9.0034
    Std Dev: 5.2432

When i run ./polar.sh 60 1 i got the following error again.

  • FILE:
    Records: 128633
    Out of range: 0
    Invalid: 0
    Column headers: 0
    Blank: 0
    Data Blocks: 1

  • COLUMN:
    Mean: 0.0000
    Std Dev: 0.0000

Generating all altitudes heatmap…
“/dev/stdin” line 34: Can’t plot with an empty cb range!

The ./polar.sh 60 1 process only produces two files. One PNG polarheatmap-2020-05-21.png and the data file whereas the ./polar.sh -1 produces the full suite of png files and updates the /plots page.

474

Let me have a play and see if I can reproduce the problem.

Edit - OK I’ve reproduced it and the problem is in the part that grabs the live data from dump1090. I rarely use that way of plotting so I’ve not seen it before. Not sure whether the aircraft.json format has changed slightly or if something else changed.

475

It should be fixed now - grab the updated polar.sh and give it a try.

The problem was some variables were being defined inside a loop that isn’t used when getting data locally so they never got set. I’ve moved them outside of it so it should now work as intended.

476

Hi caisus,

So i updated polar.sh and ran a very short ./polar.sh 5 1. the output was as following;

Number of data points collected: 246
Calculating Range, Azimuth and Elevation data:
Filtering altitudes
Generating all altitudes heatmap…
Generating high altitude heatmap…
Generating low altitude heatmap…
“/dev/stdin” line 49: warning: Skipping data file with no valid points
“/dev/stdin” line 49: warning: resetting rrange
Generating close range heatmap
Generating elevation heatmap…
Generating Range/Altitude plot…
Generating Close Range altitude plot
Generating low heatmap with map overlay
“/dev/stdin” line 142: warning: Skipping data file with no valid points
Graphs available at :
http://raspberrypi/dump1090-fa/plots/
Graphs rendered in 311 seconds

For the avoidance of doubt, there was very little traffic visible, and no traffic below 25000ft. I believe this will be why the lines in bold were reporting no valid points?

I am please to say that the full /dump1090/plots is displaying now as expected, allbeit with very little traffic. I will now run a longer sample nohup ./polar.sh 180 3 rate and report back.

Thank you very much for your help and a brilliant script! :slight_smile:

477

Yes, that will probably be because there’s no data there. The main problem seems to be fixed though.
Looking at this problem I noticed another bug which I’ve also fixed, so it was useful.

478

Hmm, I am having an issue, I have moved the location of my pi by about 100 miles. Updated the HWT link and also the location in polar.conf. The displayed data and the HWT range ring is showing correctly but the map overlay is centering the position over my previous location, so the plot data is not aligned with where the airports are etc. … I also deleted the polar.conf file and the script created a new one, but still the map overlay is not correct. Am I missing a location setting somewhere?

479

The map data is stored in a file called world_10m.txt. This is contains map data for a radius around your position to keep the size down, so if you move you should delete that file and let it regenerate for the new location. You should also delete the existing upintheair.json which will force a download a replacement corresponding to your new HWT id.

480

Thanks for the quick reply. I have renamed both those files, and re-run the script. It seems to suggest it has picked up the correct HWT id, the newly created polar.conf has that same ID and lat/lon are empty in the conf file? Is that correct? the script does show the correct lat/lon from HWT id.

The map is still centered around my old location though.

481

If you are using HWT, the lat/lon in the config are ignored and that position contained in the upintheair.json is used instead, so it being blank isn’t a problem.

If you want to start from scratch, delete polar.conf, world_10m.txt, upintheair.json and airports.csv. Set your HWT id in the script itself. Those are all files that are created or modified on the first run, and not subsequently refreshed. If the script was run and one of those is not as it expects then it can cause problems - I should probably try and add some better error/status checking in there at some point because it’s not particularly robust at the moment.

You should see it download the world map on first run with an output something like this:

Length: 17502745 (17M) [text/plain]
Saving to: ‘world_10m.txt’

world_10m.txt                                             100%[===================================================================================================================================>]  16.69M  22.8MB/s    in 0.7s    

2020-09-29 15:41:57 (22.8 MB/s) - ‘world_10m.txt’ saved [17502745/17502745]

It does some calculations to format the data correctly and discards most of the data for elsewhere in the world as otherwise it increases processing and plotting time a lot for something that doesn’t appear on the plot. Once it’s done that, it won’t go and get it again if it detects the data is still present.

482

AH!! My mistake sorry… the map overlay was indeed correct, it was the Airports that were in the wrong place … I Should of noticed that. I cleared all the files and it is now working as expected. Amazing! Thanks.

1 Like
483

I don’t know if there is a new discussion on this topic as this one is pretty old.

Anyway, since a recent upgrade of skyaware, I’ve been getting messages about /dump1090-fa being deprecated and needing to use /sykaware instead. In case future updates remove the folder /usr/share/dump1090-fa altogether, I’ve modified all the references to /usr/share/dump1090-fa in my polar.sh to /usr/share/skyaware.

484

Yes, I need to update that. I’ll take a look at it.

485

Thanks. Its a great bit of software that I use on a regular basis.

486

This isn’t really directly related to the previous plotting script, but this thread is a bit of a dumping ground for my stuff so it’s going here.

I quite like the hourly received reports plot on the FA stats page but it only shows the previous week:

image

I wanted something that covered a bit longer and in a bit more detail, so I decided to try and make something that did that. This uses data that is already collected and stored by graphs1090 so doesn’t require any other services running to work.

The graphs1090 rrds hold ~1 month of data at 15 minute resolution so that’s what is being used here. Data plotted is message rate (local and remote combined), total aircraft visible, mlat aircraft visible and range:

image
image1738×843 62.6 KB

image
image1731×843 66.7 KB

image
image1724×843 65.1 KB

image
image1731×843 60.1 KB

The range plot is intended to show the difference from the average maximum range. White is average, more intense red is longer range than average and blue is shorter. I don’t like how it looks that much so I’ll probably change it when I find a palette I prefer. The dark line on 22nd May is some missing data which wasn’t written to disk when I managed to crash the pi.

If anyone wants to try it, the script is here: adsbcompare/monthplot.py at master · caiusseverus/adsbcompare · GitHub

or you can do:

wget https://raw.githubusercontent.com/caiusseverus/adsbcompare/master/monthplot.py

Create a directory in /usr/share/skyaware/html called plots and either make it writeable as that’s where the files are saved, or run the script using sudo. It will allow you to see the output via web browser. The address is:

<ip of pi>/skyaware/plots/heatmaps.html

If you want it to output elsewhere there is a variable at the top of the script to set the location.

The script requires pandas and seaborn, so if you don’t have them installed do:

sudo apt install python3-seaborn
or
pip3 install seaborn

I’m not sure if the raspberry pi has binaries available via pip, so it might be preferable to use the apt method, or it will take long time to compile.

Then to run it:
python3 monthplot.py

3 Likes
HOWTO: Airspy mini and Airspy R2: Piaware / dump1090-fa configuration
487

It works very well (I used the apt installation).
That a very welcomed addition to the other great tools you provide!

488

Thanks for this script. It confirmed that my recent hardware changes for were for the better.

Would it be possible to get UAT heatmaps as well?

489

Yes it’s likely possible, but since UAT isn’t used here in the UK I don’t know what files the data is stored in, and what metrics are available. If someone gives me some sample data I can probably add it.

490

Tell what sample data you need and I’ll see if I can provide it.

Thanks

491

If you can provide examples of the rrd files used for UAT that would be easiest - they are usually in /run/collectd or /var/lib/collectd/.