That’s works good now with your modifications : thanks !!!
That would be me 
They should really bring back the shuttle and do a few more scans a bit further north 
Or I’ll run my slightly older version of your scripts with enough code removed to make it draw some graphs.
Would be nice if you can fix a version that works without HWT, but I manage with the version I have running now. I might do some modification of one of your newer versions to see if I can get it running.
Thanks for the effort you’re putting into this 
It’s pretty much done - I’m just in the middle of doing something else, but when I’m finished with that I’ll test it some more and upload. It wasn’t actually that hard to do.
@bive Give the new version a try. Enter your lat/lon/height in the script and leave the hwt field empty. That will make it write blank data files for the hwt data and should prevent gnuplot from complaining about missing files. Let me know if you have any problems with it.
Config file found
No HeyWhatsThat ID found
ADS-B and MLAT aircraft will be plotted
Using tmpfs : /tmp/tmp.1WhdCooN8a
Using local archive:
Unpacking compressed data:
.........................................................................
Retrieving recent history:
........................................................................
Number of data points collected: 287772
Calculating Range, Azimuth and Elevation data:
Filtering altitudes
Generating all altitudes heatmap...
Generating high altitude heatmap...
Generating low altitude heatmap...
Generating close range heatmap
Generating elevation heatmap...
Generating Range/Altitude plot...
Generating Close Range altitude plot
Generating low heatmap with map overlay
"/dev/stdin" line 143: warning: Skipping data file with no valid points
"/dev/stdin" line 156: warning: Skipping data file with no valid points
Graphs available at :
I get no fancy overlays, I get markers for two airports (one is closed, the other has one or two planes per month), closest major airport for me is ARN, some 80Nm from my station).
Besides that, it works pretty well
The overlays only show coastlines as that’s all that’s in the data set unfortunately. I can’t add things like borders or state lines easily. That’s why you are getting the 2 warnings - the outline file is empty. The airport data might not be completely accurate, especially for very small fields since it’s only what people have added to the database themselves. If you want more labels, especially for the smaller fields you can change the label lines eg 516 in the current version:
dir.'/large' u ($9/1852):($10/1852):($6) with labels offset char 2,-1 tc rgb "green", \
Change the $6 to $1 and it will use the ident (which is usually the ICAO code) rather than IATA - the ICAO one is more complete in that most fields have something there, though I think some small ones aren’t official designations.
Edit - actually there might be country data available. I’ll look into it.
1 Like
@bive It just occurred to me you are in Sweden aren’t you? If so you should be able to see coastline data on the map.
Could you try deleting the world_10m.txt from the directory the script is in and run it again? I tried it using your approximate coordinates in your profile and I get output including the map (ignore the airport dots as they aren’t correct for your location):
I’ve found some data that includes country borders as well now and that new data set is uploaded to github. In order to use it, just remove any existing world_10m.txt file to force the script to grab the new data. It looks like this:
1 Like
Worked without errors after it downloaded the new world_10m.txt. Also changed line 517 as described above. The smaller airports are still showing, probably because they still have ICAO codes assigned (one airport is an old military airport that house an airforce museeum and they get the odd historical flight every now and then, the other one is closed but probably not reported as such since there are disputes about the closing).
Anyway, it looks way prettier now and I’m a happy camper Thanks for the work!
2 Likes
I have deleted the world_10m.txt file, and download again the script, and i have this error :
Config file found
HWT OK
Processing heywhatsthat.com data:
Setting receiver position from heywhatsthat data. If these values do not match what you are expecting, please check the heywhatsthat ID is correct and that it was generated with the correct location
Latitude: 48.54764
Longitude: 2.42108
Height: 85.344
ADS-B and MLAT aircraft will be plotted
Using tmpfs : /tmp/tmp.N5wA1QAUs1
Using local archive:
Unpacking compressed data:
.....................................
Retrieving recent history:
..............................................................................................................................................................................................................................................
Number of data points collected: 1522396
Calculating Range, Azimuth and Elevation data:
Filtering altitudes
--2019-10-07 10:19:26-- https://raw.githubusercontent.com/caiusseverus/adsbcompare/master/world_10m.txt
Résolution de raw.githubusercontent.com (raw.githubusercontent.com)… 151.101.120.133
Connexion à raw.githubusercontent.com (raw.githubusercontent.com)|151.101.120.133|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 17502745 (17M) [text/plain]
Sauvegarde en : « world_10m.txt »
world_10m.txt 100%[========================================================================================================================================>] 16,69M 883KB/s ds 19s
2019-10-07 10:19:48 (880 KB/s) — « world_10m.txt » sauvegardé [17502745/17502745]
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8178k 100 8178k 0 0 775k 0 0:00:10 0:00:10 --:--:-- 876k
Generating all altitudes heatmap...
Generating high altitude heatmap...
Generating low altitude heatmap...
Generating close range heatmap
Generating elevation heatmap...
Generating Range/Altitude plot...
Generating Close Range altitude plot
Generating low heatmap with map overlay
"/dev/stdin" line 138: undefined variable: set
“/dev/stdin” line 138: undefined variable: set
Thanks.
1 Like
I thought I’d fixed that one once already. Ah well, it’s fixed now. I don’t know why that doesn’t seem to throw up an error for me when I’m testing.
If you want to remove the very small airports completely, then remove the plot command lines in the gnuplot part of the script that refer to the small file. eg removing line 520 will take them off the map plot. They aren’t shown on the wider plot, so it could be that some of those are classed as “medium” in the database.
1 Like
It’s working perfectly now ! Thanks again for your work !
I’ll do that
They are both equipped with long runways, so that may be the case.
one small edit/typo fix
Line 46
FIR_MAXITER = 100 should be FIT_MAXITER = 100
my script got stuck in a loop trying to get convergence (up 10000 iterations). Once I changed fix the typo it stopped looping at 100
1 Like
Thanks. It’s now fixed.
I’m very impressed with the work being done on this idea - my thanks to all involved. It’s really cool to be able to see the theoretical horizon for my ADSB receivers, and to see how close to reality the theoretical actually is.
Apologies if this is a dumb question (I’m having a migraine day today and not exactly having a lot of joined-up thinking for complex things…)…
How would one go about having a cumulative heatmap in addition to e.g. a previous 24 hours?
I’m hoping to cronjob each day’s visible flights, with the data collected via timelapse1090. That I am fairly sure I can do with ease, and I can do an archive of each day of that.
I’ve currently for timelapse1090 set for a week of data collection just to see how far I can go with 32Gb cards on a pi3 and pi4, and how much pain it causes for the flight volumes seen from my locations.
How would I ensure that I would have e.g. a month’s data available to generate a monthly heatmap?
Would I just concatenate each day’s airplanes.json file together somewhere else and then run the polar script on that data once per month? Or am I missing something painfully obvious with this idea? I’m aware that the time taken to generate this would be large, but I have that capability on my other machines, that monthly one would not be done on the Pi. At work I’ve got an 8-core Xeon cloud vm that would be used.
I’ll take a better in-depth look at this later in the week when the brain is behaving better.
The period of time covered is set by the amount of data you collect, either by setting the parameters when the script runs (duration and interval) or by changing the length of time that timelapse1090 keeps data for. The advantage of the latter is that you can run the script retroactively rather than having to anticipate when you might want to create plots.
timelapse1090 keeps its data in /run which is mounted to ram. This will be more of a restriction than your sd card size - although the data is gzipped jsons, it does become quite large - I have it set to maintain 24 hours at 3 second intervals which is around 310MB. From memory the default interval is 10 seconds, so about 100MB per 24 hours, which means you will need 700MB of ram for a week of data - on a pi 3 that will probably cause problems for other processes. A pi 4 with more ram would be OK though.
The easiest way is to change the timelapse1090 time to keep as much data as you require, since the script will then work on it with no modifications (other than file location). You would need to alter the configuration of timelapse1090 to store its files on disk rather than ram. It would break the timelapse1090 web interface, but then it won’t work with that much data anyway (it already crashes the browser with the settings I’m using).
The other option is to collect the data in another way. Since the script just processes collected aircraft.json, you could run a script remotely to just save those somewhere. It should be fairly simple to modify the script to read those files - the code is basically in there to do that already.
As you note, it would take quite a long time to run. A month of data would be many millions of points. Gnuplot should cope with that OK providing it has sufficient ram.
That’s with your amount of traffic, for the typical installation it will probably be 50 MB or less.
If you create the graphs every 24h from the timelapse data, you could move the created data file somewhere else and label it fittingly.
A wrapper script for running it nightly which also saves the data might be interesting.
Then you’d need another input method in the main script to take the preprocessed data according to date or sth.
1 Like