As much for a bit of a challenge as for a bit of fun, I thought I’d have a go at updating the database in dump1090-fa with the data from my installation of VRS using the tools provided in the master zip file I’d downloaded from GitHub.
I now realise that I reinvented the wheel rather as I later discovered that this has been done before by various people who have documented their progress on this site but here goes:
I soon found out that the tools are a mix of python 2, python 3 and JavaScript. While they will run happily on the RPI, I didn’t know how much of a performance hit there would be and I didn’t want to take either of them offline during the day time.
The other thing is that because I’ve been running VRS continuously since 2015 the basestation database is 25GB in size and I wasn’t sure whether the Pi would be up to dealing with that volume of data.
So I decided to do as much of the processing as possible on one of my PCs by installing whatever software was required. Just to make things more complicated, I don’t currently have any PCs running LINUX so I had to use Windows instead.
I tried using the vrs-to-csv python script to start with but had issues with it because the column names in the my VRS database are different to the ones in the original (no longer updated) basic aircraft lookup database the script was written for. I tried modifying it to suit but couldn’t get it to output in the correct format so I created a modified version of the SQLite3 script I use to do the conversion from VRS to adsbSCOPE instead.
This gave me a CSV file containing a little under half a million records.
Next job was to run all the CSV files through the filter-regs JavaScript to reduce the number of duplicate and unnecessary records. This has dependencies which meant that the easiest thing was to run it on the Pi.
Next came the csv-to-json python script. I was actually surprised at how quickly this ran on my Pi 3B+. All that was left to do was to make a backup of the original db folder and drop the newly created one in its place.
There was initially a bit of slowness loading the maps but after a couple of minutes, the Pi seemed to be back to normal and the difference was immediately obvious.
2 Likes
So, now I’ve got an updated database but there are anomalies in the plane outlines shown due to the way the outlines are chosen.
Reading through posts on this site I discovered that the aircraft type is calculated using data specified in /usr/share/dump1090-fa/html/markers.js and that there are some newer aircraft that aren’t listed and some older aircraft where the code has been ‘recycled’.
I made a start on updating the list but wondered if anyone has already gone through this procedure.
I’d be interested to hear.
I’m watching this with interest - Not that I can add anything but if you’re willing to share, I’d be very keen in a copy of your database once you’re happy with it please.
Sure, but my database is probably rather selective because I’ve only been getting planes at a maximum of about 180 miles to the north and a lot less in other directions due to the terrain at my location. Having my antennae are in the loft doesn’t help of course but I’m hoping the situation will improve when the rest of my bits and pieces arrive.
Although we’re near a couple of international routes and only thirty miles from Birmingham, we’re in a bit of a ‘dead’ triangle and we don’t seem to get that many planes close to here.
Even so, being as the data that came with dump1090-fa is almost 9 months old, I think almost half a million recent records makes a useful update.
I’ll let you know when I’ve done a bit more messing about with the outline mappings, assuming that you’re still interested.
I see an awful lot here and am happy to fire up VRS with my two feeders connected to it if it’ll help you gather more data.
I’m sure that would help.
Thank you for your help. I am not tech savvy at all so I won’t be much help except words of encouragement. To be selfish, any improvements/upgrades could you please include detailed step-by-step how-to instructions (or video) on how to update.
Thank you and good work.
That’s not selfish - more like shrewd IMHO.
No point starting from scratch when someone else is already doing what you want to achieve. Isn’t that what place like this are for? Everyone helping everyone else out.
1 Like
This project has stalled at the moment because dump1090 stops working a day or so after copying over my modified json and js files. It starts working when I put the old ones back.
I need to understand what is going wrong obviously but ATM I fear I may have succumbed to a touch of hubris.
For anyone who was hoping this might result in some updates for PiAware, the good news is that in concert with wiedehopf my VRS database has been converted to json files and seems to be working fine.
I found some errors in the export and typos in my modified markers.js but the problems I had seem to have been caused by something else.
details here: https://discussions.flightaware.com/t/updating-local-db/53347
1 Like
