Check /etc/lighttpd/conf-enabled files.
That will explain to you how the stuff is mapped.
If you look at your local interface … /skyaware you press F12 and check what’s being transferred.
Anyhow you figured it out.
This is actually produced by readsb which is used as the adsbexchange feed software, the slight differences you can read here: https://github.com/wiedehopf/readsb/blob/dev/README-json.md
Sorry both, as e newbie I was not allowed more than 5 replies and had to wait 6 hours.
But it is a huge succes, all data from Raspberry Pi bursting into my PC with Python.
Update rate is unbelievable, more than I need even.
Making nice plots using QGIS.
And my Propensity calculations are running nicely!
Massive thanks to you!
If you want to add something after your latest post where nobody else replied, simply edit your last post.
I think the community would be interested in your results.
I wrote a PHP program that runs on a Unix box. It make a TCP connection to the PI on port 30003. Every second there is a .csv format “dump” of the various messages received. This is the “Base Station” format.
The program then has an internal list (array) of all aircraft seen, and as the various bits and piece of information show up, they are updated. If a new aircraft shows up, it is added to the list and if the timestamp for any aircraft in the list exceeds 1 minute, that entry is removed.
Periodically (typically set in the program as a constant of 1 to 4 seconds) the list of aircraft is scanned and all entires that have sufficient information are then “CRUD”-ed to a MySql database (where CRUD = Create, Retrieve, Update and Delete).
I’m now working on a variant of this program that maintains the equivalent of the database table in memory, mostly because running a database when you have SD cards or other FLASH based storage can eat up the storage device with all those write cycles. There is nothing particularly difficult w/r/t/ how the programs work, just a lot of detail due to all the different data elements that are available from the ADS-B system.
The first version that runs on UNIX then has a Daemon that can send the current entires in the database via UDP to other systems, which have a daemeon that can receive and maintain their own database. As such, databases on multiple systems are in sync with each other.
The second version has the PI sending a UDP message with all the currently valid entires to other systems. So there is no need for a database running on the PI. For this second version, I will still need to figure out how to launch Daemons on boot, the equivalent of the Unix /etc/rc.d directory mechanism.
So w/r/t your post, consider a TCP connection to the pi and the somewhat easy to parse format of the “Base Station” output. I’ve not working in Python, but it strikes me that a .csv format with constant, known data elements would be easy to parse. Subroutines to maintain an in-memory list of aircraft, extract entries from that list which have whatever you’ve deemed are the “minimal required data elements”, and then do something with that list help break down the problem.
I suspect that there are so many different use cases with the data that any code out there will need a bit of tweaking. If you can use QGIS, you can figure out how to tweak code. Like any programming language, QGIS has its own learning curve.
For what I’m doing, giving first responders flying drones a “heads up” as to other low flying aircraft and providing ETA to landing sites, I also created a local database so I can enter information about known local aircraft. This system works with just the raw ADS-B data and can work with or without internet connectivity.
Encloses are some pix of what you see, an ETA example, and then recently I added AGL information on the aircraft for when you have Internet connectivity. In drone land, everything is AGL based. This lets me overlay aircraft location on top of responder locations with various map tile sets. When flying a drone, warnings will pop up if a low flying aircraft will be in your area in 5 minutes or less. Integrating low flying aircraft information into emergency responders information systems is a huge win.
How elegant! Exactly how I figured I would do it. However, scanning the file aircraft.json for complete messages was much easier, so I did that. I get a ‘dict’ containing a ‘list’ with ‘dicts’. With Python and Pandas together, I convert it to a DataFrame and BAM! after that anything is possible.
Amazing! Very nice and indeed very useful.
What do you use to plot on a map and update it? I was using QGS but am now looking at IpyLeaflet for more control over what happens (hover with mouse over elements etc.)
For now, my ‘proof of concept’ worked, next step is to buy a better antenna and place it on the chimney of my house. For the moment, there is nothing more to see than just a few aircraft plots on a Geo package and an occasional yellow line between aircraft pairs with their propensity value. (which are not present as the a/c are far apart anyway)
The core of the map is Google Maps. It’s a pay-for service, so this cost money. Every time you load up the web page, there is a charge. I can use different map tile sets, but there is still the charge for their API. The win is all the various functions you get:
routes, Traffic, various business icons. You also get APIs for forward and reverse geocoding (address to lat/lng, lat/lgn to address), lat/long to elevation (critical for working in AGL), time and distance between two points over roads (to find the closest water resource)and others. Key Concept here: We are at a point with information where no one system can be isolated, and sharing data is critical.
There is a package called Open Street Layers that is open source, it’s a rough learning curve. Google Maps started with that and ran with it.
You can get an account with Google that comes with some amount of free access, when you exceed an amount they they will charge a credit card. For just your use, that effectively makes it free. The goal was to enable developers to make systems.
Adding in other data is key- I have icons that have links to traffic cameras, river water level gauges, I get radar images from the national weather service, process them, and overlay them. The fire departments have hydrant and water point information. So adding ADS-B low level flight information was a natural for landing zones for helicopters for patients, and then especially with the introduction of drones. I also import address databases from counties to save costs on Google Maps and be able to do geocoding without a slow net access. I loaded the FDA Medicine Database for patient care on initial contact .
This is a big system, have been at it full time for 10 years, over 250K lines of code, about 50 modules, and the underlying database of information is 50+ tables with about 700 columns of information. Any mapping view that is nice has a lot of time to generate. The PiAware SkyAware web page is work of art.
I did have to create one more table of information for this project. It lets me enter aircraft I see locally. I’ve also downloaded and imported the FAA registration database. My new database table also lets me select what icon to associate with any given Hex ID. The ability to add additional text is critical. For both everyday use and deployed scenarios, you can provide more information to the responders than is in the FAA database. Pictures enclosed.
So depending on your programming skill, Google Maps is nice but if you plan to share, it will cost money. Open Street Layers is free. Both have a learning curve on their library to create and manage icons, draw lines, draw polygons that overlay over the base map. I suspect there are similar systems in Microsoft land, I just don’t swim on those waters. Three seconds after you draw your first map, everything changes to getting and displaying the data relevant to your use case.
Meanwhile i switched over to ModesMixer2
This little tool is not only able to receive and distribute the different streams, but also capable to write all recorded flights into an SQLite Database. It’s pretty much the same as used by the Virtual Radar Server.
The tool itself takes most of all available data formats (Beast, ADSB, AVR etc) as input, decodes the data and write a record.
The database now has two tables. One with the latest aircraft information (currently > 200.000 records) and a table “flights” where the flights of my receiver are recorded. Using a unique column in both tables gives the ability to join them together.
Result: A list of flights together with the aircraft information.
If needed it can be also linked to a different database with flight information.
Another more comfortable option would be using Virtual Radar Server with the database writer plugin. That is doing the similar thing, but the database then contain already all details (e.g. routes, airports, aircraft etc.)
ModesMixer2 does have a web interface with tables, stats and a map.
It can be used with only few ressources on the same system as the feeder or a different device accessing the feeder via network.
You’re confusing a couple of things here.
Openstreetmap is a user-built open data set and source of map tiles.
Openlayers is an open-source map display library that handles fetching tiles and displaying them.
Leaflet is another open-source map display library that is similar.
Google maps combines the (proprietary) tile source and the (proprietary) display library (which means that migrating away from it later is a big deal, it’s not just a case of swapping to a new datasource if you decide you need something that Google won’t provide at a price you’re willing to pay)
dump1090 originally used google maps. dump1090-mutability migrated to openlayers + openstreetmap when that became too painful to continue using the google maps API. Derivatives like SkyAware & adsbxchange continue to use openlayers.
Yes, I remember it.
It happened in July 2016, and there was lot of opposition and hue & cry by users, mainly because we were addicted to Google Maps 
Your are correct, I was mixed up in my names of things.
For the deployed responders, I have the USGS (United States Geological Survey) map tile set up to a zoom level of 16, which takes about 860 GB on the edge severs. It’s not too bad in Google Maps to bring in another base tile set, I can select the Openstreetmap development tiles. On my to-do list is to get an account with one of the companies that servers up the openstreetmap tiles.
Google does present a dilemma, in that they have the various features I mentioned. I am always careful when using a library like that to just use the minimum number of features. I’ve got the beginnings of a Openlayers web page going for deployed responders with no Internet connection. The change at Google from a fixed price to a variable one is also a driving factor, it completely changes the economics of making a service available. And again for responders, you have to have in Internet connection to use Google. Google is like a small car: Easy to get into and hard to get out of. Once I get the mapping good, then I suspect I’ll still use Google Maps and let the “pay as you go” work for me as I would only use it for routes, geocoding and lat/long to elevation for the AGL world that responders live in. And since the Google tiles and OpenStreetMap tiles have different information, it will be good to let the responders select which one they want to see- they can switch back and forth at any time.
The PiAware SkyAware web page is just amazing, great job! Another challenge with Google is handling memory leaks with canvas elements. Safari is particularly bad about dealing with that. You site has no problems there.
And thank you for clarifying the terminology, I got a big lazy there, sorry.
Great info, thanks! ModesMixer2 sounds like a data processing beast !
The world of sharing data is interesting. Generally speaking, just about every set of users wants only a subset of the data that is available. And of course the specific hardware platforms drive that too. The PHP programs are pretty small, around 300 lines or so for all each module. That works only because the subset of all the aviation information needed is small and focused. When there is an Internet connection, then the Google lat/long to elevation makes it all work. When there is not, the responders will need to have some idea of their elevation, which they can get from the USGS topo map tile set.
I’m also constrained a bit in that my operating system of choice is Unix, not Linux. So for importing and exporting data, my first task is to get the data into the world I’m comfortable in. As such, the PI is the source of data.
Does ModesMixer2 run native on the Pi? If so, what is the impact of running a database on a system that uses flash based storage w/r/t/ TBW limitations? I would imagine the Virtual Radar Server would want to run on a larger machine or require a large sized SD card. I’ve spent a lot of time finally getting MySql to run out of a ram disk so that the edge server can be SSD based. It was not a trivial project, and with it comes some restrictions that users must accept. But now even the new hosted sever I’m deploying can run with SSD handling 50 to 100 updates a second and the SSDs will take about 25 years to hit their TBW limit.
w/r/t (With respect to, i keep using that abreviation) users getting upset and the change over away from Google maps, any change to a GUI (Graphical User Interface) always comes with such turmoil.
Thanks for the information on both packages, if someone is looking to play with the data and doesn’t have my “must work in the field with no internet connection” constraint it clearly would be the way to go. You only want to write the code you can’t find that is specific to your use case.
You can install it on RPI by a bash script, which
For 32-bit Raspberry Pi OS Buster & Bullseye
sudo bash -c "$(wget -O - https://raw.githubusercontent.com/abcd567a/mm2/master/install-mm2.sh)"
I combine data of 3 RPis and feed various sites
My config file:
sudo nano /usr/share/mm2/mm2.conf
--inConnectId 10.0.0.21:30005:Pi1
--inConnectId 10.0.0.22:30005:Pi2
--inConnectId 10.0.0.23:30005:Pi3
--inConnectId 10.0.0.21:30157:Adsbx-Pi1
--inConnectId 10.0.0.22:30157:Adsbx-Pi2
--inConnectId 10.0.0.23:30157:Adsbx-Pi3
--outConnect msg:data.adsbhub.org:5001
--outServer avr:31002
--outServer beast:32003
--outServer beast:31005
--web 8787
--location 43.5xx:-79.6xx
Very nice. This has me thinking that adding modmixer2 and the web access you mentioned could be added to the responder tool box for scenarios where there is high speed internet access.
For anyone wanting to play with the raw data (because sometimes writing software is a great hobby), this sounds like a good way to get the raw data. Of course, one can always parse the raw data “just for fun”.
Can you advise on the impact on the SD card of all the database activity over a long period of time (a few years) ?
Does this run OK on a PI 3, or would it be better suited for a PI 4 ?
I am using it on an 8 Gb Class 10 microSD card of OrangePiPCi for over 2 years with no sign of failure of microSD card. However to avoid filling of microSD card, I have created a Windows share of VRS database, so that it can be accessed by ModeSMixer2 over network. The other alternative I have used was to copy (by SCP) the VRS database to the Pi running ModeSMixer2
That OrangePiPC has following:
I have successfully installed and used ModeSMixer2 on:
I still have my first SD card (SanDisk 64 GB) in the device.
The database is SQLite, write access is done in blocks after a short period of buffering.
You could improve it to have the database stored on a ramdisk and copy it to the file system in a defined time period (e.g. once per hour or per day). That reduces DB writes, but also increase the risk of a data los at least for the given period.
I have ModesMixer2 running on a Raspberry 3B
On this device also runs WeeWX (weather station software), the Planefinder feeder, the Radarbox Feeder (both getting the stream from my Jetvision device)and ModesMixer2
The Pi 3 does not have any issues with it, that’s the CPU load over the last two weeks:
The drop yesterday was caused by some testing. ModesMixer2 usually does not use more than 5% of CPU
A bit off topic, but hopefully someone might know. What is the raw data output speed of ADS-B data of the Pro-Stick Plus? I’m looking at a new device from Aerobits which claim an output speed of 3Mb. I might be mixing apples and oranges at this point.
Yes you are.
Also the claimed output speed is something irrelevant, it’s like saying rtl-sdr communicated at 480 MBit/s (USB2 speed).
Likeley the device is not compatible with the MLAT system in use on several aggregation sites.
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.
