Capturing data from dump1090 (or from other ADSB sources)

FlightAware ADS-B Flight Tracking
1

So what to do with it aside from looking at historical data? Well just that… historical data!

Toronto has seen a fair bit of fog lately and thought what a better time to look at the data to see how it has affected arrivals.

After tinkering with Python and Apache Spark (fast db engine for analyzing big data) for a month, I’ve been able to come up with a script that will generate KML files which can be viewed in Google Earth. Can view all the flights that have been captured in the data, or look at flights that arrive/departs from a specific airport (basically by filtering out flights that falls within a certain coordinate and altitude), and the one thing I’ve been wanting to do, an easy way to create the KML files for flights to and from an airport and found to be in a holding pattern (previously had to eyeball these, needless to say it was a bit time consuming).

1024×576 182 KB

The Python script can work with data that has at a minimum… date/time, ICAO, coordinate, and altitude. Now wishing that all of the planes were using ADSB today (but will in the future, Jan 2020), MLAT data doesn’t quite have the accuracy and thus the flight paths when plotted is pretty rough around the edges.

2

Great!
If you dont mind, sharing the scripts and other details will help many.

3

Plagiarism is productivity tool! :wink:

4

are you sure? how about github? big plagiarism website?

Sent from my SM-N910F using Tapatalk

5

I’ve done something similar - see github.com/stephen-hocking/googleearthplot

6

Actually our code is on github but not quite ready to release it, need to pretty it up and make it a bit more “adaptable” but the getting it to work the way I want it to was the biggest challenge, especially finding those racetracks in the sky :slight_smile:

7

if you need some help with the code just tell me, I did recently a python course and need training to not forget. :blush::+1:

Sent from my SM-N910F using Tapatalk

8

First thing’s first, if you’re not saving your data to a CSV file then you’re not going to be able to generate anything :slight_smile:. I used vbscript to capture and store the data to a CSV file and decided to change it over to Python. Everything I’ve done is with Python 2.7. Comments are in the code if you want to use Python 3.4. Don’t ask me how to setup Python or Pyspark, there are resources that you can google to figure this out though installing Apache Spark/pyspark is a bit of a pain. Apache Spark/pyspark is what will be required for creating the KML files.

So what the below will do is capture everything that is on my ADSB receiver at 192.168.10.58. The line with url=, change this to your URL. The line with fileName, change the T:/TEMP to the folder where you want to save your CSV files. Note that the CSV files are named based on UTC date, so for example, dump1090_2017-01-30.csv.

Just run the script from a laptop and have the data saved locally to the laptop or to a network drive.



#import urllib.request      #for Python 3.4
import urllib2              #for Python 2.7
import json
import os
import time
from time import gmtime, strftime
from datetime import datetime

# change the below to your receiver IP address
url = 'http://192.168.10.58/dump1090/data/aircraft.json'
headers = {'Cache-Control':'no-cache','Pragma':'no-cache','If-Modified-Since':'Sat, 1 Jan 2000 00:00:00 GMT',}

while 1:
    #Python 3.4
    #request=urllib.request.Request(url, None, headers)
    #response = urllib.request.urlopen(request)
    #Python 2.7
    request=urllib2.Request(url, None, headers)
    response = urllib2.urlopen(request)

    data = response.read().decode('utf8')
    jsonPlane = json.loads(data)

    # change the below to the folder you want to save the CSV files, filenames format is dump1090_YYYY-MM-DD.csv
    fileName = 't:/temp/dump1090_'+datetime.utcnow().strftime("%Y-%m-%d")+'.csv'

    if os.path.isfile(fileName):
        ADSBFile = open(fileName, 'a')
    else:
        ADSBFile = open(fileName, 'w')
        ADSBFile.write('UTC,ICAO,flight,spd,lat,lon,altitude,mlat
')

    for plane in jsonPlane'aircraft']:

        resultStr = datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S") + ','

        if int(plane'seen']) <= 1:
            if 'seen_pos' in plane:
                if 'hex' in plane:
                    resultStr = resultStr +str(plane'hex']) + ','
                else:
                    resultStr = ','

                if 'flight' in plane:
                    resultStr = resultStr + str(plane'flight']).strip() + ','
                else:
                    resultStr = resultStr + ','

                if 'speed' in plane:
                    resultStr = resultStr + str(plane'speed']) + ','
                else:
                    resultStr = resultStr + ','

                if 'lat' in plane:
                    resultStr = resultStr + str(plane'lat']) + ','
                else:
                    resultStr = resultStr + ','

                if 'lon' in plane:
                    resultStr = resultStr + str(plane'lon']) + ','
                else:
                    resultStr = resultStr +','

                if 'altitude' in plane:
                    resultStr = resultStr + str(plane'altitude']) + ','
                else:
                    resultStr = resultStr +','

                if 'mlat' in plane:
                    resultStr = resultStr + 'True'
                else:
                    resultStr = resultStr + 'False'

                #write results to a file
                ADSBFile.write(resultStr+'
')
                print (resultStr)
    ADSBFile.close()
    time.sleep(1)

The contents of the CSV will be something like this:


UTC,icao,flight,spd,lat,lon,altitude,mlat
2017-01-30 03:58:00,ad2c19,AAL1360,479,42.410392,-80.064335,37000,False
2017-01-30 03:58:00,0c20a6,CMP470,413,42.646086,-79.040499,25475,False
2017-01-30 03:58:00,c013de,,492,45.459983,-81.852633,37025,True
2017-01-30 03:58:00,c04602,ANT511,453,46.588303,-78.498118,37000,False
2017-01-30 03:58:00,ac0417,ASH6101,472,43.187714,-81.081273,28875,False
2017-01-30 03:58:00,a60c7a,4104,463,42.726901,-79.686712,31700,True
2017-01-30 03:58:00,c04829,,318,43.461914,-79.463,9525,True
2017-01-30 03:58:00,c05d43,,179,44.010113,-79.70993,8800,True
2017-01-30 03:58:00,a836fb,UAL391,344,43.52178,-80.147029,14175,False
2017-01-30 03:58:00,c05ead,,282,44.078219,-80.3765,17725,True
2017-01-30 03:58:00,c00f7d,,202,43.754309,-79.404471,3250,True
2017-01-30 03:58:00,4ba95a,THY18A,491,43.703659,-78.226361,26650,False
2017-01-30 03:58:00,a88700,,441,43.943916,-81.547525,36125,True
2017-01-30 03:58:00,c028cd,WJA675,437,44.36142,-81.45316,33850,False
2017-01-30 03:58:00,c00b7d,SKV7668,291,43.665208,-79.579402,5675,True
2017-01-30 03:58:00,c080ac,WJA725,441,44.362152,-81.641759,32325,False
2017-01-30 03:58:00,c05bd2,ACA140,224,43.840104,-79.578683,4200,True
2017-01-30 03:58:00,c00b7c,SKV7598,485,44.212383,-78.037094,37000,True

You could put this into a mysql if you want, but I don’t so not sure how to get pyspark to pull the data from a db but pretty sure that is possible.

I’ll followup sometime this week with the code to analyze the CSV to generate the KML files.

9

Waiting for release of code at github. Thanks.

P.S.
I recall that you had another hobby of taking snapshots of birds. Am I right?

10

No, somebody else, mine is taking snapshots of race cars at the track :slight_smile:

11

Documenting the code took me a while but here it is finally!

github.com/tanner74/ADSBFlighttoKML