Hi guys,
I’ve made a script to discover how much of your ADS-B data FlightAware are actually using.
It’s super easy to install and it seems to work fairly well, but I’d love your feedback.
Please see here for full details:
1 Like
Looks interesting - How will it work if we have more than one receiver linked to our username?
FlightAware ADS-B contribution monitor has been launched
Downloading airport data...
/home/pi/fa-tracklog-contribution/backend/airport-library/icao2iata.csv: No such file or directory
Airport data downloaded successfully.
Current configuration (in config.me):
Installation Directory: /home/pi/fa-tracklog-contribution
FlightAware Username: KeithMa
Searching for the term "EGSQ" within FlightAware's tracklog pages
Each cycle repeats after: 60 seconds
------------------------------------------
Cycle 1 (Started at: 2023-10-17 10:48:33, Elapsed Time: 00h 00m 00s)...
Running step1-fa-stats-page.py...
Traceback (most recent call last):
File "step1-fa-stats-page.py", line 3, in <module>
from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'
Error executing step1-fa-stats-page.py
Running step2-download-tracklog-htmls.py...
Traceback (most recent call last):
File "step2-download-tracklog-htmls.py", line 3, in <module>
from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'
Error executing step2-download-tracklog-htmls.py
Running step3-scrape-local-htmls.py...
Traceback (most recent call last):
File "step3-scrape-local-htmls.py", line 6, in <module>
from bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'
Error executing step3-scrape-local-htmls.py
Running step4-build-final-report.py...
Error: Time zone ('tz') could not be determined from the tracklog CSVs.
Running step5-erase-temp-files.py...
Waiting for 1 seconds before erasing temporary files....
Cleaning up temporary files...
^CSleeping for 56 more seconds before starting the next cycle... (Cycle 1)
1 Like
I see no reason you couldn’t run this in multiple sessions, one session for each of your receivers. The essential part of this app is to find a phrase or text string which will uniquely identify your receiver within the Receiver Facility column of the tracklog page. I have found it more reliable to nominate my nearest airport for this, rather than the name of my receiver.
I would suggest finding an obscure local airfield near each of your receiver (which no other receivers in your area have already nominated as their "nearby receiver), and use this as your "Receiver Facility’ in the config.me file for each of the seperate sessions you run for this (for each of your receiver)
Lastly, I need to mention, that I have noticed that if you load the tracklog page too frequently, the Receiver Facility column mysteriously disappears. I suspect this is a defensive measure from Flight Aware, designed to throttle users who misbehave. If the Receiver Facility column doesn’t load, this app doesn’t work. It is essential to prevent your configuration from accessing FlightAware too often in too short a period of time.
As such, given you’ll be using one account to access tracklogs for multiple receivers, be mindful of the fact that all tracklog request for all your receivers will be linked to your single FlightAware user name. This may become a limitation for you.
Honestly, though. If everyone just sets “loop_frequency” in config.me to between 600-1800 (10-30 minutes) you won’t have any troubles with throttling by FlightAware. You really don’t need it to be any more frequent than that.
Let me know how you go!
I checked your config and it all looks good
Here is how yours would look once those dependencies are installed:
Hmm, no errors installing selenium or bs4 but it’s still not happy. I am using a Pi4 with Raspbian.
pi@pi4-tracker:~/fa-tracklog-contribution $ python3 -c “import selenium”
File "<string>", line 1
“import
^
SyntaxError: invalid character in identifier
pi@pi4-tracker:~/fa-tracklog-contribution $ python3 -c “import bs4”
File "<string>", line 1
“import
^
SyntaxError: invalid character in identifier
pi@pi4-tracker:~/fa-tracklog-contribution $ ./start.sh
FlightAware ADS-B contribution monitor has been launched
Downloading airport data...
/home/pi/fa-tracklog-contribution/backend/airport-library/icao2iata.csv: No such file or directory
Airport data downloaded successfully.
Current configuration (in config.me):
Installation Directory: /home/pi/fa-tracklog-contribution
FlightAware Username: KeithMa
Searching for the term "EGSQ" within FlightAware's tracklog pages
Each cycle repeats after: 60 seconds
------------------------------------------
Cycle 1 (Started at: 2023-10-17 11:05:46, Elapsed Time: 00h 00m 00s)...
Running step1-fa-stats-page.py...
Traceback (most recent call last):
File "/home/pi/.local/lib/python3.7/site-packages/selenium/webdriver/common/driver_finder.py", line 38, in get_path
path = SeleniumManager().driver_location(options) if path is None else path
File "/home/pi/.local/lib/python3.7/site-packages/selenium/webdriver/common/selenium_manager.py", line 81, in driver_location
args = [str(self.get_binary()), "--browser", browser]
File "/home/pi/.local/lib/python3.7/site-packages/selenium/webdriver/common/selenium_manager.py", line 60, in get_binary
if not path.is_file() and os.environ["CONDA_PREFIX"]:
File "/usr/lib/python3.7/os.py", line 678, in __getitem__
raise KeyError(key) from None
KeyError: 'CONDA_PREFIX'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "step1-fa-stats-page.py", line 65, in <module>
scrape_data()
File "step1-fa-stats-page.py", line 36, in scrape_data
driver = webdriver.Chrome(options=chrome_options)
File "/home/pi/.local/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 50, in __init__
keep_alive,
File "/home/pi/.local/lib/python3.7/site-packages/selenium/webdriver/chromium/webdriver.py", line 51, in __init__
self.service.path = DriverFinder.get_path(self.service, options)
File "/home/pi/.local/lib/python3.7/site-packages/selenium/webdriver/common/driver_finder.py", line 41, in get_path
raise NoSuchDriverException(msg) from err
selenium.common.exceptions.NoSuchDriverException: Message: Unable to obtain driver for chrome using Selenium Manager.; For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/driver_location
Error executing step1-fa-stats-page.py
Running step2-download-tracklog-htmls.py...
Error: tracklog_urls.tmp file not found.
Error executing step2-download-tracklog-htmls.py
Running step3-scrape-local-htmls.py...
/usr/lib/python3/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (2.0.6) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Running step4-build-final-report.py...
Error: Time zone ('tz') could not be determined from the tracklog CSVs.
Running step5-erase-temp-files.py...
^CTraceback (most recent call last):g temporary files...
File "step5-erase-temp-files.py", line 41, in <module>
main()
File "step5-erase-temp-files.py", line 31, in main
countdown_timer(cleanup_delay)
File "step5-erase-temp-files.py", line 9, in countdown_timer
time.sleep(1)
KeyboardInterrupt
Error executing step5-erase-temp-files.py
Sleeping for 32 more seconds before starting the next cycle... (Cycle 1)
@keithma I’ve obviously got some bugs to squash. I’m going to install this from scratch on a vanilla Raspberry Pi to iron out any dependency issues. Please hang tight, I’ll let you know when it’s ready
1 Like
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.