Timestamps

SESAPI · April 28, 2022, 8:27pm

timestamps from API are not coming in UTC, why and how to fix?

bovineone · April 29, 2022, 3:03pm

Can you show an example of a server response that you are talking about?

SESAPI · April 29, 2022, 3:26pm

Do mean the error message from databricks or the sample data with inconsistent timestamps?

bovineone · April 29, 2022, 3:48pm

Both, or anything you feel that is wrong. Your initial report is not actionable without more information.

SESAPI · April 29, 2022, 4:48pm

Hi,

Problem 1)

We generate the report via the attached code, the results we get are 2 files flight track and flight schedule, both attached

We noticed the time in flight track do not match the NMS UTC timestamps, we then found out that there is 4 hours offset from UTC, I need to get the timestamps in plane UTC

Problem 2)

We found some missed data (data gaps) example of one is attached, book7 file has example of the data gaps

Problem 3) to automate the API we would run it on DataBricks, and we got this error message that we are not sure what it means

“HTTPSConnectionPool(host=‘aeroapi.flightaware.com’, port=443): Max retries exceeded with url: /aeroapi/flights/N733KA (Caused by SSLError(SSLEOFError(8, ‘EOF occurred in violation of protocol (_ssl.c:1131)’)))”

Please let me know if you need additional information

Thank you

(Attachment Python.7z is missing)

(Attachment flights_track 25 April 2022.csv is missing)

(Attachment flights_schedule 25 April 2022.csv is missing)

(Attachment Book7.xlsx is missing)

SESAPI · April 29, 2022, 4:52pm

Hi bovineone,
looks like all the samples I sent you as requested got blocked, should I copy and past as content or do you have a site that I can put them there?

bovineone · April 29, 2022, 7:04pm

You can copy/paste the parts you think relevant. It’s generally unnecessary to include your entire program.

Additionally, all timestamps returned by AeroAPI should already be in ISO8601 format in UTC (timestamps ending in “Z”), so you’re likely doing some interpretation incorrectly. If you’re relying on the output in Excel, then it may be Excel that is mis-handling or mis-displaying it.

SESAPI · May 6, 2022, 9:28pm

I’m not applying any transformation on the data pulled from the API, I use ipynb file

from aero_api import AeroAPI
aero = AeroAPI()
aero.get_flights_schedule()
aero.schedule_df.to_csv('flights_schedule.csv', index = False)
aero.get_flights_tracks()
aero.track_df.to_csv('flights_track.csv', index = False)

to call aero_api.py
from concurrent.futures import ThreadPoolExecutor
import pandas as pd
from tqdm.notebook import tqdm
import requests
class AeroAPI():
    def __init__(self):
        self.tails = ["N733KA","N415LT","9HILI","FHXRG","9HLXX","A6MVD","N604RC","9HILV","9HVJO","9HILZ","9HVFI","9HVJQ","N336EB","9HVJY","9HVJP","N999NB","N455KH","9HILA","A6CBO","9HILY","ANNAPLAB","9HVJS","9HVJR","9HVJT","9HILB","9HVJW","N811TD","9HVJU","9HVJJ","9HVJK","9HVJC","N357VJ","9HVFB","9HVFE","9HVJX","9HVJV","9KGAA","9HVJD","N503VJ","9HVJZ","9HVFC","9HVJG"]
        self.count = 0
        self.tails_idents = {}
        self.idents_flights = {}
        self.schedule_data = []
        self.track_data = []
        self.schedule_df = pd.DataFrame()
        self.track_df = pd.DataFrame()
        self.header = {"x-apikey" : "xxxxxxxxxxxxxxxxx"}
    def get_flight_schedule_of_tail(self, tail):
        self.response = requests.get(f"removedhhtps://aeroapi.flightaware.com/aeroapi/flights/{tail}", headers=self.header)
        df = pd.DataFrame()
        ident = None
        flight = None
        self.count += 1
        if len(self.response.json()["flights"]) > 0:
            ident = self.response.json()["flights"][0]["ident"]
            flight = [flight['fa_flight_id'] for flight in self.response.json()["flights"]]
            df = pd.json_normalize(self.response.json()["flights"])
            df["tail"] = [tail]*len(df)
        output = [tail,ident,flight,df]
        return output
    def get_flights_schedule(self):
        with ThreadPoolExecutor() as executor:
            self.output  = list(tqdm(executor.map(self.get_flight_schedule_of_tail, self.tails), total = len(self.tails)))
        for tail, ident, flight, df in self.output:
            if ident is not None and flight is not None:
                self.tails_idents[tail] = ident
                self.idents_flights[ident] = flight
                self.schedule_data.append(df)
        self.schedule_df = pd.concat(list(self.schedule_data))
    def get_track_of_tail(self, ident_and_flight):
        ident, flight = ident_and_flight
        self.response = requests.get(f"removedhttps://aeroapi.flightaware.com/aeroapi/flights/{flight}/track", headers=self.header)
        df = pd.json_normalize(self.response.json()['positions'])
        df["fa_flight_id"] = [flight]*len(df)
        tail = list(self.tails_idents.keys())[list(self.tails_idents.values()).index(ident)]
        df["tail"] = [tail]*len(df)
        return df
    def get_flights_tracks(self):
        self.idents_and_flights = [(ident,flight) for ident in self.idents_flights for flight in self.idents_flights[ident]]
        with ThreadPoolExecutor() as executor:
            self.track_data = list(tqdm(executor.map(self.get_track_of_tail, self.idents_and_flights), total=len(self.idents_and_flights)))
        self.track_df = pd.concat(self.track_data)
        self.track_df = self.track_df.drop_duplicates()

the output is 2 csv files for flight tracking and schedule, here is and example of data gaps we got

Tail	fa_flight_id	FT_timestamp	FA_altitude	latitude	longitude
N733KA	KAI99-1650135201-1-0-227	4/16/2022 18:45	4454.545455	19.74504545	-156.1240591
N733KA	KAI99-1650135201-1-0-227	4/16/2022 18:50	15890.90909	20.02273545	-156.0981818
N733KA	KAI99-1650135201-1-0-227	4/16/2022 18:55	25945.45455	20.43895273	-155.6439718
N733KA	KAI99-1650135201-1-0-227	4/16/2022 19:00	33630	21.056892	-155.36452
N733KA	KAI99-1650135201-1-0-227	4/16/2022 19:05	38290	21.619238	-154.966581
N733KA	KAI99-1650135201-1-0-227	4/16/2022 19:10	39000	22.200908	-154.541818
N733KA	KAI99-1650135201-1-0-227	4/16/2022 19:15	39000	22.76809556	-154.1233811
N733KA	KAI99-1650135201-1-0-227	4/16/2022 19:20	39000	23.32852	-153.7059156
N733KA	KAI99-1650135201-1-0-227	4/16/2022 19:25	39000	23.86056444	-153.21314
N733KA	KAI99-1650135201-1-0-227	4/16/2022 19:30	39000	24.130165	-152.77964
N733KA		4/16/2022 19:35
N733KA		4/16/2022 19:40
N733KA		4/16/2022 19:45
N733KA		4/16/2022 19:50
N733KA		4/16/2022 19:55
N733KA		4/16/2022 20:00
N733KA		4/16/2022 20:05
N733KA		4/16/2022 20:10
N733KA	KAI99-1650135201-1-0-227	4/16/2022 20:15	40000	27.23333	-147.21667
N733KA		4/16/2022 20:20
N733KA		4/16/2022 20:25
N733KA		4/16/2022 20:30
N733KA		4/16/2022 20:35
N733KA		4/16/2022 20:40
N733KA		4/16/2022 20:45
N733KA		4/16/2022 20:50
N733KA		4/16/2022 20:55
N733KA		4/16/2022 21:00
N733KA		4/16/2022 21:05
N733KA	KAI99-1650135201-1-0-227	4/16/2022 21:10	40000	30.4	-140.7
N733KA		4/16/2022 21:15
N733KA		4/17/2022 21:20
N733KA		4/18/2022 21:25
N733KA		4/19/2022 21:30
N733KA		4/19/2022 21:35
N733KA		4/19/2022 21:40
N733KA		4/19/2022 21:45
N733KA	KAI99-1650135201-1-0-227	4/16/2022 21:50	40000	33.28333	-133.61667
N733KA		4/19/2022 21:55
N733KA		4/20/2022 22:00
N733KA		4/20/2022 22:05
N733KA		4/20/2022 22:10
N733KA		4/20/2022 22:15
N733KA		4/20/2022 22:20
N733KA		4/20/2022 22:25
N733KA		4/20/2022 22:30
N733KA	KAI99-1650135201-1-0-227	4/16/2022 22:35	40000	35.61496	-126.49993
N733KA	KAI99-1650135201-1-0-227	4/16/2022 22:40	40000	35.88499857	-125.7683343
N733KA	KAI99-1650135201-1-0-227	4/16/2022 22:45	39500	36.26235	-124.987817
N733KA	KAI99-1650135201-1-0-227	4/16/2022 22:50	34130	36.641651	-124.19364
N733KA	KAI99-1650135201-1-0-227	4/16/2022 22:55	20970	36.996925	-123.432442
N733KA	KAI99-1650135201-1-0-227	4/16/2022 23:00	10109.09091	37.27439364	-122.8249509
N733KA	KAI99-1650135201-1-0-227	4/16/2022 23:05	5941.666667	37.49089	-122.39157
N733KA	KAI99-1650135201-1-0-227	4/16/2022 23:10	3492.857143	37.62640429	-122.087585
N733KA	KAI99-1650135201-1-0-227	4/16/2022 23:15	811.1111111	37.67333333	-122.1719389

notice that Take off 12:46PM HST should be 22:46UTC Landing 8:18PM PDT should be 3:18 UTC

another example is

Tail	fa_flight_id	FT_timestamp FA	FA_altitude	latitude	longitude
9HILA	VJT602-1650071601-ed-0001	4/16/2022 8:30	4330	45.523827	12.439617
9HILA	VJT602-1650071601-ed-0001	4/16/2022 8:35	14125	45.46195917	12.28361417
9HILA	VJT602-1650071601-ed-0001	4/16/2022 8:40	23600	45.492916	11.605315
9HILA	VJT602-1650071601-ed-0001	4/16/2022 8:45	28960	45.713791	10.939166
9HILA	VJT602-1650071601-ed-0001	4/16/2022 8:50	31940	45.898973	10.220898
9HILA	VJT602-1650071601-ed-0001	4/16/2022 8:55	35672.72727	45.96999364	9.432490909
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:00	36000	46.02909273	8.697575455
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:05	36000	46.146369	7.980882
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:10	36000	46.361198	7.276578
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:15	36000	46.58866818	6.511283636
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:20	36000	46.841586	5.847811
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:25	34991.66667	47.25110083	5.29463
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:30	28910	47.66604	4.736654
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:35	18470	48.061813	4.192474
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:40	10090	48.404715	3.709238
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:45	7000	48.67882091	3.282829091
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:50	5046.153846	48.82995231	2.762904615
9HILA	VJT602-1650071601-ed-0001	4/16/2022 9:55	3886.666667	48.850822	2.253849333
9HILA	VJT602-1650071601-ed-0001	4/16/2022 10:00	862.5	48.946305	2.35551
that is a flight between Paris and Venice, the difference from UTC take of at 02:32PM and landing at 04:04PM, there is 6 hours difference noted

nasageek · May 9, 2022, 4:19pm

Hello,

I wanted to inform you that we’ve decided to revoke the AeroAPI api key that you mistakenly included in your code snippet. This is to ensure that there is no unauthorized access to your account. As for the issue you’re experiencing with timestamps: we may be able to provide better support to you via email. You can use the “Contact Us” button at AeroAPI Developer Portal - FlightAware.

-Chris

SESAPI · May 9, 2022, 5:18pm

Hi nasageek,

I actually used the chat and opened a ticket, but they directed me to the discussion board as shown in the email, thank you for catching my error in providing the key on the discussion board, truly appreciated

The reason I provided the code I’m using is the response I got from bovineone, I extract csv not excel files via the API and wanted to make sure that I understand if we are doing anything wrong, or the code needs to be adjusted to provide the correct timestamp values

Thank you

nasageek · May 9, 2022, 6:28pm

I apologize, I think there was a misunderstanding of the level of support available to you. Email or chat should be an appropriate way to seek support.

Unfortunately, we are unable to support code we didn’t provide to you. To diagnose issues with AeroAPI, we would need to see a JSON response directly from the API. You should be able to gather such a response from the interactive documentation at AeroAPI Developer Portal - FlightAware.

PavelBraha · June 9, 2022, 6:03pm

Hello there

Did you fix the Problem 1?
I have the same problem with timestamp

SESAPI · June 9, 2022, 6:51pm

No, still have 4 hours difference, I added manually to my reports until we figure it out

PavelBraha · June 10, 2022, 11:55am

Thank you for your reply. If we come up with something, I’ll ping it here.

Topic		Replies	Views
Web site and flightXML2.0 API data does not match! AeroAPI	4	455	September 14, 2018
FlightXML Time Data AeroAPI	1	1579	October 9, 2013
Scrapping Flight Data AeroAPI	2	395	May 30, 2023
Erro message when using the API via DataBricks AeroAPI	0	528	May 6, 2022
Help with Python AeroAPI	1	632	September 25, 2017

Timestamps

Related Topics