Here are a handful of specific examples. It appeared that most of the flights that were mis-labeled departed between about 12:00 and 15:00 UTC, and remain flagged as “scheduled” despite actual tracks visible on the map complete with departure and arrival times.
It appears that subsequent legs of these flights were recorded properly so track logs and KML tracks could be accessed, so this could have been a glitch of some sort.
shows as “Scheduled”.
contrast to this following trip that was properly recorded:
Here is another bogus “Scheduled”, which appears like an upcoming flight although it completed hours ago:
And here is the return trip which appears to have been properly recorded:
From the KATS / 7T7 flight link, the early morning 7T7 - KATS flight shows up as a past flight, but when you click on it, it shows up as “scheduled” still. Note that on both of these phantom scheduled flights, you can replay the actual flight, so the data is somewhere on the server, but apparently mis-labeled somehow.
Normally on any morning under VFR conditions there is a gaggle of somewhere between 10 and 21 aircraft which depart 7T7 over about a 30 minute window, fan out all over the oil field regions, fly 4 to 8 hours of grid search patterns, and return to base. Some individual aircraft make 2, sometimes even 3 separate flights out and back on the same day, and they appear to rotate specific routes.
Here is another bogus “scheduled” flight that actually has been completed today, that did not originate or arrive at 7T7:
This departure was at 17:31 Zulu, 11:31 local, landing at 12:24 local, which was the latest one improperly flagged that I know of for certain (but I probably missed a few).
Since there are normally about 12 to 16 aircraft that depart 7T7 in a gaggle within about 30 minutes of local sunrise under VFR conditions, I suspect that 5 to 9 flights were not logged at all. I’ll have to sift through my .json and .csv files for this morning to look for other hidden flights that were heard directly but not logged by FlightAware.
Thus, the times of 3 flights mistakenly marked as “scheduled” interleaved with other flights recorded properly with departures between 7:30 and 11:31 Central time, make me think something intermittent rather than a single outage.
Whatever is going on, though, subsequent flights appear to get logged properly, while those improperly flagged were not corrected as subsequent flights go on like the laying down of sedimentary layers in the geological records.
Definitely some kind of new bug introduced yesterday. Today, there are 16 aircraft up from 7T7 at this moment, but eleven of those clearly airborne are shown as “scheduled”. Only 5 of the 16 are being processed correctly.
In nearly a year I have never, ever seen this as there are no scheduled flights out of 7T7, only nearby KMAF.
Whatever software upgrade someone did Tuesday night / Wednesday morning, they clearly broke something or uncovered a previously undiscovered bug.
Meanwhile, I see no such erronious scheduled flights show up from nearby KODO. Meanwhile, KMDD does show several scheduled executive bizjet type flights, which is somewhat expected.
Screen shots to document the specific malfunction.
I thought of another possible failure mechanism. Possibly the oil field patrol company has started filing flight plans using a new system, which then is confusing the FlightAware software if the flight plans were not properly opened? Whatever is going on, the flight tracks are clearly making it to FlightAware, but the actual flight is never flagged as en route or completed even though they can be clearly seen on live maps.
The above may provide more clues as to the nature of the newly introduced bug. Note the inconsistency between “Scheduled” and the actual flight. This capture time was after the return flight from Brownfield had already been logged. Now let’s scroll down to the list portion of the page:
Specifically note the time stamp in the Past Flights of the return trip from Brownfield to Midland, departing at 11:47 AM and arriving at 1:32 AM.
Now specifically note the “Upcoming Flights” which shows a departure at 07:05 AM and arrival at Brownfield at 11:25 AM. An upcoming flight shown several hours before a past flight.
OK, now if we click specifically on the completed Brownfield to Midland flight, we get the following screen shots.
It looks normal, and has provisions to view the track log and on that page save the KML file. Specifically scroll back and look at the “Scheduled” page where the aircraft clearly completed the trip (and confirmed by checking my feeder log files), and the conspicuous absence of any access to the track log, because the server thinks that it is a scheduled flight rather than a completed one.
Scroll back down to the second half of the Brownfield to Midland page. Specifically note that the “Scheduled” Midland to Brownfield flight is shown from this page as a completed flight. Yet if you click on it, it takes you back to the scheduled page, ad infinitum.
My suspicion just from years of industrial type programming - something corrupted part of the database, and the system does not have a “sanity check” provision to detect and fix these kinds of internal inconsistencies. The fact that a subsequent flight made the “bogus scheduled” flight appear as a normal past flight, yet clicking on that link returns to the corrupted page, and this has been only observed in the past 2 days, has the fingerprints of a newly introduced bug from some recent update that corrupts new database entries only under certain “magic circumstances”.
I draw that hypothesis from observing about 1 in 3 to 1 in 4 flights departing 7T7 which appear as normal. The fact that all of the aircraft are not from the same owner, and I think I found at least one originating from a different airport with the same kind of problem, strongly indicates that it was not a sudden change in procedure by the aircraft operator that triggered this, but an oddly specific bug somewhere within FlightAware.
I hope the screen shots and very verbose details of what has been observed can give software engineers a few more clues to narrow down where the bug actually lurks, and maybe have a better chance to find the underlying cause without too many long sleepless nights.
Since the error has proven to be repeatable consistently with departures from 7T7, and there are the same general group of aircraft that depart and return on every single day that conditions are VFR to MVFR, at near the same time, this may be a good place to observe after a bug fix attempt to see if the bug was truly found and swatted, as opposed to the problem showing up far more infrequently and thus far more difficult to observe in the wild with departures from other airports.
Whatever is going on, I hope y’all are able to find and fix it soon. I am surprised to apparently be the first to have observed and reported this. So we’ll wait and see.
Another observational update:
Maybe a partial workaround patch has been done to put a link to the track log anyway? This specific example from a flight last night that was not near 7T7 still shows “scheduled” but has an available link to get to flight tracks, which at least for me is a sufficient workaround for the moment.
It still looks like as of late Thursday night to early Friday morning, someone at least may have put a bandage on the problem to buy time while attempting to find the underlying bug(s). I suspect it means searching through hundreds of separate programs and modules each of which may have hundreds to thousands of dependencies and result in tens of millions of lines of executable code.
That could be like trying to find a needle in a haystack, after figuring out which state and county the farm is in which contains the field which contains the haystack in it, then which haystack in that field has the needle. Although admittedly if you have a dangerously powerful neodymium magnet in the tens of kilogauss realm, and the needle is made of ferrous metal, finding the needle can be much easier than if it were made of, say, straw colored plastic or ceramic.
I’ll keep collecting evidence as long as it’s helpful to whoever is working on the fix and provides clues where to look.
I also did a test with one of the “scheduled” flights (which have all vanished except for the open tabs from Wednesday), just appended tracklog to the URL of the “scheduled” flight, and was able to save the track.
from
It appears the others from Wednesday / Thursday only show up now by querying the specific aircraft. No “scheduled” departures appear on the specific airport page on the flights tab any more although the truly en route ones show up on the map. Fortunately I have a good aircraft list stored off-line of every departure or arrival that my feeder logged.
And I found another little workaround for those that don’t appear in the airport departures / arrivals, although this might be a pain in the rear for most other people who have not written some custom code to archive this stuff off-line.
On the flights that show up as “scheduled” but were really complete, you can manually type in to append to the URL (not the live but the upcoming / past pages) ‘/google-earth’ (not including the quotes) and get the flight track although there is no link available.
This looks like a great starting point to write a Python script that will just parse the URL, build up a time stamp compliant filename that will be unique even when more than one flight between the same departure and destination takes place on the same day in the same aircraft, and feed that wget so it fetches the KML and shoves it into the compliant filename.
Just checked the oil field patrol list and it looks like about 2/3 to 3/4 of the actual en route flights (as I can see them on the feeder “radar screen”) are marked as “scheduled”. But the direct URL modification workaround did let me go back and construct deep links to get those tracks that are still not accessible by just following links. I’ll see if any of them get converted to completed past flights or remain bogus “scheduled” as the day wears on and I see them land.
Unmatched scheduled flights get expired/cancelled if we don’t find confirmatory data within some timeout after the expected departure, so any problems where we’re unable to match a scheduled flight to in-the-air data will clean themselves up after a bit.
I currently see no departures scheduled out of 7T7, is this still a problem?
The scheduled departures all vanished,which looks more like a Band-Aid patch to hide the symptom, but the underlying disease is still there.
It appears that a significant fraction of completed flights to and from other locations in addition to 7T7 randomly appear as “scheduled”. Here are examples in and out of KSJT this weekend.
Example 1:
Example 2:
So the above two did not “drop off the map” at all, or get marked as “estimated arrival”. They were just incorrectly marked as scheduled, consistent with the bug that first appeared Wednesday, November 10th and still persists.
Looking directly at the aircraft’s page, notice the “upcoming flight” that clearly was completed, and in the list, the subsequent flight shown as completed. That and the first dynamic link in the page go to the examples above:
An example of “first seen” and good airport landing (I suspect it really took off from Del Rio) which show up as past flights as they should, (although of course the bug involving a presumed off-airport departure mentioned in another thread fails to put the arrival into the KML file once you click on the track log):
and another “normal” example:
Note there are no flights from KSJT in that part of the aircraft history. Presumably it got to Del Rio somehow, and I doubt the ADSB beacon was off or all the feeders were down both times and miraculously booted back up on different days. Those are what I think are flights that were just discarded outright.
So far I haven’t seen a definite repeatable pattern tied to any specific airport or aircraft, and think it looks kind of random.
My experiences with programming over the years tell me this could be some kind of a logic race taking place between two or more processes. These can be maddeningly difficult to replicate consistently enough to actually figure out what is going on, especially if the application has a slow / infrequent enough sample rate.
Checking the 7T7 page (20:09 UTC Nov 15 2021) shows 10 departures today and 10 arrivals.
Checking my aircraft “favorites”, I counted only 8 departures and arrivals of the known patrol aircraft with 7T7 today.
Following the direct aircraft links from my off-line list, I counted 5 more that were NOT listed as departures / arrivals from 7T7 which clearly were, and showed up as “scheduled”.
One showed as having departed / arrived but was neither linked from the airport departure / arrival list nor mistakenly shown as "scheduled.
So someone put a Band-Aid on the “scheduled” list on the airport pages to simply make a futile attempt to hide these mis-handled flights, but didn’t really fix the underlying problem.
Complete “roll call” from my hand-curated off-line database of aircraft known to be based at 7T7 that usually fly daily that I manually did a few minutes ago (sorted by tail number):
N12783 - missing from 7T7 list, completed flight today shown as “scheduled”
N12901 - present in 7T7 list, completed flight today shown as normal 1
N13683 - missing from 7T7 list, completed flight today shown as “scheduled”
N13692 - Present, completed, normal 2
N20341 - missing from 7T7 list, completed flight today shown as “scheduled”
N20823 - Present, completed, normal 3
N323SM - Present, completed, normal 4
N4384R - probably not flown today
N53822 - Present, completed, normal 5
N5383K - Absent from 7T7 list, completed flight today shown as normal
N5480K - out of state for week, probably not flown today
N55374 - out of state this week, completed flight, normal x
N61672 - missing from 7T7 list, completed flight today shown as “scheduled”
N64219 - Present, completed, normal 6
N64542 - probably not flown today
N7082Q - probably not flown today
N7128Q - missing from 7T7 list, completed flight today shown as “scheduled”
N7156Q - Present, completed 2 flights, normal 7, 10
N7199G - Present, completed, normal 8
N7823G - probably not flown today
N7857G - probably not flown today
N7970G - probably not flown today
N80490 - Present, completed, normal 9
Totals completed flights 15 November 2021 from 7T7:
9 aircraft, 10 departures logged properly
5 aircraft, 5 departures NOT logged, completed, show “Scheduled / upcoming” in aircraft history.
1 aircraft 1 departure NOT logged, or shown as “Scheduled / upcoming” in aircraft history
15 aircraft total from 7T7, 16 departures (1 aircraft twice), 10 correctly logged, 6 not logged,
Bogus “scheduled” error rate: 31.25%
Missing entirely error rate: 6.25%
Total error rate for 15 November 2021: 37.5%
I can understand an occasional glitch, but this is a substantial fraction of GA VFR flights that are not appearing at all on the departure list. This set of errors first appeared on Wednesday, November 10 and continue to occur. I would rather see bogus “scheduled” departures than see them vanish entirely.
So far I have not spotted any missing departures / arrivals from aircraft solely running 978 mhz UAT but then these are a tiny fraction of the 1090 mhz ADSB data points.
There is definitely a new bug that was introduced on the 10th. Hopefully this new set of data can be passed to the system engineers to yield some clues as to what they might have broken.
I’ll keep checking my off-line “roll call” every afternoon and compare with the “official” 7T7 log and keep documenting these errors and certainly notice when the underlying problem is truly fixed.
Don’t let them do what I saw at one place a few years back, which, when faced with a bunch of “in your face” errors and users complaining about weird inconsistencies in their reports, management didn’t want to pull any of their system engineers off of another big project to dig into it and find the root cause. Instead, the IT folks were ordered to just omit the “funny lines” from the report.
So they did. Until the night a pipe blew apart from a vessel due to overpressure from an intermittent valve and a pump not being told to shut off. It turned out the pressure transducer hardware could report values up to 0FFF but the algorithm, assumed it would never exceed 07FF. So when the actual pressure resulted in an A-D count of 0800 or higher rolled over to 0000. The code had been hastily put together late one night in the midst of another rushed assignment, and the “just get it to work for the dog and pony show” code stayed in when the thing went live. They had balked at the price tag of a solid well-debugged code library and licensing fees, and decided to just write their own on a shoestring.
So if I understand correctly, to summarize your bug report, the bug is that this flight:
shows as “scheduled” when it should be shown as completed?
I’ll raise a ticket for that.
Unlikely to be a race condition in the traditional sense - the majority of the data fusion in Hyperfeed happens deterministically. This one looks like a presentation bug, for historical reasons the set of rules for turning the underlying database row into a status on the flight page are complicated.
Thank you for the detailed bug report. There was indeed an underlying flight tracking problem which we fixed and released to production yesterday. Hyperfeed sometimes holds onto schedule and position data until it is confident that a flight will actually occur. When that confidence threshold is achieved, releasing the held information with correct ordering is a complex process. The bug was internal to that system, and it caused the odd downstream effects you discovered.
Thanks. Awesome.
Yep, it could be a presentation bug.
The related symptom is, when you look at the departures / arrivals from an airport, when these incorrectly marked flights are en route, they do not appear as departures and en route either.
I first discovered the bug when I started seeing some arrivals on return trips that had no corresponding departures and no overnight stops or legs between two other airports, and wondered “how did they get to KATS from 7T7 when they clearly arrived from KHOB to 7T7 hours later and looking at the aircraft track, there was a KATS to KHOB leg?”
Then since about 1/4 of that traffic passes either directly overhead or so close I can grab the binoculars out of the truck and read the tail numbers both in-bound and out-bound, it becomes conspicuous when direct visual ground truth did not align with the database.
You’re welcome. I have learned from having been on both the tech support end of things and the field user end of stuff that the more details that can be observed and noted of a phenomena, the better the odds of finding the cause, especially if it is an intermittent / transitory or a “hit and miss” kind of thing that is hard to replicate consistently.
Just checked my off-line list against the departure list and it appears to be fixed. All of the aircraft except one in my off-line hand-maintained list either show completed flights or en route this morning.
The only one that shows up as “Scheduled” apparently did not fly on Tuesday, and the “upcoming flight” is from a week ago, despite several subsequent flights over the weekend and Monday.
It kind of makes me think that it takes some new flights after the bug fix “for the rat to pass all the way through the snake and its remnants come out the other end”.
But this is the only artifact that remains that I found doing a “roll call” of the same list I checked Monday.
The 7T7 departure page list appears to match my roll call so far.
I have a whole other thing that maybe you can look at that goes back quite a while, concerning KML files where there is a bogus arrival and / or departure.
While I manually edit the files after downloading to fix this, it would be nice to do it upstream at the server level.
That is the two fake airport placemark tags at the start of the KML file, which have coordinates of zero, zero.
Symptom appears in Google Earth when you open one of these files (if unedited first). It gives you whiplash because the globe spins violently around to the Atlantic Ocean off the coast of Africa, thousands of miles from where you were looking. Because, of course, coordinates of zero, zero are there, where the equator intersects the Prime Meridian.
My workaround is to just delete those placemarks from the KML file outright. I replace them with a comment “invalid departure” or “invalid arrival”. It keeps Google Earth happy.
A better solution is to replace the bogus zero zero lat/long with the first seen coordinates in the linestring, or last seen coordinates, respectively.
Eventually when I’m fiddling around with file and string manipulation within a file with Python I might do exactly that - open the file, and replace those coordinates.
Second - “first seen near” type flights are not putting the arrival airport into the KML file content or into the filename. This has been going on for at least a year, probably a lot longer.
An on-airport departure and off-airport track end will show up as (airport)_INVALID in the filename, and the departure airport appears in the KML file - so it’s just the bogus zero zero coordinate “arrival” that I have to go fix.
But an off-airport departure and on-airport arrival shows both as bogus. It should show INVALID_(airport) in the filename and have the arrival airport placemark in the KML for the naming and structure convention to be symmetrical / complementary and consistent.
