What's up with ADS-B Statistics?


#1

This thread is for updates on the ADS-B Statistics which are viewable at flightaware.com/adsb/stats or on your own “My ADS-B” page.

Statistics are calculated and updated on the web site on an hourly basis. During busy times statistics may get more than 1 hour “behind” real-time. Over time we address this with the addition of more hardware and further performance optimizations of the software.

3-Apr-2017 - Additional hardware deployed to support the continued rapid growth of the ADS-B network. We are adding over 100 new sites per week.
4-Apr-2017 - An issue has been identified where some sites’ statistics are not updating properly after the maintenance performed yesterday.
6-Apr-2017 - All known issues with statistics calculation are resolved as of around 16:00Z. Statistics should start generating in the next hour for sites that were experiencing missing stats. We are looking into filling-in missing daily stats for sites that experienced issues this week.
10-Apr-2017 - Stats for the April 3 and 4 were restored over the weekend. Stats for April 5 and 6 are incomplete and not yet recalculated.
12-Apr-2017 - Stats for April 5 and 6 are being regenerated.
13-Apr-2017 - Stats for April 5 are restored.
18-Apr-2017 - Stats for April 6 are pending while we make some further optimizations and perform testing.
2-May-2017 - Some users are reporting isolated issues with hourly or daily stats not appearing on their My ADS-B page. We are investigating this issue.
11-May-2017 - The issue with isolated stats missing in one-off intervals has been identified and a solution is under development.
16-May-2017 - Isolated missing stats issue has been resolved on a go-forward basis. Please report any further issues on this thread.
20-Jun-2017 - There are two known issues being investigated by FlightAware: low hourly counts in apparently random hours of the day (sporadic) and stats generation running more than an hour behind real-time during peak hours of the day for some sites. We are aware of the issues and working to address them with long term fixes.
7-Aug-2017 - All known issues with stats calculation have been resolved and scalability for future growth has been significantly improved.

Even though the stats may not be updating for your site, your data is still being received and captured by FlightAware for flight tracking (barring any other unrelated issue that your site may have at the time).


Announcing PiAware 3! (Latest version: 3.5.3)
Hourly Received Reports not showing updates from my station
Major drop in aircraft seen
lost data
lost data
Broken stats page?
#2

Great! Thanks for the update.


#3

Thanks for the explanation.


#4

They love us, they really do love us! :stuck_out_tongue:


#5

Back filling? Looks more like back emptying here :unamused: :

ads-b-flight-tracking-f21/lost-data-t38874.html


#6

just noticed, my stats for 04 are now gone completely, and the stats I did have for 03 were ok , but have been reduced significantly in the last 30 minutes. to next to nothing…


#7

Yes, that’s the backfill in progress; we clear the existing stats and then regenerate them from scratch. You’ll also see that you have a day-long hole in the hourly data, slowly filling in. Regenerating the stats is a slow process so it’ll be like this for a while.

edit: the 3rd’s daily totals have been restored. Data for the 4th is still processing; daily totals and hourly data are partially updated.


#8

Thanks for the update and the investment in money and time.


#9

Eric, is there still more to import on the 04/04?

I noticed now I have some stats but not all. 103,831 at present, there was at least 190,000 that day with 900 Aircraft. The hourly received reports dont tally up with the 4ths daily received/aircarft values.

Also 1am-2am on the 05th has no data either so looks like some data is still incomplete and you had not mentioned the 5th at all.


#10

See what I said:

There was a DB problem overnight that may have delayed the reprocessing further. Will look at it on Monday.
The regeneration is done day by day so we sort out the 4th before we look at the 5th.

To give you an idea of the processing required, to reprocess one day needs about 40-50 CPU-days of processing and 400GB of temporary storage, so regenerating several days is an involved process. It might take a week or more before everything is restored.

edit: also, you can’t sum the hourly aircraft values to get the daily values. If you could do that, the regeneration would have been a whole lot easier! Unfortunately you actually need to run the whole day’s data in order to get a correct count of unique aircraft.


#11

Thank you for the detailed analysis of the situation.


#12

04/03 and 04/04 are back but now my data from 04/05 has gone missing, where it was fine before?


#13

I just noticed that as well. Maybe 04/05 was mostly OK but they’re redoing it to catch what little was missing?


#14

Yes, exactly. Same for the 6th. Only a few hours / a few sites were affected but the simplest way to make sure everything ends up consistent is to regenerate the whole day.


#15

Just wondering if the regeneration f(this does start to sound like a Doctor Who board :smiley: ) for the 5th is working correctly.

After one hour is completely regenerated, the hourly position/aircraft values for the hour before are changed, eg. I had 22137/421 from 4 to 5, now after the values from 5 to 6 have been completed, the values from 4 to 5 dropped to 1404/124

https://www.dropbox.com/s/c3896h73cgean69/IMG_7405.JPG?dl=0

https://www.dropbox.com/s/2bwm9tb59ikosj4/IMG_7406.JPG?dl=0


#16

Now I can’t access the statistics page…


#17

I noticed the same thing and diagnosed the problem. It only affects the hourly data, not the daily data. I guess we’ll rerun the 5th again with that fixed. (It doesn’t affect the 4th)

Unrelated database performance issues that are making the whole thing a lot more “interesting”…


#18

not only that but each time a day was removed it was not from 00:00-23:59 but from 02:00 to 02:220 the next day.

my 5th day is short of numbers, but this has also affected upto 02:00 on the 6th morning.


#19

That is because the hourly stats are in your local timezone (and I think there’s an off by one display error in there somewhere too), but daily stats are in UTC.


#20

I’m seeing new holes in hourly data for the 4th. Daily data seems good.