FA Stats Aircraft Reported different from the numbers on my Pi


#1

the last days i wrote some scripts that make use of the data my raspberry pi harvests via dump1090. what confuses me now is that the unique hex counts per utc-day differ from what on my fa-stats page is shown. so i tried to gather this information with another different method - but again the numbers showed nearly the same (1 aircraft less) but very different result from flightaware.

example saturday 9/9/2017:
http://flightaware.com/adsb/stats/user/TomMuc#stats-60978

fa-stats: 3532 aircrafts
my pi: 3619 aircrafts

my first idea was - maybe there are some ‘one hit wonders’ means aircrafts where i got just one message and flightaware says these are just errors. so i looked into this - but no. there are two hex numbers with less than 100 messages - all others got from 100-16000 hits. while further investigating i found 11 starting with a ~ what means they are anonymized and because of that probably duplicates of real ones. another is hex 000000 with more than 70000 hits that probably is rubbish too.

but in the end i have 12 faulty hex numbers from 3619 => 3607 aircrafts remaining and that is still 75 more than my flightaware stats page shows!

did i overlook something? are my counts wrong or flightawares?

you find my first test-method here if you want run the same test for your site or have a look into for logic and/or code errors: Daily Log/E-Mail Aircraft Report directly on your RaspberryPi

cheers
tom

p.s. the second test generated 14,500,000 database entries and 1.6 GB data on a single utc day for my site. must be big fun for FA to handle this multiplied with 10,000 sites :smile:

p.p.s. i’m aware that of course the fa-sytem over the day lags a little - but yesterday it began similar …


#2

faup1090 uses different rules to your script to decide when to forward reports.
There is additional processing on the server side before accepting reports.
So it’s not surprising that the numbers are different. Neither are necessarily “wrong” - just counting different things.


#3

@obj

ok understand. what i already recognized was that faup uses just a small subset of received messages probably to limit bandwidth and computing demand on fa backend. that’s why i see about 800,000 to 1,000,000 messages/day in my stats while having about 15,000,000 entries in aircraft.json which already is a combined subset from raw-messages.

do you have an idea what kind of rule could prevent these 75 aircrafts/hex from being counted on fa side? i’m just asking because i did not find indications like e.g. the 000000 or single hits and moreover as all hex had more than 100 hits (except of 2) i found that faups lower frequency could not be a reason to miss single aircrafts completely …


#4

The faup1090 code is here if you want to compare: https://github.com/flightaware/dump1090/blob/0d7177fe9a68ffd7e1b779ccd8825c7f40d96968/net_io.c#L1816


#5

@obj

wow - thanx! will see if i’m capable to understand some parts of the code :smiley:

edit: to be honest i’ve been on github before and again obviously searched the wrong place. i downloaded the whole piaware container and searched with my texteditor all files for faup - but did not find something that enlightened me. so this file looks much more promising!


#6

ok - tried my best. what i found (and hopefully right understand) is that hex where only one message was found are skipped, some plausibility checks are done and the rhythm for messages transfer to flightaware depends on aircraft maneuver stability. but what validation process there could be the reason for the ‘missing’ hex numbers from my site i did not really figure out …

what i additionally figured out is that there is lot i can’t figure out with my skills :smile:

edit: or could heavily altering positions/altitudes within the group of messages for a given aircraft/hex be the reason for being dropped?