the last days i wrote some scripts that make use of the data my raspberry pi harvests via dump1090. what confuses me now is that the unique hex counts per utc-day differ from what on my fa-stats page is shown. so i tried to gather this information with another different method - but again the numbers showed nearly the same (1 aircraft less) but very different result from flightaware.
my first idea was - maybe there are some āone hit wondersā means aircrafts where i got just one message and flightaware says these are just errors. so i looked into this - but no. there are two hex numbers with less than 100 messages - all others got from 100-16000 hits. while further investigating i found 11 starting with a ~ what means they are anonymized and because of that probably duplicates of real ones. another is hex 000000 with more than 70000 hits that probably is rubbish too.
but in the end i have 12 faulty hex numbers from 3619 => 3607 aircrafts remaining and that is still 75 more than my flightaware stats page shows!
did i overlook something? are my counts wrong or flightawares?
p.s. the second test generated 14,500,000 database entries and 1.6 GB data on a single utc day for my site. must be big fun for FA to handle this multiplied with 10,000 sites
p.p.s. iām aware that of course the fa-sytem over the day lags a little - but yesterday it began similar ā¦
faup1090 uses different rules to your script to decide when to forward reports.
There is additional processing on the server side before accepting reports.
So itās not surprising that the numbers are different. Neither are necessarily āwrongā - just counting different things.
ok understand. what i already recognized was that faup uses just a small subset of received messages probably to limit bandwidth and computing demand on fa backend. thatās why i see about 800,000 to 1,000,000 messages/day in my stats while having about 15,000,000 entries in aircraft.json which already is a combined subset from raw-messages.
do you have an idea what kind of rule could prevent these 75 aircrafts/hex from being counted on fa side? iām just asking because i did not find indications like e.g. the 000000 or single hits and moreover as all hex had more than 100 hits (except of 2) i found that faups lower frequency could not be a reason to miss single aircrafts completely ā¦
wow - thanx! will see if iām capable to understand some parts of the code
edit: to be honest iāve been on github before and again obviously searched the wrong place. i downloaded the whole piaware container and searched with my texteditor all files for faup - but did not find something that enlightened me. so this file looks much more promising!
ok - tried my best. what i found (and hopefully right understand) is that hex where only one message was found are skipped, some plausibility checks are done and the rhythm for messages transfer to flightaware depends on aircraft maneuver stability. but what validation process there could be the reason for the āmissingā hex numbers from my site i did not really figure out ā¦
what i additionally figured out is that there is lot i canāt figure out with my skills
edit: or could heavily altering positions/altitudes within the group of messages for a given aircraft/hex be the reason for being dropped?
