Stats are behind


#1

Hi everyone,

We are having some performance issues (growing pains) with the stats infrastructure, so it is behind during most of the day in US/Europe. We’re making some major improvements and hope to have those online in the next week, including new MLAT stats.

These issues do not impact data processing, MLAT redistribution, etc. It’s purely a stats issue and it catches up eventually.

Thanks for your support and patience.

Daniel


need help with mysterious feeder dropouts
#2

One of my feeders are not showing up on FA Dashboard even though its connected from PiAware 2.1.3 logs and dump-mutability-1.15~dev, FA antenna and SAW Filter. I have rebooted Raspi 2 (4.1.12) few times but no luck. The other feeders (same packages) are working just fine. I have seen this happening since upgrading to 2.1.3, atleast few times a month, right after the daily rotation at 0000 UTC but it always goes away after I reboot the Pi. All other systems parameters are normal and both dump-mutability logs and piaware logs are showing active communication to FA servers just like the other (working) Pi 2. Any suggestions ?

Update 0320Z: I switched apt repository back to raspbian/wheezy instead of flightaware and it immediately updated ntp and ntpdate packages and after rebooting, its now showing up again on FA Dashboard. No such issue on the other working RPI-2 with FA repository. Not sure it helped but I will know more by daily rotation tomorrow. The strange observation is that the logs keep showing active communication long after it stops showing up on dashboard.


#3

… as we are at the end of ‘next week’ and things today went from bad to worse - how is status quo?


#4

Next week! Seriously!


#5

cross my fingers - thanks for supersonic reply - have a nice weekend :slight_smile:
tom


#6

My stats from today went from over 17,000 positions to 2000. This happened around the same time the forums were down. I spent most of yesterday hooking up my awesome new antenna and saw the fruits of my labor, but then they reverted. Could the feeds from today be re-run?


#7

We had a glitch and are re-processing Monday, sorry about that, no data was lost.


#8

Awesome! Thanks for the super fast reply!


#9

We’ve started processing today (2015-11-10 UTC) in parallel to the reprocessing of yesterday (2015-11-09 UTC) so that the fresh data can get reflected a little more quickly. It’ll still be an hour or so before today’s data is fully reflected and caught up.


#10

Just my opinion, but slower and accurate is better than faster and wrong. I support any upgrades that improve accuracy.

My previous job involved transitioning a 24 hour ETL to real time fluming of data to a Hadoop cluster.

It too some effort but was worth it.

Keep up the good work.


#11

I would hazard the glitch is bigger than thought. I just went from 117 aircraft to 2 aircraft for 2015-11-10 UTC


#12

Stats are caught up again now. There would have been some temporary instabilities of values during the catchup period, which would explain the issues you saw last night. Thanks for your patience.


#13

We are transitioning to a new stats processing engine tonight (2015-12-18 00:00:00 UTC) to accomodate our new MLAT user stats. There is currently a few minutes of backlog as it catches back up, but that should be done in the next few minutes.

Also expect to soon see larger total aircraft and position counts, since the totals will now include MLAT derived numbers as well. In the morning, the accompanying website changes will be deployed to allow the new MLAT counts to actually be visible.


#14

Cool. Thanks for the heads up. Keep up the great work. :smiley:


#15

For feeders, stats have dual role:
(1) Help feeders monitor their system ad become aware of any trouble in their system, so they get alerted and take remedial measures.
(2) Ego satisfaction.

Once Flightaware has told us the system will be normal, lets not get carried away by (2) and worry about our ranking on charts going down.


#16

Cool! Love those graphs…


#17

My stats don’t show MLAT although under Site information MLAT configuration shows: Supported / Enabled (synchronized with 17 nearby receivers) :frowning:


#18

Hmm, I thought there had been a disturbance in the Force (sorry, Star Wars Day, USA) when I found double my normal counts at 6 a.m. EST today.

https://drive.google.com/file/d/0B5eBCoqedqaxWkR5ckIwYm9IT1k/view?usp=sharing


#19

Ego!!! Surely not! The only people this would be detrimental too are those who don’t have enough surrounding receivers to mlat with anyway!

Phill :wink:


#20

Mlat Graphs now Showing. Double with Star Wars! heheh :slight_smile: