The changes helped with processing speed but didn’t entirely solve the problem so stats will still be somewhat behind at peak times (and then catch up after the peak passes, as you saw). The next step is to throw more hardware at it.
Mine shows the ‘bars’ for the last three hours as well now. I’m glad the data is being received at least.
Tough problem to solve. I suppose demand on servers increases with the square of the number of sites or something like that. Adding more servers, though, will also increase traffic among the servers, right? Anyway, it seems like performance has dropped since the change. Are configurations still being tweaked?
Edit: fixed spelling error
We made some software configuration tweaks. We’re planning to provision additional hardware today which will give us many months of additional growth capacity.
I take it there is a massive outage today? Mine was reading quite a few 1000 reports now I have 36. The “Hourly Collection Graphs (BST)” was at 8000 or there abouts until 10am then started dropping off until its now at near zero.
Users here do not have a way of checking Flightaware status of servers/systems. Perhaps a Status page would be a good idea?
See twitter.com/FlightAwareOps for outage info. We’re just recovering from a big DB outage and stats are some way down the list of stuff to fix, it might be a few hours before they start updating again.
TBH that twitter account is worse than useless…
Example; before today there has been no activity/posts on https://twitter.com/FlightAwareOps since March 31, yet I have had MULTIPLE hours of lost reports, random graph spikes and graph zeros between March 31 and now with nothing said by FlightAwareOps.
Because theres no word from FlightAware on the official twitter feed I’m left guessing if its my antenna, Pi, LAN, WAN etc but then find it IS a FlightAware issue by checking nearby sites …or hours later someone says something the forums.
The ops feed generally posts only about more widespread outages (like the DB outage we were talking about above).
(/me goes back to fighting fires)
What happened yesterday ?
Everything wnt normal as usual, in the afternoon I took my RasPi off only for some minutes in the afternoon to work on the case, reconnected/restarted and everything seemed well. Today I looked at my stats, and I was wondering: the stats shows only a very few a/c and messages ony the hourly reports looks as usual.
For today everything OK again …
Can any body of the staff explain that ? Never seen that before …
Database issues apparently. it affected a number of user, but not all. Everything is not back to normal as your average stats are now wrong. I checked mine throughout the day and it was up and down constantly. about 23:00-23:30 last night it reverted from being correct at approx 190000 reports and 900 Aircraft to 37/2 after I refreshed the screen.
Looks like the import of yesterdays data may not be imported since FA were having so many issues yesterday.
you can check for outages here: twitter.com/FlightAwareOps but not explanation on why and if any issues are still occurring. it says normal service back to normal, but that is not the case, with so many users missing data yesterdays data.
Stats for the 3rd/4th have some known problems; but some unrelated database problems need to be solved first, then we’ll look at stats.
The underlying data is all there and nothing got lost, it’s just that stats are wrong or slow.
OK, thanks, database problems …
I will bookmark the twitter-link.
See now how is wrong thinking this is exactly the kind of information that should be broadcast via https://twitter.com/FlightAwareOps? Not only that there is an issue with “X” but whats happening with the issue.
Why can’t it be if that is where we are supposed to look for outage/anomaly information?
Sounds pretty widespread…
You answered earlier that only “widespread” issues were announced on twitter. Could you please explain what “widespread” is? Is it based on total number of feeders affected? Number within a certain radius? Actually only “important” locations/countries?
I remember taking the time to fill out a survey for FlightAware some months back and I stated then that the lack of clear communication regarding outages was an issue.
Isn’t the amount of threads started and comments by feeders trying to find out what is going on, every time an issue occurs, clear evidence that the communication about these issues needs to be addressed?
Another option would be a message on user stats. pages. Similar to when there is an MLAT issue with their data feed.
While you may not find it ideal, the simple answer is that the FlightAwareOps twitter account is run by our Operations department that is responsible for back-end infrastructure of all of our systems. We are looking into better ways to communicate information in situations such as this to the ADS-B community, such as via the stats page itself as you mentioned.
There are a lot of things going on in recent days. To summarize:
- PiAware 3.5 released (it is not related to stats display issue).
- Deployment of additional hardware for ADS-B Stats and required software reconfiguration to support it (this is where the hiccup occurred for some users this week). We are testing a fix for this with plans to deploy it later today. That will address the ongoing stats inconsistencies affecting some users.
- Unrelated to ADS-B Stats specifically, the Ops team is addressing an issue with the database infrastructure which Stats uses along with many other FlightAware services.
We are looking into restoring the missing stats from this week for those affected users. It will probably take several days for us to make that happen but we will do our best to avoid the appearance of any gaps for our users. Our first priority is making sure the go-forward stats are working for all users.
Thank you for your detailed reply! That is awesome.
I guess it’s just frustrating as a feeder to see an anomaly via the stats. page which could be caused by many things locally, check https://twitter.com/FlightAwareOps, which 9 times out of 10 has nothing recent tweeted, then have to look around nearby sites to see if others are affected, to finally end up searching the forums and see others reporting similar issues with a reply from an Admin acknowledging and explaining the issue pointing people back to https://twitter.com/FlightAwareOps adding that only “widespread” (which remains undefined) issues are announced there.
The Admin got the information from somewhere to post it in 2 or 3 threads on the subject, why not just “tweet” it and be done. Even if it needs to be a new twitter account for now.
I look forward to the implemented solution whatever it is.
Perhaps just creating an “Announcement” post in this forum would suffice.
Hmmm - interesting. I was not really affected by those glitches (just a few one hour gaps here and there - was curious about them, but not too much) untill Ops started to work on the issue. With the fix - almost 24 hours of my data on Monday-Tuesday is suddenly gone. It was there earlier today, but now no more. Interesting fixes there…
Also - question, as i have not yet seen it anywhere. You say “affected users” - how do you determine those? Do we need to raise a ticket or something? Thanks!
My stats dropped after an almost daylong outage on Wednesday April 5th. Stats after that were half of what I was previously seeing with no explanation other than the outdoor antenna, dongle or cable is bad, all new with in the last year. I have a solid Comcast high speed connection too. I am also seeing a corresponding reduction in range (distance) of aircraft seen. Very weird…
It does sound like a hardware problem to me if you are seeing reduced range in the data you view directly.
nb: stats for the 5th on the FA side are currently being regenerated.