Identifying aircraft 250+ miles out


#1

My stats page radar usually shows 1 or 2 positions per day supposedly received from 250+ miles out. I’d like to be able to see the details for those so I can establish if I’m seeing a real aircraft via an atmospheric effect or whether it’s just bad data. If the former I’d love to be able to see where on the map where I picked them up from.

I have a couple of Skyview screens open, one of which is zoomed out and reverse-sorted by distance, but it’s not ideal since I will have to happen to be looking at it during the 58 seconds that the position is in the table. Is there any way I can see the position details after the fact? For example can I run something on another computer which is logging data over the local network from PiAware? I’m running stock PiAware 3.5.3.

There is a crude hack which could be used. I could leave Skyview on display with the table reverse-sorted by distance, and then use Quicktime to constantly screen-record that top portion of the table. If I see a 250+ miler I can then review the recording and try to find it. I’d like to find a better method that isn’t so painful.

I’ve scoured the forums and web but can’t find anyone asking about recording over the network or enabling some logging solution. Anyone got any suggestions on how to track down these elusive 250+ aircraft? Thanks.


#2

I second this request, but that would require FA to save your historical feed data, which is not done currently. I asked a couple of weeks ago if the feed data was available, since I wanted to kind of keep an online log of planes seen by my receiver.


#3

I believe though that it’s possible to access PiAware’s data over your local network and use that to support other feeders running on other computers. I’m wondering if there’s a suitable program which can access PiAware’s data in that manner and which does support logging. Even if it can log an ICAO, distance and timestamp, that’s enough to be able to find out where it was on the map at that time and confirm that it’s valid.


#4

The scripts from here allow for logging. They are by default disabled because of the possible SD card wear (lots of writes).


#5

Thanks, I noticed that build, I’ll be looking at that anyway later on as I want to feed FR24 in a few months time. For now I’m sticking with stock PiAware and hoping I can do something with its local network features.


#6

There are some scripts by user tommuc which store the dump1090 data daily to excel or to mysql. I use them to identifying interesting flight types in the neighborhood.


#7

@chrislfa
@Dxista
@SoNic67
@georgforster

Have a look at following two threads created by @TomMuc last year. I am not sure, but may be this is what you need:
.

dump1090-mysql-database-script-with-alert-and-filter

.

.

dump1090-mysql-database-script-e-mail-alert-lat-lon-alt-hex-flight-filter


#8

@georgforster @abcd567 Thanks, they look familiar but would need me to bin PiAware which I’m sticking with for now. Handy to know for the future though. I’m going to look at Plane Finder to see if I can log data from PiAware.


#9

Tried that, no go, as far as I could see. Tried other sites as well, same thing. Using VRS Reports for now.

Planeplotter may be an option, but I cannot convince myself to like using it.


#10

Oh bugger, thanks, you’ve saved me some time. It’s going to be simpler to have Skyview running on a spare laptop and use a car dashcam to record the screen 24/7 so I don’t miss it!


#11

@chrislfa
@Dxista
@SoNic67
@georgforster

This may not be what you want, but is somewhat related, and is interesting:

Exploring Port 30003 and 30106

.


#12

Thanks @abcd567 actually I think it’s exactly what I’m looking for! I can use that to log to a csv continuously. Periodically I can check the radarscope for a 250+ miler. If there is none then I can quit the logging and restart it to abandon the data collected and start afresh. If there is one then I can pull that data into a spreadsheet and sort it by columns O or P which will immediately show positional outliers. I can then look up the data and timestamp from the other columns to confirm the aircraft’s position.

If this works I will post a followup which explains how to do it step by step. For those with some advanced grep/sed/awk knowledge it should be easy to enter a single command to filter the logs to only show those aircraft outside of a 250 mile width square box centered on your location, and that would avoid any need for a spreadsheet at all.

For reference on the Mac, logging to a file called log.csv on the user’s desktop, with PiAware on the 192.168 address shown:

$ telnet 192.168.0.100 30003 > ~/Desktop/log.csv

Thanks again.

[UPDATE: running this for 10 minutes here now gave me 14,000+ lines. If it was logging for an hour or more the file would need some processing in order to make it small enough to be imported into a spreadsheet. For example splitting it into multiple files. So it’s not quite a solution but the data is all there, just a case of filtering it on the fly or post-processing it as mentioned above. I’ll post instructions once I’ve figured it out.]


#13

Based on these logs I’ve worked out a way to visualise the positions received. Since then I had a 250+ mile entry appear on the coverage graph but when I checked the map there was no sign of it, so, in that case at least, I suspect it was bad data. Here’s how to do it.

Start logging the data as per the earlier posts in this thread. If you can pre-filter to log only those entries with lat/lon then even better. I’m on a Mac and I’m using the quick and dirty command below (replace the x’s with your PiAware’s IP). The grep command tells it to only log lines that have a comma followed by two digits, a dot and another digit and ignore lines that don’t have that. In and around the UK that sequence matches lines that contain a latitude co-ordinate, so only lines with lat/lon get logged. It’s a crude filter but it works. Logging is taking place to a file called log.csv on my desktop.

$ telnet 192.168.x.x 30003 | grep '\,[0-9][0-9]\.[0-9]' > Desktop/log.csv

You dont have to pre-filter the logging, it just makes things much easier in the spreadsheet later on. If you’re using Windows just log everything using telnet or Putty (see @abcd567 link earlier) or else someone else may be able to suggest a way to achieve the same pre-filtering on Windows.

Either way, the result is that you end up with a file of comma-separated logs, all or some of which have latitude and longitude entries. In my case they all have lat/lon entries because I pre-filtered with that grep command. I left this running for an hour and ended up with 63,000 lines in the log, each one being a message received with a lat/lon entry.

Next, open that file in Excel or Libreoffice and import it as a comma-separated file. Columns O and P will contain the latitude and longitude respectively. If you pre-filtered the log then every row has an entry. Select all the entries in both columns and copy them to the clipboard.

If you didn’t pre-filter then you’ll have many more lines without lat/lon and you’ll need to use the ‘sort’ feature in your spreadsheet to get all the lat/lon lines together, then copy to the clipboard as above.

Finally go to this very useful site

http://www.hamstermap.com/

and click Quick MAP. In the lower left select the example entries and delete them and paste in the entries you copied. Then hit the Regenerate button and all the positions will be displayed. You can use this to scroll around and see if you can locate any distant positions which could be caused by an atmospheric effect.

(This coverage is from my mini mag mount on a coffee tin)

If I did notice a distant position of interest I would establish its approximate lat/lon from the tool here

and I would then search the orginal spreadsheet for the entry in question. That would give the date and time and ICAO so I could use FlightAware to positively identify and confirm the aircraft’s presence in that location at that time.

Suggestions for future improvement to PiAware

  • Add the ability to easily log data continuously to either an attached hard drive or a network share.
  • Add the ability to load a log file and visualise it on the Skyview map, being able to toggle details when zoomed in on individual entries.
  • Add the ability to use this data to build comprehensive coverage maps. This wil allow individuals to start a new log when they make a site change, such as adding a preamp or changing their antenna, and visualising the difference it makes over a period of time.

#14

Note that that many writes on an SD card will quickly wear it out. Filtration of data needs to happen before writing the file.


#15

@SoNic67
For the adventuresome: with a little work, one can configure their SD card to use a filesystem made for SD cards (f2fs). A summary is posted here Raspberry Pi, SD Cards and there are several writeups elsewhere. Not without risk, but if one is careful, it becomes ‘normal’. I routinely convert to F2FS for use in my android phones’ SD cards, and have done so for all the Pi units I have in use. From the looks of things, people have a wide variety of experiences of running SD cards for years without issues, others sandpaper them to death every few months. I believe the “right” answer is: it depends on the quality of the cards and how they are used; meaning, there is no one ‘right’ answer… :roll_eyes: Gather the experiences of others, filter them through one’s fact-check sensibilities, and make a choice - freedom to experiment and succeed or fail. Good luck!


#16

I believe so, but note that in the above there is no change to the way the card is being used since that’s not where the logging was taking place. I was capturing the data from the Pi over the network and logging it on a Mac, so pre-filtering is merely a convenience in the spreadsheet, nothing more.


#17

There was a ticket opened by someone with 1GB/day of access logs (located in /var/log/ in the auth.log file) . If your PiAware is not behind your router’s firewall it will probably be hit by many access attempts per day. This can fill up your SD card and cause your SD card to wear out quicker.

You can check the size of all your log files in /var/log and see if the daily files are big. Usually the log files will start a new log file each day and remove the oldest one from the SD card. If your daily log files are showing multiple GB/day you can disable logging (not recommended). Usually log files should be in the few MB/day range.

ls -la /var/log

The average site sends about just under 1000MB/month to FlightAware. The average site generates about 10GB/month of Flight data (Not stored but this is approximately how much space you need If you want to store every message locally).
The average logs files are usually around 1MB/day.

Log file size is just another thing to check if you are having SD card wear problems.


#18

I feed the output of Piaware into ADSBscope and set the tails to last as long as possible (10 hours)

It ends up with a mess of aircraft and tracks but every now and then I see an aircraft a long way from the receiver.

Here is one

The aircraft is QF719 Canberra to Perth so the position is plausible.

The distance is about 800NM from here.

I noticed it well after the fact and I was not astute enough to work out where the plane should have been so my confidence level is fairly low. It is the right place but not sure it is the right time.

I am also aware that I or others may be feeding incorrect data for a plane location into Flightaware and all the other crowd fed reporting services makes me nervous about checking the actual location by some other service. I may simply be using my own data to confirm my own data. This would be bad.

Some day I hope to capture one in real time (or near real time) so I can actually work out where the plane should be just by dead reckoning.

S.


#19

Many thanks, I’ll take a look at that software because it’s essentially doing what I was doing with the spreadsheet and plotter, only automatically without hassle. I’ve also upgraded to an external antenna (still indoors) and I’m seeing around a 75% increase in positions and a 35% increase in aircraft, and more of those 250+ mile aircraft reports. So I’d still like to try and track them down.

Is this the software?

http://www.sprut.de/electronic/pic/projekte/adsb/adsb_en.html