running scripts against FlightAware & being logged in


#1

Hello,
I use a UNIX program called lynx to grab text from web pages and use other UNIX commands (like sed, awk, grep, etc.) to get the info I want. I have registered and I see 50 flights per page. But when I run lynx on a page, I am not logged in so I only see 20 entries per page. Is there a way I can add something into the URL I use in the lynx command to let the web page know that I am registered and can view 50 entries per page? No big deal really, I just loop through however many times I need to get all the information but would be a little nicer. Thanks.

Jeff


#2

Hi, Jeff. I appreciate your enthusiasm, but unfortunately, what you’re describing is actually a violation of FlightAware’s terms of service. FlightAware provides this service for free in exchange for users viewing advertisements that fund the company. Your “web scraping” program not only results in FlightAware paying for equipment and bandwidth to serve your requests without the accompanying advertisements, but making lots of requests in rapid succession results in the program being even more intensive than a traditional user that is viewing the advertisements.

I would recommend using a web browser to access the site through traditional means and posting any ideas you have to the forums and perhaps they’ll be offered in the future as a service.

Thanks for your support!