Monitoring PIAware status?

FlightAware ADS-B Flight Tracking
1

Hey guys, so I’m trying to figure out a script that can monitor both piaware and dump1090 to see if they are running and if not reboot (drastic I know but it resets the pi as well.) Reason I want to figure this out is for true headless operation it needs to stay up and running and since right now FlightAware doesn’t seem to have a notification that our feed has gone silent (or at least I can’t find it.) I overloaded my HUSB hub yesterday and it shutdown the RTL-SDR dongle is what triggered this thought process. Since it took me 20 hours to notice.

Also in case any body is interested I’ve modified this script http://www.satsignal.eu/raspberry-pi/updatedump1090.zip from the satsignal.eu page to shutdown dump1090 and piaware and update dump1090 and restart it. I’m working on a way to monitor git for both piaware and dump1090 so that this script can just run periodically in a cron job and detect when its time to update. If you are interested let me know. All credit for that script goes to Roger Peggram, G7RUH, I’m just editing it a bit.

2

A few thoughts…

Of course, to see if the processes are running your script could contain:

ps -ef | grep piware
ps -ef | grep dump1090

With some string searching you could find PID#, etc.

However. I don’t think that solves the problem. piaware and dump1090 could be running but not reporting to FlightAware just as you stated in your example.

I just looked at my /tmp/piaware.out file. It seems to be updated every 5 mins with a status. I think you may have to monitor piaware.out as well for the solution.

Just some ideas… I could be way off base.

Watching this thread because I am interested in the solution as well.

3

OK, my RasPI stopped feeding last night. Unplugging/replugging the WiFi dongle ‘fixed’ it.
The “Anomalies” tab on my status page tipped me off as it told me that a feed had not been seen in awhile.

Perhaps your code could hit and search the anomalies report every 30 mins or so the check for issues and branch accordingly. Good news is that it does not appear that you be logged in to see your (or anyone else’s) report.

4

I was in a hurry to get it done before I went away on holiday so I went for the quick and dirty approach…

cron job to reboot at a couple of minutes after midnight my local time each day.

Ugly I know - but it does the job.

When I get back I would be interested in something more sophisticated… although I would think twice about an automated update process. Sometimes updates break working system for all sorts of reasons.

5

Here’s a quick script I use to make sure programs are always running. Cron runs it every minute, and if it finds the process is not running, then it attempts to restart it. I’m sure there’s ways to improve it - as well as other ways to do this, but this has worked for me.

This will only make sure the processes are running, doesn’t compare against your User Page Feeder data to see if the “Last Seen” is live or not. Depending on your criticality factor, you could screen scrape your User Page, and look for the word (Live), and if it’s not present then email you.

Top that off with some /tmp/piaware.out log file monitoring, and you’d have the trifecta of alerting. :slight_smile:

Anyways, onto the little script. You could modify it to add dump1090, faup1090 and piaware - and instead of $RESTARTing it, you could have it email you. Of course, modify the cron as needed, otherwise you would get emailed every minute :slight_smile:

Run: crontab -e and add


* * * * * /root/processCheck.sh >/dev/null 2>&1

Run: nano /root/processCheck.sh and paste:


#!/bin/bash
#
# This is a PID checker script to make sure the process
# in MYPROG is always running. This script
# will make sure it restarts so that the program will
# continue to run.
#

#dump1090
MYPROG="dump1090"
RESTART="/etc/init.d/dump1090.sh start"
PGREP="/usr/bin/pgrep"

# Find PID
$PGREP ${MYPROG}

# If its not running, restart
if  $? -ne 0 ]
then
   $RESTART
fi

Lastly, run: chmod +x processCheck.sh

1 Like
6

drsprite:

Thanks for posting this as it has a few million uses other that piaware.

7

No problem. I use it to monitor a few other process outside of piaware and it’s worked pretty well.

Back to piaware, you could also use piaware’s status command too,


sudo piaware-status
8

Thanks guys, that script is going to get WELL used.

9

Figured I’d post my update that monitors both piaware and dump1090. Thanks drsprite made my life a ton easier. now if I could just figure out how to get my script to watch git for changes and if present to update to work This would be all auto magic.


#!/bin/bash
#
# This is a PID checker script to make sure the process
# in MYPROG is always running. This script
# will make sure it restarts so that the program will
# continue to run.
#
#

#dump1090
MYPROG1090="dump1090"
MYPROGPIAWARE="piaware"
RESTART1090="/etc/init.d/dump1090.sh start"
RESTARTPIAWARE="/etc/init.d/piaware start"
PGREP="/usr/bin/pgrep"


# Check for dump1090
# Find PID for dump1090
$PGREP ${MYPROG1090}

# If its not running, restart
if  $? -ne 0 ]
then
   $RESTART1090
fi

#Check for piaware
$PGREP ${MYPROGPIAWARE}

# If PIAware is not running, restart
if   $? -ne 0 ]
then
	$RESTARTPIAWARE
fi
10

One way would be to have a separate cron job that:

  1. Get the current commit hash (git rev-parse --short HEAD)

  2. Update the repo (git pull)

  3. Get the new commit hash (git rev-parse --short HEAD)

  4. If the two hashes aren’t the same, rebuild/reinstall it (sudo make install, or other command(s))

  5. Restart dump1090 and piaware as necessary.

Note that if you’re using SSH access for the repo, and you put the above cron job under the root user (so that you won’t have to enter the password, and so that it will work), you won’t be able to update from the repo. You could have the root user clone the repo through something besides SSH.

11

:question:
Not to push the solution to FA, but is there a way for me to config my account to send an email if an anomaly is reported on one of my piaware feeds?
:question:

12

I just set up my piaware today so haven’t set up any monitoring yet, but monit (mmonit.com/monit/) is perfect for this. It will auto-restart processes if they are down and can monitor ports to make sure they are responding.

You can install it with apt:
sudo apt-get install monit

I’ll see if I can get it up and running for paware in the next day or two and post my configs (maybe tonight if I have time…)

13

I did some work with monit and have the basics working.

You need to make one change to the dump1090 start script, as is it won’t create the pid file that monit needs in order to monitor the process. Add the option --make-pidfile before the --pidfile option in the line that starts with “/sbin/start-stop-daemon --start” in /etc/init.d/fadump1090.sh.

Place the following in /etc/monit/conf.d/piaware after you install the monit package::


check process piaware with pidfile /var/run/piaware.pid
  start program = "/etc/init.d/piaware start"
  stop  program = "/etc/init.d/piaware stop"
  if 5 restarts with 5 cycles then timeout

check file piaware.out with path /tmp/piaware.out
  if timestamp > 10 minutes then exec "/etc/init.d/piaware restart"

check process dump1090 with pidfile /var/run/dump1090
  start program = "/etc/init.d/fadump1090.sh start"
  stop program = "/etc/init.d/fadump1090.sh stop"
  if 5 restarts with 5 cycles then timeout
  if failed host localhost port 8080 with protocol http and request "/" with timeout 5 seconds for 4 times within 5 cycles then restart
  if failed host localhost port 199 type tcp then restart

If you want to be able to monitor monit remotely you’ll need to update the listen config in /etc/monit/monitrc. You’ll find documentation under Web Service on digitalocean.com/community/ … gure-monit.

14

I modified the fadumo1090.sh and piaware files and when I check status it says that the faup1090 is not running and the piaware is not running, see below:

pi@piaware ~ $ sudo piaware-status
dump1090 is running.
faup1090 is not running.
piaware is not running.
dump1090 is listening for connections on port 30005.
dump1090 is listening for connections on port 10001.
piaware is NOT connected to port 10001.
piaware is NOT connected to FlightAware.
dump1090 is producing data on port 10001.
dump1090 is producing data on port 30005.

I modified the file and placed the new code near the end not knowing where exactly it should go as shown below:

pi@piaware /etc/init.d $ cat piaware.work
#!/bin/bash

BEGIN INIT INFO

Provides: piaware

Required-Start: $network $remote_fs $syslog $time

Required-Stop: $network $remote_fs $syslog

Default-Start: 2 3 4 5

Default-Stop: 0 1 6

Short-Description: Communicate ADS-B data to FlightAware

Description: piaware is a way to connect a homebrew Raspberry Pi-based

ADS-B receiver running dump1090 over a TLS connection to

forward ADS-B data to FlightAware

END INIT INFO

. /lib/lsb/init-functions

DAEMON=/usr/bin/piaware
PIDFILE=/var/run/piaware.pid

test -x $DAEMON || exit 5

LOCKFILE=/var/lock/piaware

start() {
log_daemon_msg “Starting piaware server” “piaware”

–quiet

/sbin/start-stop-daemon --start --oknodo --background --pidfile $PIDFILE --exec $DAEMON -- -p $PIDFILE
status=$?
log_end_msg $status
return

}

stop() {
log_daemon_msg “Stopping piaware server” “piaware”
/sbin/start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
log_end_msg $?
rm -f $PIDFILE
return
}

status() {
/usr/bin/piaware-config -status
}

case “$1” in
start)
start
;;
stop)
stop
;;
status)
status
;;
restart|force-reload)
stop && sleep 2 && start
;;
reload)
exit 3
;;
status)
status_of_proc $DAEMON “piaware server”
;;
*)

check process piaware with pidfile /var/run/piaware.pid
start program = “/etc/init.d/piaware start”
stop program = “/etc/init.d/piaware stop”
if 5 restarts with 5 cycles then timeout

check file piaware.out with path /tmp/piaware.out
if timestamp >10 minutes then exec “/etc/init.d/piaware restart”

check process dump1090 with pidfile /var/run/dump1090
start program = “etc/init.d/fadump1090.sh start”
stop program = “/etc/init.d/fadump1090.sh stop”
if 5 restarts with 5 cycles then timeout
if failed host localhost port 8080 with protocol http and request “/” with timeout 5 seconds for 4 times within 5 cycles then restart
if failed host localhost port 199 type tcp then restart

    echo "Usage: $0 {start|stop|restart|try-restart|force-reload|status}"
    exit 2
    ;;

esac

After making the changes I ran the pi@piaware ~ $ less /tmp/piaware.out command and it came back stating that no log file can be found.

Any help is greatly appreciated!

15

Hi psellati what is the purpose of your modification or what are you trying to solve?

16

My unit functions normally for about 3 days and then it locks up for no apparent reason. I am able to ping the unit both locally and on the public IP but the web services seem to stop functioning.

I was hoping to monitor the services and restart them automatically when they quit responding. In the mean time I have to physically monitor the device and when it freezes I POR the unit with a remote power switch.

I would like not having to worry about if the unit is running or not!

Thank you for your help!

17

Heat maybe causing the lockup?

Also, my RasPIs are ‘glitchy’ with the a WiFi dongle. Not just with piaware but other programs I have written that need a lot of continuous internet access.

18

Are you able to connect to the RPi using SSH & Putty? If so, can you execute “sudo shutdown -r now” which should reboot remotely w/o the POW.
It’s not automagic, but it avoids the need to get physical with the power source.

19

Very very (VERY) low tech and cheesy solution proposed here; but you could plug your RasPI into one of these for a POR.
I had to use this method on a DVR/Cable Box many years ago.

amazon.com/401A-Mechanical-M … wall+timer

20

Powering the box on and off could lead to SD card corruption. I’d definitely recommend a “cron” (scheduled job) solution rather than messing with the power. If the Pi is freezing/hanging, then there’s a heat or other serious problem. We are working on improvements to PIAware to restart dump1090 automatically if there are any problems.