System Monitoring

FlightAware ADS-B Flight Tracking
381

Here is a solution that doesn’t require an external drive:

There is a package in the Arch user repository that allows caching of anywhere in the file system using /tmp. It keeps the data in ram, and syncs to disk once per hour by default. On a Pi 2, there is plenty of ram spare to do this. Details are here: wiki.archlinux.org/index.php/An … ync-daemon

It uses systemd timers, so also works on Raspbian Jessie OK.

To install on arch, do “yaourt anything-sync-daemon” or whatever AUR tool you use.

On raspbian, you have to do it manually:


git clone https://github.com/graysky2/anything-sync-daemon.git

Then change directory to anything-sync-daemon and do:


sudo make install-systemd-all

Edit /etc/asd.conf to select the directories you want to cache. Be selective, as it uses RAM to cache. Mine is as follows:


WHATTOSYNC=('/var/lib/monitorix' '/var/lib/collectd')
VOLATILE="/tmp"
USE_OVERLAYFS="yes"

Then do:


sudo systemctl enable asd
sudo systemctl start asd

and you can check it’s status by running


asd p

which should give output similar to:


Anything-sync-daemon v5.76 on Raspbian GNU/Linux 8 (jessie)

 Systemd service is currently active.
 Systemd resync service is currently active.
 Overlayfs v23 is currently active.

Asd will manage the following per /run/asd.conf settings:

 owner/group id:     root/0
 target to manage:   /var/lib/monitorix
 sync target:        /var/lib/.monitorix-backup_asd
 tmpfs target:       /tmp/asd-root/var/lib/monitorix
 dir size:           75M
 overlayfs size:     71M
 recovery dirs:      none

 owner/group id:     root/0
 target to manage:   /var/lib/collectd
 sync target:        /var/lib/.collectd-backup_asd
 tmpfs target:       /tmp/asd-root/var/lib/collectd
 dir size:           28M
 overlayfs size:     7.8M
 recovery dirs:      none
382

I just installed this.

Thank you.

383

Thanks everyone for the suggestions , particulary caius :smiley: .
I ended up installing the ync-daemon, so far so good, disk writes are greatly reduced so this one will go on the standard build recipe.

http://i.imgur.com/z9dGXdG.png

Did quite a bit of digging around the web on the subject SD wear leveling. There is plenty of spin out there but real info is a bit hard to find so what i d d come away with is that :

1 there is no requirement for wear leveling in the official SD card specification
2 the cheap ebay cards etc probably don’t have any implemented (no surprise)
3 consumer grade cards from reputable manufactures generally have some type of leveling algorithm implemented but these tend to be more suable for the intended market i.e. File storage for cameras and holding your pirated mp3 collection in your smart phone.
4 if you want real ware leveling and error detection / correction for applications that hammer the card then the “high data integrity” industrial cards are the way to go. But at 5+ times the price I’ll probably stick with the $8 ones from kmart and a good backup :slight_smile:

384

This might be the issue. I’ve been using Rpi’s for many things as soon as the came out. With a good, brand name cards, I don’t have the issues you’re having.

385

Maybe. Ive killed 2 so far (one from ebay before i realized there was a problem) and 1 Verbatum.
The Verbatum SD from kmart for $8 are bubble packed premium 8gb class 10 in bubble pack. I can certanly pay a lot more around town but I’m not convinced ill have much more luck short of getting the industrial rated ones from RS or Element14.

386

I tried this but it does not work because systemctl is not available/installed on my Raspian Wheezy system. I found on the Internet that I may need to install


sudo apt-get install systemd systemd-sysv

but this is telling me:


WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  sysvinit
0 upgraded, 7 newly installed, 3 to remove and 1 not upgraded.
Need to get 1,580 kB of archives.
After this operation, 3,557 kB of additional disk space will be used.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?]

Naturally, since I don’t know exactly what I’m doing, I answered ‘no’ and the installation aborted. Would you know if it was safe to install/remove these packages?

387

If you are running wheezy then I wouldn’t mess with installing systemd unless you are already familiar with it, otherwise you are asking for a world of hassle getting things working. It will affect everything installed on the machine, so in most cases it’s not worth it and is easier to just start with a fresh Jessie installation (or arch if you like tinkering) rather than try to upgrade.

388

OK, thanks, makes sense! Time to start over with a fresh Jessie then :slight_smile:

389

With you there caius as far as systemd goes. Unless your going from Wheezy to Jessie doing a dist-upgrade using Jessie repositories I wouldn’t attempt it. Even then your probably better off going with a fresh installation of Jessie. Jessie is coming up on a year old and for applications such as those discussed here is more than matured enough to run them stably. I do not see why it is not used more so than Wheezy here.

390

The foundation images only started using Jessie about 3 months ago, and the piaware sdcard images are still wheezy-based (I’m working on that…) which is probably why wheezy is still mostly used here.

391

Something I noticed - If you are going to use the anything-sync-daemon it’s worth checking that your /tmp directory is actually mounted on tmpfs. The standard Raspbian Jessie image doesn’t do this by default, so you won’t see any reduced disk writes.

The /etc/fstab file should contain:


tmpfs /tmp  tmpfs defaults,noatime 0 0

Add this, and either reboot or do


sudo mount /tmp

to get it working.

To check all is OK do


df -T /tmp

You should see output with the type indicating tmpfs.

Make sure you stop anything-sync-daemon before doing this to allow it to write anything new back to its proper location, then restart afterwards.

392

Quite an improvement there. Might have to give it a try and possibly free up a WD Elements drive for use someplace else to be figured out at a later date. Although it has made for an extra place to store files on my home network where it’s at now…

Defiantly makes sense. Know your a busy person but glad to hear your working on a Jessie image.

393

Perfect timing. :slight_smile:

I was looking at my collectd graphs and trying to figure out why my I/O stats weren’t lower.

Thanks!

394

I have been curious how much data is flowing to/from my “production” piaware box. My primary feeder is used solely for Dump1090-mutability and piaware with no feeds to other sites or utilities. I have been playing with two tools: nethogs, and vnstat.

Nethogs: shows current network utilization for running processes
Installation: sudo apt-get install vnstat
Operation: sudo nethogs
Sample output:


NetHogs version 0.8.0

  PID USER     PROGRAM                                         DEV        SENT      RECEIVED
1985  root     /usr/bin/piaware                                eth0       0.756       5.240 KB/sec
29446 pi       sshd: pi@pts/0                                  eth0       0.000       0.000 KB/sec
?     root     unknown TCP                                                0.000       0.000 KB/sec

  TOTAL                                                                   0.756       5.240 KB/sec

Another tool for long term network monitoring: vnstat (tracks network statistics and saves to database)
installation: sudo apt-get install vnstat
Operation: Automatically collects stats on network in background
Sample of default output:


pi@piaware ~ $ vnstat
Database updated: Wed Jan  6 12:10:29 2016

   eth0 since 01/05/16

          rx:  426.56 MiB      tx:  654.82 MiB      total:  1.06 GiB

   monthly
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
       Jan '16    426.56 MiB |  654.82 MiB |    1.06 GiB |   18.62 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated      2.34 GiB |    3.59 GiB |    5.94 GiB |

   daily
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     yesterday    313.57 MiB |  478.69 MiB |  792.25 MiB |   75.12 kbit/s
         today    112.99 MiB |  176.13 MiB |  289.12 MiB |   54.04 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated       220 MiB |     347 MiB |     567 MiB |

Sample of hourly rates:


pi@piaware ~ $ vnstat -h
 eth0                                                                     12:10
  ^                  t
  |               t  t     t                                   t  t  t
  |   t     t  t  t  t  t  t                                t  t  t  t
  |   t  t  t  t  t  t  t  t                                t  t  t  t  t
  |   t  t  t rt rt rt  t rt  t  t                          t rt rt rt  t
  |  rt rt rt rt rt rt rt rt  t  t  t                    t rt rt rt rt  t
  |  rt rt rt rt rt rt rt rt rt  t  t                    t rt rt rt rt rt
  |  rt rt rt rt rt rt rt rt rt rt rt                   rt rt rt rt rt rt
  |  rt rt rt rt rt rt rt rt rt rt rt  t              t rt rt rt rt rt rt
  |  rt rt rt rt rt rt rt rt rt rt rt rt           t rt rt rt rt rt rt rt  t
 -+--------------------------------------------------------------------------->
  |  13 14 15 16 17 18 19 20 21 22 23 00 01 02 03 04 05 06 07 08 09 10 11 12

 h  rx (KiB)   tx (KiB)      h  rx (KiB)   tx (KiB)      h  rx (KiB)   tx (KiB)
13      17169      25675    21      12987      20597    05       3216       8438
14      17671      24686    22      11740      18995    06       9363      15977
15      18454      27364    23       9850      15916    07      16643      25209
16      19573      27380    00       3885       7233    08      20044      28596
17      19116      28175    01        752       2090    09      20257      29536
18      21781      31176    02        605       1774    10      21279      28601
19      18186      27137    03        766       2563    11      15437      22333
20      19275      28533    04       1202       4852    12       2251       3155

Other options:


pi@piaware ~ $ vnstat -?
 vnStat 1.11 by Teemu Toivola <tst at iki dot fi>

         -q,  --query          query database
         -h,  --hours          show hours
         -d,  --days           show days
         -m,  --months         show months
         -w,  --weeks          show weeks
         -t,  --top10          show top10
         -s,  --short          use short output
         -u,  --update         update database
         -i,  --iface          select interface (default: eth0)
         -?,  --help           short help
         -v,  --version        show version
         -tr, --traffic        calculate traffic
         -ru, --rateunit       swap configured rate unit
         -l,  --live           show transfer rate in real time

Cheers!
LitterBug

395

In relation to Anything-sync-daemon,

Implemented this on a new Jessie install. Also followed your advice about /tmp directory being mounted on tmpfs (http://discussions.flightaware.com/post186256.html#p186256
I specified the following directories:


WHATTOSYNC=('/var/log' '/var/lib/collectd' '/var/www/html/graphs')

And Disk I/O is practically down to zero :smiley: Thank you for the advice and instructions!

396

Thanks for the awesome info. I was looking for something like this. I’ll have to share this nice program. :smiley:

397

iptraf is another tool to consider. It is ncurses based and can be cryptic for customizing, but the default will easily show all port traffic, both on the loopback and externally. I find iptraf especially helpful to check that all my expected connections are flowing data.

Edit: typo fix.

398

Avg 0.9 write iops (and 0.2 write iops for last 6 hours):

797×213 40.7 KB

399

Hi guys,

Sorry to drag this thread out, but I’m having a problem, I am trying to generate the following graph and rrdtool is giving me a segmentation fault and no other info…

aircraft_message_rate_graph() {
rrdtool graph
“$1”
–start end-$4
–width 428
–height 200
–step “$5”
–title “$3 Message Rate / Aircraft”
–vertical-label “Messages/Aircraft/Second”
–right-axis-label “Aircraft”
–lower-limit 0
–units-exponent 0
–right-axis 10:0
“TEXTALIGN:center”
“DEF:aircrafts=$2/dump1090_aircraft-recent.rrd:total:AVERAGE”
“DEF:messages=$2/dump1090_messages-local_accepted.rrd:value:AVERAGE”
“CDEF:rate-provisional=messages,aircrafts,/”
“CDEF:rate=aircrafts,0,GT,rate-provisional,0,IF”
“CDEF:aircrafts10=aircrafts,10,/”
“VDEF:avgrate=rate,AVERAGE”
“VDEF:maxrate=rate,MAXIMUM”
“LINE1:rate#0000FF:Messages / AC”
“LINE1:avgrate#666666:Average:dashes”
“GPRINT:avgrate:%3.1lf”
“LINE1:maxrate#FF0000:Maximum:”
“GPRINT:maxrate:%3.1lf\c”
“LINE1:aircrafts10#990000:Aircraft Seen / Tracked (RHS) \c”
–watermark “Drawn: $nowlit”;
}

Has anyone else had this issue and have a solution they would care to share? rrdtool version is 1.5.4… On Ubuntu ARM… (Pi2 with Ubuntu on it) (not the radar host)…

Thanks people!

400

Hi jepolch. I am using Raspbian Stretch, when I get as far as the cd ~/pi/dump-tools there is no pi directory. Not sure if this is a difference in the Raspbian version. When I check for /collectd I get a 404 error. I created a pi directory and re ran the script but the directory is empty and still get a 404. Do you have any ideas? Sorry bit of a Luddite.
Regards
Rick