FlightAware Discussions

Dump graphs 1090 stopped working

I know I posted my question in another forum, but think this forum is a better place

Yesterday morning around 6 am my Graphs1090 stopped working. I did not change anything and I am still feeding.

After reading Wiederhopfs githup about this I ran his commands

https://pastebin.com/embed_js/vw8zXCvD

Hopefully some one can tell me what’s wrong.

cat /etc/os-release

That should enlighten the issue.

Irrelevant of the system, this is the issue you have:

https://github.com/wiedehopf/graphs1090#issues-with-some-collectd-versions

No worries you didn’t miss it, just added it.

Thank you for the reply. Did as you mentioned. NO result, used the options mentioned.
outcome:
pi@raspberrypi : ~ $ systemctl status collectd.service

● collectd.service - Statistics collection and monitoring daemon

Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor preset: enabled)

Active: activating (auto-restart) (Result: exit-code) since Sun 2020-11-01 14:03:45 GMT; 2s ago

Docs: man:collectd(1)

man:collectd.conf(5)

Process: 8206 ExecStartPre=/usr/sbin/collectd -t (code=exited, status=0/SUCCESS)

Process: 8207 ExecStart=/usr/sbin/collectd (code=exited, status=1/FAILURE)

Main PID: 8207 (code=exited, status=1/FAILURE)

Nov 01 14:03:45 raspberrypi systemd[1]: Failed to start Statistics collection and monitoring daemon.

Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “rrdtool” successfully loaded.

Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “table” successfully loaded.

Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “interface” successfully loaded.

Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “cpu” successfully loaded.

Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “aggregation” successfully loaded.

Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “match_regex” successfully loaded.

Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “df” successfully loaded.

Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “disk” successfully loaded.

Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “python” successfully loaded.

pi@raspberrypi : ~ $

pi@raspberrypi : ~ $ journalctl -xe

Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www

Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,

Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www

Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,

Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www

Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,

Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www

Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,

Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www

Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,

Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www

Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,

Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www

Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,

Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www

Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,

Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www

Nov 01 14:04:10 raspberrypi rbfeeder[659]: [2020-11-01 14:04:10] ******** Statistics updated every 30 seconds ****

Nov 01 14:04:10 raspberrypi rbfeeder[659]: [2020-11-01 14:04:10] Packets sent in the last 30 seconds: 322, Total p

Nov 01 14:04:10 raspberrypi rbfeeder[659]: [2020-11-01 14:04:10] Data sent: 20.2 MB

Nov 01 14:04:10 raspberrypi rbfeeder[659]: [2020-11-01 14:04:10] Data received: 55.4 KB

lines 1032-1052/1052 (END)

This is a fairly self-explanatory error. You’re out of filesystem space.

1 Like

Checked it

pi@raspberrypi:~ $ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 15117936 2177708 12296024 16% /
devtmpfs 469540 0 469540 0% /dev
tmpfs 474148 48 474100 1% /dev/shm
tmpfs 474148 10764 463384 3% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 474148 0 474148 0% /sys/fs/cgroup
tmpfs 51200 328 50872 1% /var/log
tmpfs 204800 12 204788 1% /tmp
/dev/mmcblk0p1 258095 54398 203697 22% /boot
tmpfs 948

Guess I do have some space left. I am using a 16Gb sd card

That’s useless, please show the same as you showed before.

sudo systemctl restart collectd
sudo journalctl --no-pager -u collectd | tail -n40

Also you didn’t show the system you are working on:

cat /etc/os-release

pi@raspberrypi : ~ $ cat /etc/os-release

PRETTY_NAME=“Raspbian GNU/Linux 10 (buster)”

NAME=“Raspbian GNU/Linux”

VERSION_ID=“10”

VERSION=“10 (buster)”

VERSION_CODENAME=buster

ID=raspbian

ID_LIKE=debian

HOME_URL=“http://www.raspbian.org/

SUPPORT_URL=“http://www.raspbian.org/RaspbianForums

BUG_REPORT_URL=“http://www.raspbian.org/RaspbianBugs

pi@raspberrypi : ~ $ sudo systemctl restart collectd

Job for collectd.service failed because the control process exited with error code.

See “systemctl status collectd.service” and “journalctl -xe” for details.

pi@raspberrypi : ~ $ sudo journalctl --no-pager -u collectd | tail -n40

Nov 01 15:26:21 raspberrypi systemd[1]: Failed to start Statistics collection and monitoring daemon.

Nov 01 15:26:31 raspberrypi systemd[1]: collectd.service: Service RestartSec=10s expired, scheduling restart.

Nov 01 15:26:31 raspberrypi systemd[1]: collectd.service: Scheduled restart job, restart counter is at 131.

Nov 01 15:26:31 raspberrypi systemd[1]: Stopped Statistics collection and monitoring daemon.

Nov 01 15:26:31 raspberrypi systemd[1]: Starting Statistics collection and monitoring daemon…

Nov 01 15:26:31 raspberrypi collectd[19044]: ERROR: ld.so: object ‘/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

Nov 01 15:26:31 raspberrypi collectd[19045]: ERROR: ld.so: object ‘/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “syslog” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “rrdtool” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “table” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “interface” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “cpu” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “aggregation” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “match_regex” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “df” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “disk” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “python” successfully loaded.

Nov 01 15:26:31 raspberrypi collectd[19045]: Traceback (most recent call last):

Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 554, in

Nov 01 15:26:31 raspberrypi collectd[19045]: main()

Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 536, in main

Nov 01 15:26:31 raspberrypi collectd[19045]: known_paths = addusersitepackages(known_paths)

Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 272, in addusersitepackages

Nov 01 15:26:31 raspberrypi collectd[19045]: user_site = getusersitepackages()

Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 247, in getusersitepackages

Nov 01 15:26:31 raspberrypi collectd[19045]: user_base = getuserbase() # this will also set USER_BASE

Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 237, in getuserbase

Nov 01 15:26:31 raspberrypi collectd[19045]: USER_BASE = get_config_var(‘userbase’)

Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/sysconfig.py”, line 587, in get_config_var

Nov 01 15:26:31 raspberrypi collectd[19045]: return get_config_vars().get(name)

Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/sysconfig.py”, line 533, in get_config_vars

Nov 01 15:26:31 raspberrypi collectd[19045]: _init_posix(_CONFIG_VARS)

Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/sysconfig.py”, line 417, in _init_posix

Nov 01 15:26:31 raspberrypi collectd[19045]: from _sysconfigdata import build_time_vars

Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/_sysconfigdata.py”, line 6, in

Nov 01 15:26:31 raspberrypi collectd[19045]: from _sysconfigdata_nd import *

Nov 01 15:26:31 raspberrypi collectd[19045]: EOFError: EOF read where object expected

Nov 01 15:26:31 raspberrypi systemd[1]: collectd.service: Main process exited, code=exited, status=1/FAILURE

Nov 01 15:26:31 raspberrypi systemd[1]: collectd.service: Failed with result ‘exit-code’.

Nov 01 15:26:31 raspberrypi systemd[1]: Failed to start Statistics collection and monitoring daemon.

pi@raspberrypi : ~ $

So you’re on Raspbian Buster … i have a similar system and don’t have the issue.
The change to the configuration was wrong anyhow, so let’s get rid of it.
Also please update the system, let’s see if that fixes it.

sudo sed -i -e 's#LD_PRELOAD=/usr/lib/python3.8.*##' /etc/default/collectd
sudo apt update
sudo apt dist-upgrade

Then recheck

sudo systemctl restart collectd
sudo journalctl --no-pager -u collectd | tail -n40

Putting the logs on https://pastebin.com/ or surrounding it with ``` would be helpful for readability:
```
log
```

Looking at your first log and the errors about insufficient disk space, i’d say you’re sd-card is dying.
Can’t help you.

I had a somewhat similar tar1090 and total Pi fail about a month ago and it was caused by 100% Inodes use instead of disk space use. (I had broken my lighttpd.conf configs)

While “df -h” showed lots of space, the problem was actually running out of Inodes revealed with “df -i” which showed 100% of Inodes consumed.

The methods to find the problem might help someone else some day…

df -i
df -i originally showed 100% of Inodes in use. Below image is Inodes free after I had deleted 55% of the files. (It took maybe an hour to delete 100% of those million+ files off the sdcard)
chrome_eNt0t6YstK

If your Inodes use is high, use the command below to find out where all the files are. Look at the bottom rows of the output of this:

sudo find / -xdev -printf ‘%h\n’ | sort | uniq -c | sort -k 1 -n

vncviewer_iqGmaRbP2o

Then figure out the source of what you broke to cause the problem. In my case messing with lighttpd.conf files since PiHole abuses those configs on my test station. Normally I hack through and fix lighttpd.conf files after reinstalling/updating PiHole or after doing a fresh Pi installation with all the ADS-B tracking software. I failed to do that correctly the last time. Everything worked, but I eventually kept running out of free Inodes a few weeks later as files didn’t get cleaned up. The hard part was finding the right command to list where all the files/Inodes were.

Inodes still happy a month later
vncviewer_Jq1iZNTQDw

1 Like

Thanks :+1: :+1: :+1:

Hmm, i might disable the compress directory in lighttpd.conf when installing tar1090 …
It’s not all installs but enough where this is an issue.

Oh my … just checked after recently purging / reinstalling lighttpd on my RPi, the cache is now default:

compress.cache-dir          = "/var/cache/lighttpd/compress/"                                                                          
compress.filetype           = ( "application/javascript", "text/css", "text/html", "text/plain" )

Now with these filetypes it shouldn’t cache the dynamic tar1090 / dump1090-fa json and gz files.
But making this a default without having automatic cache cleaning built in … whoever made that lighttpd config a default is crazy.
If you make such a thing a default, you’d need automatic cleaning of old files at the very least.

2 Likes

FWIW, compress/cache is still enabled in my .conf but I am no longer building up unwanted files. So fixing whatever I broke originally seems to avoid the compounding problem. (I have lighttpd.conf PTSD, I’m afraid to edit again!)

Hmm … i suppose i’ll only mess with it automatically when cache-dir and filetype json is enabled and place a warning in lighttpd.conf otherwise.

I’m sure that will help people who break stuff.

Running
sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
on both my station shows everything is cleaning up nicely on its own.

I’d say it’s rather that lighttpd isn’t creating a new file every second you view /tar1090.

1 Like

Ahh, now I get it! (20 characters)