I know I posted my question in another forum, but think this forum is a better place
Yesterday morning around 6 am my Graphs1090 stopped working. I did not change anything and I am still feeding.
After reading Wiederhopfs githup about this I ran his commands
https://pastebin.com/embed_js/vw8zXCvD
Hopefully some one can tell me what’s wrong.
cat /etc/os-release
That should enlighten the issue.
Irrelevant of the system, this is the issue you have:
No worries you didn’t miss it, just added it.
Thank you for the reply. Did as you mentioned. NO result, used the options mentioned.
outcome:
pi@raspberrypi : ~ $ systemctl status collectd.service
● collectd.service - Statistics collection and monitoring daemon
Loaded: loaded (/lib/systemd/system/collectd.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2020-11-01 14:03:45 GMT; 2s ago
Docs: man:collectd(1)
man:collectd.conf(5)
Process: 8206 ExecStartPre=/usr/sbin/collectd -t (code=exited, status=0/SUCCESS)
Process: 8207 ExecStart=/usr/sbin/collectd (code=exited, status=1/FAILURE)
Main PID: 8207 (code=exited, status=1/FAILURE)
Nov 01 14:03:45 raspberrypi systemd[1]: Failed to start Statistics collection and monitoring daemon.
Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “rrdtool” successfully loaded.
Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “table” successfully loaded.
Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “interface” successfully loaded.
Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “cpu” successfully loaded.
Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “aggregation” successfully loaded.
Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “match_regex” successfully loaded.
Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “df” successfully loaded.
Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “disk” successfully loaded.
Nov 01 14:03:45 raspberrypi collectd[8207]: plugin_load: plugin “python” successfully loaded.
pi@raspberrypi : ~ $
pi@raspberrypi : ~ $ journalctl -xe
Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www
Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,
Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www
Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,
Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www
Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,
Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www
Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,
Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www
Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,
Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www
Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,
Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www
Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,
Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www
Nov 01 14:04:07 raspberrypi rsyslogd[321]: action ‘action-2-builtin:omfile’ (module ‘builtin:omfile’) message lost,
Nov 01 14:04:07 raspberrypi rsyslogd[321]: file ‘7’ write error: No space left on device [v8.1901.0 try https://www
Nov 01 14:04:10 raspberrypi rbfeeder[659]: [2020-11-01 14:04:10] ******** Statistics updated every 30 seconds ****
Nov 01 14:04:10 raspberrypi rbfeeder[659]: [2020-11-01 14:04:10] Packets sent in the last 30 seconds: 322, Total p
Nov 01 14:04:10 raspberrypi rbfeeder[659]: [2020-11-01 14:04:10] Data sent: 20.2 MB
Nov 01 14:04:10 raspberrypi rbfeeder[659]: [2020-11-01 14:04:10] Data received: 55.4 KB
lines 1032-1052/1052 (END)
This is a fairly self-explanatory error. You’re out of filesystem space.
Checked it
pi@raspberrypi:~ $ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 15117936 2177708 12296024 16% /
devtmpfs 469540 0 469540 0% /dev
tmpfs 474148 48 474100 1% /dev/shm
tmpfs 474148 10764 463384 3% /run
tmpfs 5120 4 5116 1% /run/lock
tmpfs 474148 0 474148 0% /sys/fs/cgroup
tmpfs 51200 328 50872 1% /var/log
tmpfs 204800 12 204788 1% /tmp
/dev/mmcblk0p1 258095 54398 203697 22% /boot
tmpfs 948
Guess I do have some space left. I am using a 16Gb sd card
That’s useless, please show the same as you showed before.
sudo systemctl restart collectd
sudo journalctl --no-pager -u collectd | tail -n40
Also you didn’t show the system you are working on:
cat /etc/os-release
pi@raspberrypi : ~ $ cat /etc/os-release
PRETTY_NAME=“Raspbian GNU/Linux 10 (buster)”
NAME=“Raspbian GNU/Linux”
VERSION_ID=“10”
VERSION=“10 (buster)”
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL=“http://www.raspbian.org/”
SUPPORT_URL=“RaspbianForums - Raspbian”
BUG_REPORT_URL=“RaspbianBugs - Raspbian”
pi@raspberrypi : ~ $ sudo systemctl restart collectd
Job for collectd.service failed because the control process exited with error code.
See “systemctl status collectd.service” and “journalctl -xe” for details.
pi@raspberrypi : ~ $ sudo journalctl --no-pager -u collectd | tail -n40
Nov 01 15:26:21 raspberrypi systemd[1]: Failed to start Statistics collection and monitoring daemon.
Nov 01 15:26:31 raspberrypi systemd[1]: collectd.service: Service RestartSec=10s expired, scheduling restart.
Nov 01 15:26:31 raspberrypi systemd[1]: collectd.service: Scheduled restart job, restart counter is at 131.
Nov 01 15:26:31 raspberrypi systemd[1]: Stopped Statistics collection and monitoring daemon.
Nov 01 15:26:31 raspberrypi systemd[1]: Starting Statistics collection and monitoring daemon…
Nov 01 15:26:31 raspberrypi collectd[19044]: ERROR: ld.so: object ‘/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Nov 01 15:26:31 raspberrypi collectd[19045]: ERROR: ld.so: object ‘/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.so’ from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “syslog” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “rrdtool” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “table” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “interface” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “cpu” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “aggregation” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “match_regex” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “df” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “disk” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: plugin_load: plugin “python” successfully loaded.
Nov 01 15:26:31 raspberrypi collectd[19045]: Traceback (most recent call last):
Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 554, in
Nov 01 15:26:31 raspberrypi collectd[19045]: main()
Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 536, in main
Nov 01 15:26:31 raspberrypi collectd[19045]: known_paths = addusersitepackages(known_paths)
Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 272, in addusersitepackages
Nov 01 15:26:31 raspberrypi collectd[19045]: user_site = getusersitepackages()
Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 247, in getusersitepackages
Nov 01 15:26:31 raspberrypi collectd[19045]: user_base = getuserbase() # this will also set USER_BASE
Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/site.py”, line 237, in getuserbase
Nov 01 15:26:31 raspberrypi collectd[19045]: USER_BASE = get_config_var(‘userbase’)
Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/sysconfig.py”, line 587, in get_config_var
Nov 01 15:26:31 raspberrypi collectd[19045]: return get_config_vars().get(name)
Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/sysconfig.py”, line 533, in get_config_vars
Nov 01 15:26:31 raspberrypi collectd[19045]: _init_posix(_CONFIG_VARS)
Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/sysconfig.py”, line 417, in _init_posix
Nov 01 15:26:31 raspberrypi collectd[19045]: from _sysconfigdata import build_time_vars
Nov 01 15:26:31 raspberrypi collectd[19045]: File “/usr/lib/python2.7/_sysconfigdata.py”, line 6, in
Nov 01 15:26:31 raspberrypi collectd[19045]: from _sysconfigdata_nd import *
Nov 01 15:26:31 raspberrypi collectd[19045]: EOFError: EOF read where object expected
Nov 01 15:26:31 raspberrypi systemd[1]: collectd.service: Main process exited, code=exited, status=1/FAILURE
Nov 01 15:26:31 raspberrypi systemd[1]: collectd.service: Failed with result ‘exit-code’.
Nov 01 15:26:31 raspberrypi systemd[1]: Failed to start Statistics collection and monitoring daemon.
pi@raspberrypi : ~ $
So you’re on Raspbian Buster … i have a similar system and don’t have the issue.
The change to the configuration was wrong anyhow, so let’s get rid of it.
Also please update the system, let’s see if that fixes it.
sudo sed -i -e 's#LD_PRELOAD=/usr/lib/python3.8.*##' /etc/default/collectd
sudo apt update
sudo apt dist-upgrade
Then recheck
sudo systemctl restart collectd
sudo journalctl --no-pager -u collectd | tail -n40
Putting the logs on https://pastebin.com/ or surrounding it with ``` would be helpful for readability:
```
log
```
Looking at your first log and the errors about insufficient disk space, i’d say you’re sd-card is dying.
Can’t help you.
I had a somewhat similar tar1090 and total Pi fail about a month ago and it was caused by 100% Inodes use instead of disk space use. (I had broken my lighttpd.conf configs)
While “df -h” showed lots of space, the problem was actually running out of Inodes revealed with “df -i” which showed 100% of Inodes consumed.
The methods to find the problem might help someone else some day…
df -i
df -i originally showed 100% of Inodes in use. Below image is Inodes free after I had deleted 55% of the files. (It took maybe an hour to delete 100% of those million+ files off the sdcard)
If your Inodes use is high, use the command below to find out where all the files are. Look at the bottom rows of the output of this:
sudo find / -xdev -printf ‘%h\n’ | sort | uniq -c | sort -k 1 -n
Then figure out the source of what you broke to cause the problem. In my case messing with lighttpd.conf files since PiHole abuses those configs on my test station. Normally I hack through and fix lighttpd.conf files after reinstalling/updating PiHole or after doing a fresh Pi installation with all the ADS-B tracking software. I failed to do that correctly the last time. Everything worked, but I eventually kept running out of free Inodes a few weeks later as files didn’t get cleaned up. The hard part was finding the right command to list where all the files/Inodes were.
Inodes still happy a month later
Thanks 
Hmm, i might disable the compress directory in lighttpd.conf when installing tar1090 …
It’s not all installs but enough where this is an issue.
Oh my … just checked after recently purging / reinstalling lighttpd on my RPi, the cache is now default:
compress.cache-dir = "/var/cache/lighttpd/compress/"
compress.filetype = ( "application/javascript", "text/css", "text/html", "text/plain" )
Now with these filetypes it shouldn’t cache the dynamic tar1090 / dump1090-fa json and gz files.
But making this a default without having automatic cache cleaning built in … whoever made that lighttpd config a default is crazy.
If you make such a thing a default, you’d need automatic cleaning of old files at the very least.
FWIW, compress/cache is still enabled in my .conf but I am no longer building up unwanted files. So fixing whatever I broke originally seems to avoid the compounding problem. (I have lighttpd.conf PTSD, I’m afraid to edit again!)
Hmm … i suppose i’ll only mess with it automatically when cache-dir and filetype json is enabled and place a warning in lighttpd.conf otherwise.
I’m sure that will help people who break stuff.
Running
sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
on both my station shows everything is cleaning up nicely on its own.
I’d say it’s rather that lighttpd isn’t creating a new file every second you view /tar1090.
Ahh, now I get it! (20 characters)
Please post in an appropriate thread or open your own maybe? 
Your graphs are obviously still working so this post was kinda confusing to read.
Added a readme section for blank range graph:
The gain seems way too high so i’d recommend installing this:
@davidinjp & @wiedehopf: I ran into this issue over the last few days and found that it was indeed /var/cache/lighttpd/compress/data that had well over 1 million files which ate up the inodes. I was able to clean up enough to get things running, but I’m not very familiar with lighttpd configuration and given David’s comments I’d prefer not to mangle my system. Can either of you suggest either how to disable the caching (which doesn’t seem to be needed) or at least automatically cleaning up?
