FlightAware Discussions

Graphs1090 - reducing write to SD

Hi!
I’m using feature of reducing writes to the sd-card

sudo bash /usr/share/graphs1090/git/malarky.sh

The issue - last weeks I faced with few power offs (energy company side), and my graphs are sometimes empty. I would like to now - how its possible to change data writes frequency from every 24h to every x hours or smth like that.

Also, does this feature reducing only frequency of data writes, or data size too?
Maybe someone made some measurements when malarky.sh enabled/disabled?

TY

Check the readme, i’ve added something about it.

It’s quite a difference … but i don’t remember anymore you’ll have to check yourself.

collectd will write to disk every minute normally, so overall bytes written and write frequency are seriously reduced.

Even changing it to a full write every 2 hours is probably better than disabling it.
Be sure you update before you change the file to make sure it’s also doing the gzip before writing to disk.
Each saving to disk with malarky is less than 10 MB, i’d say with malarky disable you’d get 10 MB an hour easily but again you’d have to test.

1 Like

Thank you so much for your efforts!
I’ll check it some later

It’s about 14-15kB/s with the caching turned off. That’s about 50MB an hour or 1.1GB a day of writes, so significant if you are using a small SD card. If you also are using a cheap SD card, then that could result in quite a shortened expected life.

I was running a Raspberry 3B for > 1 year with graphs1090 installed. And the SD card is still working without a single issue.

A second Pi 4 is still running graphs1090 for 6 months now. In addition it runs WeeWX (a weather software) with lots of writes to the card.

Friend of mine is using meanwhile his third or fourth SD card.

So it really depends on the card itself
I think it’s not only about the amount of data written, but also the frequency.
Might be better to write 1 GB every day instead of 50 MB per hour.

That’s the PI 4, started Graphs1090 later

image

You can just run the graphs1090 install again and it will automatically enabe the write reducing measures.

That can be reduced to 10 MB an hour by setting the filesystem dirty_expire to 10 minutes:
GitHub - wiedehopf/graphs1090: Graphs for dump1090 (based on dump1090-tools by mutability)

But yeah with the other system i added the IO used is much less, roughly 10 MB a day with the chance of losing a day of data of course when the power fails unexpectedly.

1 Like

Very good, thanks. Done a minute ago

Even though you make it obvious in the install/update script message, I missed the memo on the write reducing on by default update but finally figured it out weeks ago after pondering gaps in my graphs. I settled on 3 minute writes (CM4 eMMC instead of SD card).

Also recently used your guide for moving data history between devices several times. It works perfectly every time.

I mean … how often does your power fail? :slight_smile:
Normally there shouldn’t be any gaps.

Issues were when I was changing my station hardware and then troubleshooting. I did shutdown or reboot before powering down sometimes, but not always. That’s when I noticed the writes feature. You’re right though, before that I probably went 6+ months without power disruption.

Any idea about the CPU load gaps in the graphs while the other graphs do not show these?

Aggregation over multiple cores … the module doing the aggregation is a bit touchy.
Do those correspond to restarts or something?

Anyhow i don’t care enough to fix it.

Thanks for quick response.

No restarts the device is running since > 30 days now
It’s only the CPU graph, the others are perfectly fine (see below).

Agreed, as long as it’s working in general.

Last 24 hours:

Here’s mine with a whole bunch of power and airspy_adsb restarts over the past 7 days…

I’m going to blame this comment for my power going out 90 minutes ago. It’s still out due to wind storm. After 45 minutes downtime, I decided to start ADSB feeding again with backup generator and mobile phone hotspot serving internet to my home router!

1 Like

I use another approach by setting the commit time at the partition level (in /etc/fstab).
This way :

PARTUUID=6c819e43-02  /     ext4    defaults,noatime,commit=600  0   1

My understanding is that dirty_expire works globally, while commit works on a specific mountpoint only (which does not make much difference for most raspberry users).

From ext4 documentation:

            Ext4 can be told to sync all its data and metadata
			every 'nrsec' seconds. The default value is 5 seconds.
			This means that if you lose your power, you will lose
			as much as the latest 5 seconds of work (your
			filesystem will not be damaged though, thanks to the
			journaling).  This default value (or any low value)
			will hurt performance, but it's good for data-safety.
			Setting it to 0 will have the same effect as leaving
			it at the default (5 seconds).

This + log2ram makes me pretty confident the sd-card is not stressed much.

Maybe more related to the question how to remove your mountains? Just asking… :wink:

I’ve set it up based as suggested by wiedehopf. I do not see a significant change, but that’s not the fault of the graphs1090 script.
This little traffic is hidden behind the larger traffic by the weather station.