Minimizing writes on SD


#1

I am looking at the SD I/O bandwidth. It shows that there is a constant write process, at around 45-55 kB/sec. Probably this is part of the normal process of writhing the position of the planes in the web page location, but I am worried about wear of the SD card.
I did the following to alleviate this issue, at least in my mind.
First I edited /etc/fstab (sudo nano /etc/fstab) to add this line:
none /var/log tmpfs size=1M,noatime 0 0
Secondly I have tuned the write caching of the linux kernel by editeding /etc/sysctl.d/local.conf (sudo nano /etc/sysctl.d/local.conf). It was nothing there, so I added:

vm.dirty_background_ratio=20
vm.dirty_ratio=40

vm.dirty_writeback_centisecs=6000
vm.dirty_expire_centisecs=6000

To see all the settings use: sysctl -a
vm.dirty_background_ratio – This is the percentage of RAM that can be filled with dirty memory pages before it is written to disk. The default value on Raspbian is 10.
vm.dirty_ratio – This is the maximum amount of RAM that can be filled with dirty pages before writing the dirty memory pages to disk. The default value on Raspbian is 20.

Any comments/ideas?
Can we make the web page file to be part of a RAM tmpfs drive (symlinked)? I am thinking about the whole folders /var/www/html/graphs/, /var/log/ and, if present, /var/www/html/dump978/data/

LE: After the above changes the “Disk I/O bandwidth” changed to 29.4kB/sec.


#2

I have changed again the timing for writeback to be even longer, from 60 to 120 seconds.
The write I/O decreased again. So now it reads:

vm.dirty_writeback_centisecs=12000
vm.dirty_expire_centisecs=12000

And the pic. Initial values, then a while with the 1min delay, and now with 2 minutes delay.
The Writes decreased from average of 45kB/s to 15kB/s and iops from an average of 6 iops to 1.5 iops.


#3

What image is this? A standard piaware image doesn’t generate anything like that much i/o to the sdcard.


#4

It’s the ADS-B project, with one dump1090-fa and one dump978 workers feeding FA and FR24.

Q: Isn’t the normal piaware package writing the data in the /var/www/html/graphs/ folder? Isn’t that folder on the SD disc (not in RAM)?


#5

No, these graphs are not part of piaware.

A quick measurement on a piaware sdcard install here shows writes at about 3kB/s and 0.5 iops.


#6

Well, it might be true. Maybe is that measuring panel itself.
How do I measure individual tasks?


#7

Try iotop (you’ll probably need to apt-get install it)


#8

I solved microSD card failure due to writes in an entirely different way :slightly_smiling_face:

  1. Saved backup copies of microSD cards on my Desktop.
  2. Waited till there was a special offer / sale at local computer & electronics store, selling Adata 8 Gb class10 microSD card for Candain $5.99 (US $4.59, £3.54, €3.95).
    I purchased a bunch of these.

#9

did you use the ‘read’ feature of Winimager ?
how do you ‘restore’ the backup ? ( I forecast a new ‘tweaks’ addition… :wink: )


#10

I don’t get it…

itop


#11

My bad. Wrong software.

I guess I need to stop the dump978 process. It’s not behaving nice.


#12

Yes, used "Win32 Disk Imager" and clicked its read button.

Please see these:

1. Backup Restore Raspberry Pi OS (youtube video)

2. How to back up your Raspberry Pi (web page)

.

None of my microSD cards have failed yet, so I never had to restore from backup, but I think it is like burning an image. I will try and let you know.


#13

OK, I have burned one of backup images to a microSD card. It is done just like Raspbian or Piaware image is burned to microSD card.

While making backup copy, you can choose a name of backup copy with or without .img extension. However a name with .img extension will make finding & selecting the backup easier during burning of backup to microSD card,


#14

thanks @abcd567 :smile:


#15

Bear in mind you can boot and run a RPi3 up from an external USB HDD and forgo an SD card entirely. You can pick up 80GB drives from around $20 new. Not sure if earlier Pi’s can do this.


#16

I have external HDD’s but it would be a cumbersome installation. When in reality, it just needs that the writes to be done on the tmpfs partition (RAM) instead of the SD.
Caching the writes with deferring helps.
Also, well known SD manufacturers add wear leveling algorithms to their cards, but don’t expect the cheap, no-name ones to have that implemented.