FlightAware Discussions

How I created a 4Gb backup from a 16Gb microSD card

The Dirty Hack

NOTE: GParted was NOT used for Expanding the partitions. It was used only to graphically display partition details.

The trick is to set value of variable ROOT_DEV_SIZE in file /usr/lib/raspi-config/init_resize.sh to 4.5 Gb (converted to number of 512 byte blocks). Without this hack, its value is set by script equal to the size of microSD card.

 

1 - Slipped a 16 Gb microSD card in USB card reader, and pluged card reader to USB slot of Windows Desktop

2 - Wrote piaware image to microSD card by Win32DiskImager

3 - Started Debian 10.3 in VM on Windows, and attached microSd card to VM/Debian

4 - Started Gparted and checked partitions. These were as follows:

5 - Closed Gparted and started File explorer of Debian and clicked on attached devices “boot” and “rootfs”. This mounted the partitions.

6 - Started terminal and gave following command:

abcd@debian10:~$ sudo nano /media/abcd/rootfs/usr/lib/raspi-config/init_resize.sh

When the file opened in nano, pressed Ctrl+W, typed ROOT_DEV_SIZE and pressed Enter key. The cursor jumped to following line:

ROOT_DEV_SIZE=$(cat "/sys/block/${ROOT_DEV_NAME}/size")
TARGET_END=$((ROOT_DEV_SIZE - 1))

Commented out first line and inserted between first & second lines a new line, as shown below:

#ROOT_DEV_SIZE=$(cat "/sys/block/${ROOT_DEV_NAME}/size")
ROOT_DEV_SIZE=$((4500000000/512))
TARGET_END=$((ROOT_DEV_SIZE - 1))

Saved the file and closed the nano editor.

7 - Ejected microSD card from Debian and Windows, slipped into RPI, and performed first boot.
After boot checked partition sizes. The /root partition is only 4Gb:

pi@piaware:~ $ sudo fdisk -l | grep ^Device && sudo fdisk -l | grep ^/dev 

Device         Boot  Start     End Sectors   Size Id Type
/dev/mmcblk0p1        8192  352900  344709 168.3M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      360448 8789061 8428614     4G 83 Linux

8 - Configured piaware. Installed and configured other feeders (Flightradar24, Radarbox24, Planefinder, Adsbexchange, OpenSky), and Wiedehopf’s Performance Graphs.

9 - Shutdown RPi, sliped out microSD card, slipped into Windows computer and attached to VM/Debian10. Fired Gparted and took a screenshot.

10 - Created backup image by Win32DiskImager, which is only 4.4 Gb

piaware - backup 4 Gb

11 - Full expansion of microSD card after creating small backup:

11.1 - Closed Gparted and started File explorer of Debian and clicked on attached devices “boot” and “rootfs”. This mounted the partitions.

11.2 - Started terminal and gave following command:

abcd@debian10:~$ sudo nano /media/abcd/rootfs/usr/lib/raspi-config/init_resize.sh

When the file opened in nano, pressed Ctrl+W, typed ROOT_DEV_SIZE and pressed Enter key. The cursor jumped to followingline:

#ROOT_DEV_SIZE=$(cat "/sys/block/${ROOT_DEV_NAME}/size")
ROOT_DEV_SIZE=$((4500000000/512))
TARGET_END=$((ROOT_DEV_SIZE - 1))

Un-commented first line and deleted 2nd line, so it became as shown below (i.e. as it originally was):

ROOT_DEV_SIZE=$(cat "/sys/block/${ROOT_DEV_NAME}/size")
TARGET_END=$((ROOT_DEV_SIZE - 1))

Saved the file and closed the nano editor.

11.3 - Ejected microSD card from Debian and Windows, slipped into RPI, and powered up.
Checked partition sizes. Root partition is only 4 Gb.

pi@piaware:~ $ sudo fdisk -l | grep ^Device && sudo fdisk -l | grep ^/dev 

Device         Boot  Start     End Sectors   Size Id Type
/dev/mmcblk0p1        8192  352900  344709 168.3M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      360448 8789061 8428614     4G 83 Linux

11.4 - Started raspi-config
pi@piaware:~ $ sudo raspi-config

Advance options > Expand File System > Reboot

11.5 - After reboot checked partition sizes. Root partition expanded to full capacity of microSD card

pi@piaware:~ $ sudo fdisk -l | grep ^Device && sudo fdisk -l | grep ^/dev 

Device         Boot  Start      End  Sectors   Size Id Type
/dev/mmcblk0p1        8192   352900   344709 168.3M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      360448 30277631 29917184  14.3G 83 Linux

5 Likes

EXPLANATION OF THE METHOD

PROBLEM:

  1. Shrinking or trimming a partition is not permitted by most commands/softwares, and where permitted, is accompanied by warning “potential loss of data, are you sure?”. So shrinking/trimming is ruled out.

  2. When burned to microSD card, the piaware image occupies only about 2.2 Gb and rest of microSD card is “unallocated” space. Even this 2.2 Gb occupied space has about 0.3 Gb free space.

  3. After burning image, when microSD card is booted, Raspbian runs an script which expands root partition to full capacity of microSD card. So after first boot, no chance to shrink/trim.

WORKAROUND:
The solution is “progressive expansion”.

  1. Write image to microSD card (2.2 Gb)

  2. Mount microSD card’s root partition in Linux computer, and edit microSD card’s file /usr/lib/raspi-config/init_resize.sh and modify it to expand root partition to only 4.5 Gb. This is first stage of “proressive expansion”.

  3. Slip microSD in RPi and powerup. The Raspbian will expand root partition, but due to hack in item 2 above, it will expand only to 4 Gb.

  4. Since 4 Gb is enough to install all other data feeders, performance graph, rpi monitor etc, go ahead and complete the installation.

  5. After all your sofware installation & configuration is completed, shutdown RPi, slip out microSD card and slip into Desktop/Laptop and create a backup. This backup image will be only 4.3 Gb.

  6. Once backup is created, the 2nd stage of “progressive expansion” can be done. Plug the microSD card in a Linux computer, mount the root partition, open file /usr/lib/raspi-config/init_resize.sh in nano and reverse the hack done in in step 2 above.

  7. Slip microSD card in RPi and power up. Since auto expansion by Raspbian is done only on first boot, it wont automatically expand the root partition. This can be manually done by user by following method:
    sudo raspi-config > Advance Options > Expand File System > Yes to Reboot.
    As the hack to restrict expansion size was removed in step 6 above, Raspbian will expand the root partition to full capacity of microSD card. This is 2nd stage of “progressive expansion”.

1 Like

Shrinking a partition is fine if you know that the trimmed space is not in use
You can ensure that it’s not in use by running resize2fs with a size that corresponds to the new (smaller) partition size, before shrinking the partition. resize2fs will ensure that all the data of the filesystem is stored within the size that you give it.

resize2fs - Trimming 15 GiB image to 5000M (5GiB)

abcd@debian10:~$ sudo fdisk -l | grep ^/dev

/dev/sda1  *        2048 58720255 58718208  28G 83 Linux
/dev/sda2       58722302 67106815  8384514   4G  5 Extended
/dev/sda5       58722304 67106815  8384512   4G 82 Linux swap / Solaris
/dev/sdb1         8192   352900   344709 168.3M  c W95 FAT32 (LBA)
/dev/sdb2       360448 30277631 29917184  14.3G 83 Linux

abcd@debian10:~$ sudo resize2fs /dev/sdb2 5000M
resize2fs 1.44.5 (15-Dec-2018)
Filesystem at /dev/sdb2 is mounted on /media/abcd/rootfs; on-line resizing required
resize2fs: On-line shrinking not supported

abcd@debian10:~$ sudo umount /dev/sdb2

abcd@debian10:~$ sudo resize2fs /dev/sdb2 5000M
resize2fs 1.44.5 (15-Dec-2018)
Please run 'e2fsck -f /dev/sdb2' first.
abcd@debian10:~$ sudo e2fsck -f /dev/sdb2
e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: 54234/934720 files (0.7% non-contiguous), 464873/3739648 blocks

abcd@debian10:~$ sudo resize2fs /dev/sdb2 5000M
resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/sdb2 to 1280000 (4k) blocks.
The filesystem on /dev/sdb2 is now 1280000 (4k) blocks long.

Trimmed to 5 GiB by resize2fs - root partition is still 14.27 GiB

When done by progressive expansion - after first prorgessive expansion - root partition is only 4 GiB

Well, yes. Partitions are really just something like markers to show where the data starts and stops. By running resize2fs you have shuffled all the data to the start of partition, so there is unused space between the end of the filesystem and the “end marker” of the partition. Now you need to shrink the partition itself, moving the “end marker” to the new end of the filesystem.

(The grow-to-fill-sdcard process that the sdcard images use is similar, but in the other order - first the partition is grown so the “end marker” is moved to provide more space, then the filesystem is expanded with resize2fs to make use of that extra space. To shrink it, you reverse that process)

Bad analogy time: you’re renting some physical storage units (the partition) and you have a bunch of junk scattered across all of them (the filesystem contents) … but most of the units only have a few items in them. resize2fs is like consolidating your stuff so some units are full and some are empty. Shrinking the partition is like cancelling the rental on the empty units. You need to move your stuff out before cancelling the rental, and you need to actually cancel the rental before you stop paying for the empty space.

2 Likes

In progress

Tried to move end of partition on microSD card using:
Gparted > Partition > Resize/Move
The end marker wont move.

EDIT:

After rebooting Debian, GParted could shrink the paartition on microSD card. See 2nd screenshot

SCREENSHOT 1

SCREENSHOT 2

Resizing Using Two RPis

  • Shutdown Pi-1.
  • Slipped out its 16GB microSD card (which is required to be shrunk and then backed-up).
  • Slipped it into USB card reader.
  • Plugged card reader into Pi-2.
pi@piaware:~ $ sudo fdisk -l | grep ^Device && sudo fdisk -l | grep ^/dev
Device         Boot  Start      End  Sectors   Size Id Type
Device     Boot  Start      End  Sectors   Size Id Type
/dev/mmcblk0p1        8192   352900   344709 168.3M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      360448 15523839 15163392   7.2G 83 Linux
/dev/sda1         8192   352900   344709 168.3M  c W95 FAT32 (LBA)
/dev/sda2       360448 30277631 29917184  14.3G 83 Linux

 

pi@piaware:~ $ sudo e2fsck -f /dev/sda2
e2fsck 1.44.5 (15-Dec-2018)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: 54268/934720 files (0.7% non-contiguous), 465858/3739648 blocks

 

pi@piaware:~ $ sudo resize2fs /dev/sda2 2000M
resize2fs 1.44.5 (15-Dec-2018)
resize2fs: New size smaller than minimum (589091)

 

pi@piaware:~ $ sudo dumpe2fs -h /dev/sda2
.... .... ....
Block count:              3739648
Reserved block count:     154620
Free blocks:              3273790
Free inodes:              880452
First block:              0
Block size:               4096
.... .... ....

 

pi@piaware:~ $ python -c 'print 589091 * 4096 / (1024 * 1000)'
2356

 

pi@piaware:~ $ sudo resize2fs /dev/sda2 2800M
resize2fs 1.44.5 (15-Dec-2018)
Resizing the filesystem on /dev/sda2 to 716800 (4k) blocks.
The filesystem on /dev/sda2 is now 716800 (4k) blocks long.

===================================
Unplugged USB card reader from RPi and plugged into Windows Desktop.
Started creating backup image using Win2DiskImager.
Ran backup creation up to 25% (25%x14 GiB=3500M), then clicked exit button

===================================

Slipped 16Gb microSD card into Pi-1, booted.

pi@piaware:~ $ df -h | grep ^File &&  df -h | grep ^/dev
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       2.7G  1.6G  1.1G  60% /
/dev/mmcblk0p1  166M   53M  114M  32% /boot 

 

sudo raspi-config >> Advanced Options >> Expand Filesystem >> Finish >> Yes to Reboot

pi@piaware:~ $ df -h | grep ^File &&  df -h | grep ^/dev
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        15G  1.6G   12G  12% /
/dev/mmcblk0p1  166M   53M  114M  32% /boot

 

Has anyone tried this?

EDIT:

Tried this on piaware.
Build .deb package. On failure of dpkg -i due to dependencies, it recommended sudo apt --fix-broken install
When I issued this command, it went into frenzy of installing a million dependencies and took 35 minutes to do so.

Finally discovered it is for Raspbian with GUI (Desktop) while piaware is headless. It requires HDMI cable & Monitor. :angry:

This is the ADS-B Flight Tracking group and it doesn’t seem like anyone else is interested.

To create a backup I simply copy the microSD card to a PC image using Win32DiskImager and then write it out to another $10 microSD card.

YMMV

S.

1 Like

Nope, and I would not have looked for this, as I thought one could not clone a mounted fs.
For the record, yesterday, I’ve had to clone a SD (class 10) and used a USB3 reader on a suitable port, and got a steady, pretty stable read speed of 50MBps, to my surprise. I would not have imagined these µSD were able of such speed.
The 16GB image went in less than 5 minutes.

2 Likes

This is very good. Thanks.
I am curious to know:

(1) Which model/make of USB3 card reader you used. I would like to buy one like that.

(2) Which computer (Windows/Linux/RPi) and which software you used to do the job?

Some may have a need or like to maintain backups (I don’t care to personally, but some may have a need)… Threads not of interest can easily be skipped should one wish.

That said, those who are interested may like this: https://github.com/billw2/rpi-clone

1 Like

I’ve used a Transcend card reader https://www.amazon.fr/gp/product/B009D79VH4/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1
Surprisignly it’s still available. Even more surprisingly, its price has almost doubled since 2013.
The computer is a Lenovo Thinkpad T430, with a 7200RPM hard drive (it may have its importance here), running on Windows 7, card read with Win32 Disk Imager.
I’ve just ran the test again, I confirm the figures. One thing I haven’t tested yesterday, it’s even faster (~60MBps) with the integrated card reader.
Writing speeds were more conventional, around 10 or so MBps.

1 Like

Thank you @Nitr0 !

All following done on RPi-4 booted with piaware 3.8.1 on 16 Gb class 10 microSD card (microSD card to be cloned):

 

(1) Before plugging in the USB ard reader with spare microSD card:

pi@piaware:~ $ sudo fdisk -l | grep ^Device && sudo fdisk -l | grep ^/dev
Device         Boot  Start      End  Sectors   Size Id Type
Device     Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1        8192   352900   344709 168.3M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      360448 30277631 29917184  14.3G 83 Linux

 

Ensured that the USED space on 16 GB card is less than 8Gb (as I will be cloning it to 8Gb card)

pi@piaware:~ $ df -h | grep ^File && df -h | grep ^/dev

image

Yes, the used space is only 1.5G (root) + 166Mb (full size of boot), so it will easily fit on a 8Gb microSD card.

 

(2) After plugging in USB 2.0 card reader with 8Gb class 10 microSD card (target microSD card):

pi@piaware:~ $ sudo fdisk -l | grep ^Device && sudo fdisk -l | grep ^/dev
Device         Boot  Start      End  Sectors   Size Id Type
Device     Boot Start      End  Sectors  Size Id Type
/dev/mmcblk0p1        8192   352900   344709 168.3M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      360448 30277631 29917184  14.3G 83 Linux
/dev/sda1        8192 15523839 15515648  7.4G  b W95 FAT32

 

(3) Cloning:

pi@piaware:~ $ git clone https://github.com/billw2/rpi-clone.git   
pi@piaware:~ $ cd rpi-clone  
pi@piaware:~/rpi-clone $ sudo cp rpi-clone rpi-clone-setup /usr/local/sbin  

## Checked to make sure  /usr/local/sbin is in my $PATH
pi@piaware:~/rpi-clone $ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games
pi@piaware:~/rpi-clone $ sudo rpi-clone sda   

Booted disk: mmcblk0 15.5GB                Destination disk: sda 7.9GB
---------------------------------------------------------------------------
Part      Size    FS     Label           Part   Size    FS     Label
1 /boot   168.0M  fat32  --              1        7.4G  fat32  --
2 root     14.3G  ext4   rootfs
---------------------------------------------------------------------------
== Initialize: IMAGE partition table - partition number mismatch: 2 -> 1 ==
1 /boot               (52.0M used)   : MKFS  SYNC to sda1
2 root                (1.5G used)    : RESIZE  MKFS  SYNC to sda2
---------------------------------------------------------------------------
Run setup script       : no.
Verbose mode           : no.
-----------------------:
** WARNING **          : All destination disk sda data will be overwritten!
** WARNING **          : Your source /boot partition is smaller than the
                       :   Raspbian Buster 256M standard.  Consider using
                       :   the '-p 256M' option to avoid /boot clone errors.
-----------------------:
Initialize and clone to the destination disk sda?  (yes/no):  yes
Optional destination ext type file system label (16 chars max): piaware381-bkup
Initializing
  Imaging past partition 1 start.
  => dd if=/dev/mmcblk0 of=/dev/sda bs=1M count=8 ...
  Resizing destination disk last partition ...
    Resize success.
  Changing destination Disk ID ...
  => mkfs -t vfat -F 32  /dev/sda1 ...
  => mkfs -t ext4 -L piaware381-bkup /dev/sda2 ...

Syncing file systems (can take a long time)
Syncing mounted partitions:
  Mounting /dev/sda2 on /mnt/clone
  => rsync // /mnt/clone with-root-excludes ...
  Mounting /dev/sda1 on /mnt/clone/boot
  => rsync /boot/ /mnt/clone/boot  ...

Editing /mnt/clone/boot/cmdline.txt PARTUUID to use 7b08eb33
Editing /mnt/clone/etc/fstab PARTUUID to use 7b08eb33
===============================
Done with clone to /dev/sda
   Start - 16:34:32    End - 16:42:31    Elapsed Time - 7:59

Cloned partitions are mounted on /mnt/clone for inspection or customizing.

Hit Enter when ready to unmount the /dev/sda partitions ...
  unmounting /mnt/clone/boot
  unmounting /mnt/clone
===============================

pi@piaware:~/rpi-clone $

STARTED: 12:34 pm EST
FINISHED: 12:42 pm EST

Total cloning time of 16Gb card to 8 Gb card (both class 10) = 8 minutes with USB 2.0 card reader (will be shorter with USB 3 card reader)

(4) After completion of cloning:

pi@piaware:~ $ sudo fdisk -l | grep ^Device && sudo fdisk -l | grep ^/dev
Device         Boot  Start      End  Sectors   Size Id Type
Device     Boot  Start      End  Sectors   Size Id Type
/dev/mmcblk0p1        8192   352900   344709 168.3M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      360448 30277631 29917184  14.3G 83 Linux
/dev/sda1         8192   352900   344709 168.3M  c W95 FAT32 (LBA)
/dev/sda2       360448 15523839 15163392   7.2G 83 Linux

 

Booted RPi from Cloned 8Gb microSD card, worked perfect

 
 

Thanks for the detailed information. It takes me about 25 minutes to copy or write a 32 Gb Class 10 microSD card.

Since all my microSD cards are class 10, I feel these are the factors which cause this huge difference in write time:

  • The USB3 vs USB2
  • Hard Drive 7200 rmp vs 4800 rpm

The script for cloning a booted OS/mounted fs at
https://github.com/billw2/rpi-clone
seems to be very popular.

Stars: 1032
Forks: 178

These work pretty well here too - when plugged into a USB 3 port they’ll easily hit the sdcard’s max speed.

1 Like

My problem was not only speed, it was also the problem of limited space on Desktop’s Hard Drive.

I have 3 Pis, each with 16 Gb card. The three backup by Win32DiskImager will take 3x16=48 Gb space out of 200 Gb Windows partition. I end up in space problem. That is why I wanted to shrink the backup particularly when used space on each microSD card was only about 3 Gb.

To solve speed problem, I have decided to purchase Transcend USB3 card reader. However, I realized that my Desktop is old, and has only USB2 ports, so USB3 card reader wont give any improvement in speed.

One trick for reducing the backup size is to run zerofree on the partition (or, better, on the image file) before compressing it. This will zero out unused blocks which otherwise might hold old copies of deleted/replaced data, which improves the compression ratio.

3 Likes