#1201 Issue closed: Extend script with manual optimal aligned partitioning for USB flash/SSD media

Labels: enhancement, fixed / solved / done

ProBackup-nl opened issue at 2017-02-21 00:40:

  • rear version (/usr/sbin/rear -V):
    Relax-and-Recover 2.00 / Git
  • OS version (cat /etc/rear/os.conf or lsb_release -a):
    LSB Version: 1.4
    Distributor ID: Arch
    Description: Arch Linux
    Release: rolling
    Codename: n/a
  • rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):
    n/a
  • Are you using legacy BIOS or UEFI boot?
    UEFI
  • Brief description of the issue:
    USB flash memory sticks and possibly SSD in USB adapters don't supply alignment information to the Linux kernel. As a result the USB medium partitioning more often far from an optimal alignment then that these devices are optimally aligned.
  • Work-around, if any:
    Manual edit usr/share/rear/format/USB/default/300_format_usb_disk.sh
    and change the parted command to something like
    parted -s $RAW_USB_DEVICE mkpart primary ~~2~~4Mib "$((~~2~~4 + ${USB_UEFI_PART_SIZE}))"Mib ...
    for the 2097152 byte4MiB optimal alignment as derived by flashbench for a "Sandisk Cruzer Force 16GB" USB stick.

This issue also applicable for the optimal sector and node size, where an adaptation in the command mkfs … will likely improve performance and reduce wear for thumb drives.

jsmeix commented at 2017-02-21 08:17:

Alignment according to physical block size
is a very good thing!

2097152 bytes (i.e. 2 MiB) looks unexpected small to me.
As far as I know usual flash storage operates with
a physical block size of 4 MiB or 8 MiB - nowadays
perhaps even 16 MiB on huge flash storages,
cf. my last comment on
https://hackweek.suse.com/15/projects/23

Because during "rear format" we can do what we want
(in contrast to "rear recover" where we must by default
recreate the system exactly as it was before)
I suggest to simply align at 8 MiB during "rear format"
in any case.
Who cares nowadays about a few unused MiB
at the beginning of a mass storage device?

jsmeix commented at 2017-02-21 09:35:

See also the related issue
https://github.com/rear/rear/issues/102

See also in the parted manual
https://www.gnu.org/software/parted/manual/parted.html

how to partition a low-end flash device
(“low-end”, as of 2011/2012).
For such devices, you should
use 4MiB-aligned partitions (2)
...
(2)
Cheap flash drives will be with us
for a long time to come, and, for them,
1MiB alignment is not enough.
Use at least 4MiB-aligned partitions.
For details, see Arnd Bergman’s article,
http://lwn.net/Articles/428584/
and its many comments.

FYI:
Regarding why during "rear recover"
we must try to recreate the partitioning
exactly as it was before see
https://github.com/rear/rear/issues/1078#issuecomment-266099227
and the subsequent comments, in particular
https://github.com/rear/rear/issues/1078#issuecomment-266378990

jsmeix commented at 2017-02-21 09:46:

Something like

parted ... mkpart primary 8Mib "$((8 + ${USB_UEFI_PART_SIZE}))"Mib ...

cannot work well because if e.g. USB_UEFI_PART_SIZE is 7 MiB
it still results bad alignment for the ReaR data partition.
We need "more intelligent" code that rounds
the user-specified values to 8 MiB chunks, cf.
https://github.com/rear/rear/issues/102#issuecomment-149175705

ProBackup-nl commented at 2017-02-22 00:06:

Indeed 2 MiB was too small. Right now I suspect the Sandisk Cruzer Fit 16GB to have 4 MiB physical blocks. I am still struggling with correctly interpreting flashbench output. I do agree that rounding user input to "block size" is necessary.

ProBackup-nl commented at 2017-03-05 01:01:

And this wish also request for some variable that allows inserting additional parameters for the mkfs command at usr/share/rear/format/USB/default/300_format_usb_disk.sh.

As an example how I modified that file after running Linaro's flashbench on the Sandisk Cruzer Fit:

-mkfs.$USB_DEVICE_FILESYSTEM -L REAR-000 ${RAW_USB_DEVICE}${ParNr} >&2 || Error "Failed to create $USB_DEVICE_FILESYSTEM filesystem on '${RAW_USB_DEVICE}${ParNr}'"
+mkfs.$USB_DEVICE_FILESYSTEM -L REAR-000 -b 4096 -E stride=8,stripe-width=256 -O sparse_super,^has_journal ${RAW_USB_DEVICE}${ParNr} >&2 || Error "Failed to create $USB_DEVICE_FILESYSTEM filesystem on '${RAW_USB_DEVICE}${ParNr}'"

FGrose commented at 2017-03-05 15:51:

@ProBackup-nl: Have you replicated the flashbench findings on other devices of varying size and source to confirm the applicability of those results for other units?

ProBackup-nl commented at 2017-03-05 16:01:

@FGrose Yes, I have verified using flashbench that the above values are not applicable to other USB flash drives. The above code is just an example.

I need something like:
+mkfs.$USB_DEVICE_FILESYSTEM -L REAR-000 ${USB_DEVICE_FILESYSTEM_PARAMS} ${RAW_USB_DEVICE}${ParNr} >&2 || Error "Failed to create $USB_DEVICE_FILESYSTEM fil...

ProBackup-nl commented at 2017-03-05 22:01:

@jsmeix "more intelligent" code that rounds to chunks has been implemented in pull request #1217

jsmeix commented at 2017-03-06 10:07:

@ProBackup-nl
many thanks for your contribution!

@FGrose
see https://github.com/rear/rear/pull/1217
by default USB_DEVICE_FILESYSTEM_PARAMS
is empty.
The basic idea behind such variables is to give the user
some means of "final power" so that the user can specify
if needed what ReaR will actually do, cf.
"too much secretly working 'magic automatisms' in ReaR" in
https://github.com/rear/rear/pull/1204#issuecomment-282252947

jsmeix commented at 2017-03-06 10:52:

With
https://github.com/rear/rear/pull/1217
merged, this issue should be fixed.

[Export of Github issue for rear/rear.]