#2761 Issue closed: USB formatting is really slow¶
Labels: support / question, external tool,
special hardware or VM, no-issue-activity
XXLC opened issue at 2022-02-15 09:45:¶
-
ReaR version: cloned from GitHub, 2.6
-
OS version: Ubuntu 20.04.3 LTS
-
ReaR configuration files : nothing in side
-
Hardware vendor/product: PC
-
System architecture: x86 compatible
-
Firmware: UEFI - Grub
-
Storage : localdisk
-
Storage layout:
NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT
/dev/loop0
/dev/loop0 loop squash 55,4M /snap/core
/dev/loop1
/dev/loop1 loop squash 219M /snap/gnom
/dev/loop2
/dev/loop2 loop squash 65,1M /snap/gtk-
/dev/loop3
/dev/loop3 loop squash 51M /snap/snap
/dev/loop4
/dev/loop4 loop squash 32,3M /snap/snap
/dev/loop5
/dev/loop5 loop squash 55,5M /snap/core
/dev/loop6
/dev/loop6 loop squash 43,4M /snap/snap
/dev/loop7
/dev/loop7 loop squash 4K /snap/bare
/dev/loop8
/dev/loop8 loop squash 219M /snap/gnom
/dev/loop9
/dev/loop9 loop squash 61,9M /snap/core
/dev/loop10
/dev/loop10
loop squash 65,2M /snap/gtk-
/dev/loop11
/dev/loop11
loop squash 54,2M /snap/snap
/dev/loop12
/dev/loop12
loop squash 248,8M /snap/gnom
/dev/loop13
/dev/loop13
loop squash 82,9M /snap/disc
/dev/loop14
/dev/loop14
loop squash 164,8M /snap/gnom
/dev/sda /dev/sda sata disk 931,5G
|-/dev/sda1
| /dev/sda1 /dev/sda
| part ntfs WinServer2019-1
| 258,9G /media/lon
|-/dev/sda2
| /dev/sda2 /dev/sda
| part ext4 249,8G /
|-/dev/sda3
| /dev/sda3 /dev/sda
| part ntfs SharedData 407,5G /media/lon
-/dev/sda4
/dev/sda4 /dev/sda
part vfat EFI SYSTEM 512M /boot/efi
/dev/sdb /dev/sdb sata disk 931,5G
|-/dev/sdb1
| /dev/sdb1 /dev/sdb
| part vfat 100M
|-/dev/sdb2
| /dev/sdb2 /dev/sdb
| part 16M
|-/dev/sdb3
| /dev/sdb3 /dev/sdb
| part ntfs Win10 200,6G /media/lon
|-/dev/sdb4
| /dev/sdb4 /dev/sdb
| part ntfs Win10-Data 730,3G /media/lon
-/dev/sdb5
/dev/sdb5 /dev/sdb
part ntfs 508M
/dev/sdc /dev/sdc sata disk 111,8G
`-/dev/sdc1
/dev/sdc1 /dev/sdc
part ntfs BackUp 111,8G /media/lon
/dev/sdd /dev/sdd sata disk 232,9G
|-/dev/sdd1
| /dev/sdd1 /dev/sdd
| part ntfs Recovery 499M
|-/dev/sdd2
| /dev/sdd2 /dev/sdd
| part vfat 100M
|-/dev/sdd3
| /dev/sdd3 /dev/sdd
| part 16M
-/dev/sdd4
/dev/sdd4 /dev/sdd
part ntfs 232,3G /media/lon
/dev/sde /dev/sde usb disk 117,2G
|-/dev/sde1
| /dev/sde1 /dev/sde
| part vfat REAR-EFI 512M
-/dev/sde2
/dev/sde2 /dev/sde
part 116,7G
- Description of the issue (ideally so that others can reproduce it):
after cloned and moved to ReaR folder, i did a format to my usb stick
(128gb, usb 3.1) using sudo usr/sbin/rear format /dev/sde1, the
terminal stuck after i typed Yes then press Enter. I use
watch -n 0.1 iostat to check if ReaR formatting my usb or not, yes it
is formatting to ext3 but the speed is very low, only ~170 kb/s, is
there anyway to speed up the speed of formatting? Currently i don't have
any spare usb stick so i used 128gb which to stored data but with this
rate must take days for it to complete, and i don't want to interupt the
formatting process cause could break my usb.
- Workaround, if any: no
jsmeix commented at 2022-02-15 13:20:¶
ReaR calls the usual tools to set up partitioning (with 'parted')
and filesystems setup (with 'mkfs...') on the USB disk, cf.
https://github.com/rear/rear/blob/master/usr/share/rear/format/USB/default/300_format_usb_disk.sh
So things work as fast or slow as those tools do
or more precisely as the USB disk and its USB connection hardware
and the associated kernel drivers do.
In general regarding USB speed:
Nowadays there are basically three different kind of USB connectors:
- USB 1.0 and USB 1.1 with so called "Low Speed" and "Full Speed" (USB kernel driver uhci_hcd or ohci_hcd)
- USB 2.0 with so called "Hi-Speed" (USB kernel driver ehci_hcd)
- USB 3.0 and USB 3.1 with so called "SuperSpeed" and "SuperSpeed+" (USB kernel driver xhci_hcd)
Neither the color of the USB connector nor what it is labeled on the
computer
is reliable regarding what kernel driver is used for a USB port.
Only the "lsusb -t" output shows what kernel driver is used.
What USB kernel driver is used depends on what USB hardware there is in
the computer.
XXLC commented at 2022-02-16 02:32:¶
ReaR calls the usual tools to set up partitioning (with 'parted') and filesystems setup (with 'mkfs...') on the USB disk, cf. https://github.com/rear/rear/blob/master/usr/share/rear/format/USB/default/300_format_usb_disk.sh
So things work as fast or slow as those tools do or more precisely as the USB disk and its USB connection hardware and the associated kernel drivers do.
In general regarding USB speed:
Nowadays there are basically three different kind of USB connectors:
- USB 1.0 and USB 1.1 with so called "Low Speed" and "Full Speed" (USB kernel driver uhci_hcd or ohci_hcd)
- USB 2.0 with so called "Hi-Speed" (USB kernel driver ehci_hcd)
- USB 3.0 and USB 3.1 with so called "SuperSpeed" and "SuperSpeed+" (USB kernel driver xhci_hcd)
Neither the color of the USB connector nor what it is labeled on the computer is reliable regarding what kernel driver is used for a USB port. Only the "lsusb -t" output shows what kernel driver is used. What USB kernel driver is used depends on what USB hardware there is in the computer.
This is what i got from lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/8p, 5000M
|__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
That mean my usb port support either 3.0 and 3.1 right? But my usb stick
's kernel driver is usb-storage so i'm kinda confusing
github-actions commented at 2022-04-17 02:46:¶
Stale issue message
github-actions commented at 2022-06-23 03:21:¶
Stale issue message
ZENAdmin-Ops commented at 2022-11-02 03:41:¶
Hello,
I have the same issue as @XXLC
I am also using v2.6 of rear
OS is Ubuntu 22.04
lusb - t
produces no output?
I would like to identify a way to improve the speed of the formatting.
Is there a quickformat option for instance?
With Windows you can select quickformat, which does not scan the target disk for bad sectors and is many times quicker than a "standard" format.
will-code-for-pizza commented at 2022-12-26 17:15:¶
Same issue here.
Formatting a 6 TB USB 3.0 HDD with
$ sudo rear -v -d -D format -- -f -y /dev/sdd
it takes 3 days without finalization in the line
Creating ext3 filesystem with label 'REAR-000' on ReaR data partition
/dev/sdd3
Adding '-v' to the 3 related lines in '300_format_usb_disk.sh' does not bring any more information.
It also seems, that ReaR does not handle a fresh HDD correctly. Partitioning only works, if I create a GPT partition table manually first.
PS: Update: Formatting the disk with "gparted" takes 30 minutes. But I did not manage to create a functional and bootable layout like the CLI command above.
$ lsusb
Bus 004 Device 002: ID 152d:0578 JMicron Technology Corp. / JMicron USA Technology Corp. JMS567 SATA 6Gb/s bridge
$ lsusb -t
/: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
|__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
|__ Port 5: Dev 13, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 14, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
|__ Port 4: Dev 15, If 0, Class=Hub, Driver=hub/4p, 12M
|__ Port 1: Dev 16, If 0, Class=Human Interface Device, Driver=usbhid, 12M
|__ Port 1: Dev 16, If 1, Class=Human Interface Device, Driver=usbhid, 12M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
$ iostat
Linux 5.4.0-31-generic (WOPR) 26.12.2022 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
28,29 0,02 8,15 4,89 0,00 58,65
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sdd 2,32 73,98 2,04 0,00 183401 5056 0
EDIT:
Trying with
- ext4
- GRUB_CMDLINE_LINUX_DEFAULT="usb-storage.quirks=152d:0578:u"
- mkfs.$USB_DEVICE_FILESYSTEM -v -E lazy_itable_init -T big -L "$USB_DEVICE_FILESYSTEM_LABEL" $USB_DEVICE_FILESYSTEM_PARAMS $data_partition_device (300_format_usb_disk.sh)
will-code-for-pizza commented at 2022-12-26 18:31:¶
mkfs.$USB_DEVICE_FILESYSTEM -v -F -O ^64bit -E lazy_itable_init -T big -L "$USB_DEVICE_FILESYSTEM_LABEL" $USB_DEVICE_FILESYSTEM_PARAMS $data_partition_device
results in a MASSIVE speedup while creating the disk with ex4 and the
storage quirk setting above.
Finished after a few seconds !
$ iostat
Linux 5.4.0-31-generic (WOPR) 26.12.2022 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
36,79 0,03 8,87 3,93 0,00 50,39
Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd
sdd 8,72 172,21 1323,54 0,00 426509 3277920 0
So probably these options could be set with the USB_DEVICE_FILESYSTEM_PARAMS ?
ZENAdmin-Ops commented at 2022-12-26 22:03:¶
Does this statement apply in all circumstances or is it device capacity
specific?
GRUB_CMDLINE_LINUX_DEFAULT="usb-storage.quirks=152d:0578:u"
will-code-for-pizza commented at 2022-12-29 22:05:¶
Does this statement apply in all circumstances or is it device capacity specific?
GRUB_CMDLINE_LINUX_DEFAULT="usb-storage.quirks=152d:0578:u"
It's an specific entry for a dedicated USB device '152d:0578' wich works
better without UAS.
Unfortunatelly, most SATA2USB adapters from JMicron have this problem.
It's not device capacity specific and works with small and big HDD/SSDs,
because it depends on the USB controller chip.
[Export of Github issue for rear/rear.]