#1161 Issue `closed`: Change find to ls -r in 30[0]_create_extlinux.sh¶

Labels: won't fix / can't fix / obsolete

lrirwin opened issue at 2017-01-10 18:11:¶

rear version (/usr/sbin/rear -V): 1.18 forward
Brief description of the issue:
When "backup" folders exist, using "find" takes a long time.
Suggest this change at line 164 in output/USB/Linux-i386/30[0]_create_extlinux.sh:

- for file in $(cd $BUILD_DIR/outputfs; find rear/*/* -name syslinux.cfg); do
+ for file in $(cd $BUILD_DIR/outputfs; ls -r rear/*/*/syslinux.cfg); do

This is much quicker when the backup folders contain thousands of files.
It also accomplishes part of the TODO mentioned in the script.
Thanks for all you do!

jsmeix commented at 2017-01-11 10:47:¶

In general regarding using 'ls' versus 'find'
when its argument contains a wildcard '*' see
https://raw.githubusercontent.com/rear/rear/master/usr/share/rear/prep/NETFS/default/070_set_backup_archive.sh
(excerpt)

# Here things like 'find /path/to/dir -name '*.tar.gz' | sort' are used because
# one cannot use bash globbing via commands like 'ls /path/to/dir/*.tar.gz'
# because /usr/sbin/rear sets the nullglob bash option which leads to plain 'ls'
# when '/path/to/dir/*.tar.gz' matches nothing (i.e. when no backup file exists)
# so that then plain 'ls' would result nonsense.

I.e. before calling

ls -r rear/*/*/syslinux.cfg

one would need to ensure that "rear/*/*/syslinux.cfg"
does never evaluate to nothing - i.e. an additional test
would be needed before calling "ls -r rear/*/*/syslinux.cfg".

Comparison how much time is actually saved
(on my test system):
with "find"

real    0m0.007s
user    0m0.006s
sys     0m0.001s

with "ls -r"

real    0m0.018s
user    0m0.015s
sys     0m0.002s

Surprise!
With "ls -r" ist is actually much slower.

But even if it was faster with "ls -r"
the whole stuff would be about saving
at most a few thousandths of a second
which is not worth any effort.

jsmeix commented at 2017-01-11 11:01:¶

@lrirwin
can you explain what you mean with
"backup folders contain thousands of files"?

lrirwin commented at 2017-01-11 16:02:¶

Yes -- each night these run:
rear -v savelayout
rear -vdD mkbackup
That creates folders like:
$BUILD_DIR/outputfs/rear/$HOSTNAME/20170110.2215/backup
The entire linux system is backed up into that folder.
I've succeeded in being able to incorporate --link-dest into ReaR's
rsync method, so there may be as many as 15 folders on any given media.
Each of them is a complete system backup, so there are thousands of
entries in each backup folder.
ls -r runs a lot faster than using find and it gives them in reverse
order as well...

BTW - this is the --link-dest solution I came up with in /usr/share/rear/backup/NETFS/default/50_make_backup.sh at line 90 where the rsync option is:¶

(rsync) # make sure that the target is a directory mkdir -p $v "$backuparchive" >&2 LINKDEST=`ls -d $BUILD_DIR/outputfs/rear/$HOSTNAME//backup/bin 2>/dev/null | tail -n 1 | cut -f1-8 -d"/"` case $LINKDEST in "") LINKDESTOPT="";; ) LINKDESTOPT="--link-dest=$LINKDEST";; esac Log $BACKUP_PROG $v "${BACKUP_RSYNC_OPTIONS[@]}" --one-file-system --delete --exclude-from=$TMP_DIR/backup-exclude.txt --delete-excluded ${LINKDESTOPT} $(cat $TMP_DIR/backup-include.txt) "$backuparchive" $BACKUP_PROG $v "${BACKUP_RSYNC_OPTIONS[@]}" --one-file-system --delete --exclude-from=$TMP_DIR/backup-exclude.txt --delete-excluded ${LINKDESTOPT} $(cat $TMP_DIR/backup-include.txt) "$backuparchive" ;;¶

Saves a ton of space on backup media...

On 01/11/2017 06:01 AM, Johannes Meixner wrote:

@lrirwin https://github.com/lrirwin
can you explain what you mean with
"backup folders contain thousands of files"?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rear/rear/issues/1161#issuecomment-271840419, or
mute the thread
https://github.com/notifications/unsubscribe-auth/ABZTrFk9CV7El8dl9DUvStHAPHxcrkRQks5rRLafgaJpZM4LfuZ5.

jsmeix commented at 2017-01-12 08:28:¶

By guessing from your comments it seems
you use rysnc as backup method and that results
very many individual files while I use 'tar' so that
I get only a few tar archives so that for me the
whole difference is about a few millisecods.

I really helps to provide the information as requested in
https://github.com/rear/rear/blob/master/.github/ISSUE_TEMPLATE.md
to aviod wasting time with guesswork.

Furthermore preferably provide your proposed changes
as GitHub pull requests which also makes it easier for us
to see what you actually like to change.

Finally see
https://github.com/rear/rear/wiki/Coding-Style
so that over the time the ReaR code can get better step by step.

[Export of Github issue for rear/rear.]

#1161 Issue closed: Change find to ls -r in 30[0]_create_extlinux.sh¶