#426 Issue closed: Ubuntu + Duply

Labels: support / question

tyl0re opened issue at 2014-06-15 11:17:

Does not work in Ubuntu 12.04.4 LTS:

It gets better when changing in:

  1. /usr/share/rear/build/DUPLICITY/default/60_create_python_symlink.sh
    python2|python3)
    to
    python2|python3|python2.7)

Now some Missing Files Appears:
in
/usr/share/rear/prep/DUPLICITY/default/05_prep_duplicity.sh
Must be added:

For duply:
in COPY_AS_IS
/usr/bin/env

for duplicity:
/usr/include/python2.7/pyconfig.h
/etc/apt
/usr/bin/dpkg
/var/crash (I think Can be left only needed before all Librarys have been added)

in LIBS

/lib/x86_64-linux-gnu/libexpat.so.1
/usr/lib/x86_64-linux-gnu/libapt-pkg.so.4.12
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16

(when adding the libstdc++ the Boot menu disappeared
and the Message

EDD: Error 0c00 reading sector 59099
menu.c32: not a COM32R image

Appears, but Booting with typing rear still works

Now it appears:
ImportError: No module named duplicity

Now i'm a little bit lost what is missing now

gdha commented at 2014-06-16 09:30:

The LIBS should be auto detected (try not to add these by hand if possible). The libstdc++ library might be used by syslinux (menu.c32)?

$ find /usr/lib -name "libstdc++*"
/usr/lib/gcc/x86_64-redhat-linux/4.8.2/32/libstdc++.a
/usr/lib/gcc/x86_64-redhat-linux/4.8.2/32/libstdc++.so
/usr/lib/gcc/x86_64-redhat-linux/4.8.2/libstdc++.so

schlomo commented at 2014-06-16 10:23:

Though IIRC our auto detect code works only for compiled stuff, not for
Perl / Python modules.

On 16 June 2014 11:30, gdha notifications@github.com wrote:

The LIBS should be auto detected (try not to add these by hand if
possible). The libstdc++ library might be used by syslinux (menu.c32)?

$ find /usr/lib -name "libstdc++*"
/usr/lib/gcc/x86_64-redhat-linux/4.8.2/32/libstdc++.a
/usr/lib/gcc/x86_64-redhat-linux/4.8.2/32/libstdc++.so
/usr/lib/gcc/x86_64-redhat-linux/4.8.2/libstdc++.so

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/issues/426#issuecomment-46158048.

tyl0re commented at 2014-06-16 10:32:

I could create a test server and a rear bootet "test-server"r to check maybe its faster when someone could have a look? (if you want, to verify if its a bug)

gdha commented at 2014-06-16 13:47:

still not sure where you got the requirements of libstdc++* from? For the rest you got the analysis right.

tyl0re commented at 2014-06-16 14:51:

I have added the Library one by one. and at one time threre was the message of the missing libstdc++ when starting duplicity

gdha commented at 2014-07-30 14:45:

Just to be complete:
./usr/share/rear/prep/DUPLICITY/default/05_prep_duplicity.sh:/usr/include/python2.7/pyconfig.h was already added into the script (but forgot to link with this issue)

And, if I'm not mistaken, the link of python should also be corrected in ./usr/share/rear/build/DUPLICITY/default/60_create_python_symlink.sh

gdha commented at 2014-08-22 14:31:

@tyl0re @mavit If you find some time could you verify if the behavior is better in rear-1.16.1-git201408201003?

tyl0re commented at 2014-08-30 16:06:

Zitat von gdha notifications@github.com:

@tyl0re[1] @mavit[2] If you find some time could you verify if the
behavior is better in rear-1.16.1-git201408201003?

—
Reply to this email directly or view it on GitHub[3].

Hi, Sorry i was on holliday, now i get that libexpat.so.1 is missing

[1] https://github.com/tyl0re
[2] https://github.com/mavit
[3] https://github.com/rear/rear/issues/426#issuecomment-53067325

gdha commented at 2014-09-01 14:24:

@tyl0re In my case it seems to find it (on f20):

2014-09-01 10:34:09 Adding required /lib64/libexpat.so.1 to LIBS
2014-09-01 10:34:09 Adding required /lib64/libexpat.so.1.6.0 to LIBS

tyl0re commented at 2014-09-01 14:57:

Do you mean on the cd or on the Host?
On the Host it is:
root@mail:/home/lore# ls -al /lib/x86_64-linux-gnu/libexpat.so.1
lrwxrwxrwx 1 root root 17 Aug 9 2012 /lib/x86_64-linux-gnu/libexpat.so.1 -> libexpat.so.1.5.2

tyl0re commented at 2014-09-03 20:29:

From the /var/log/rear/rear-server.log from building:

cp: cannot stat `/lib/libexpat.so.1': No such file or directory
`/lib/libgssglue.so.1.0.0' -> `/tmp/rear.Czhc93WWLZQO6b4/rootfs/lib/libgssglue.so.1.0.0'

On The Server:

rear# locate libexpat.so.1
/lib/x86_64-linux-gnu/libexpat.so.1
/lib/x86_64-linux-gnu/libexpat.so.1.5.2

I have a suggestion mayby to get all Library rights,mayby it solves problems when using other Method like webdav,ftp,.....

strace -f -e open duply $DUPLY_PROFILE status 2>&1 1>/dev/null|egrep "lib.+.so.*" |cut -d \" -f 2 > /tmp/lib_list

The Lib list is:
http://pastebin.com/CEsWY1f0

gdha commented at 2014-09-04 08:40:

thank you for the tip (not bad at all - perhaps should add this to our developers guide [which does not yet exist]:) :

$ sort -u /tmp/lib_list | while read a ; do
> echo ${a%/*}
> done | sort -u
/lib/x86_64-linux-gnu
urllibmodule.so
/usr/lib
/usr/lib/pymodules/python2.7
/usr/lib/python2.7
/usr/lib/python2.7/dist-packages
/usr/lib/python2.7/email
/usr/lib/python2.7/email/mime
/usr/lib/python2.7/encodings
/usr/lib/python2.7/lib-dynload
/usr/lib/python2.7/lib-tk
/usr/lib/python2.7/logging
/usr/lib/python2.7/plat-linux2
/usr/lib/python2.7/xml
/usr/lib/python2.7/xml/dom
/usr/lib/python2.7/xml/parsers
/usr/local/bin
/usr/local/lib/python2.7/dist-packages
/usr/local/lib/python2.7/dist-packages/duplicity
/usr/local/lib/python2.7/dist-packages/duplicity/backends

On my fedora box it looks like:

[gdha@fedora20 rear]$ sort -u /tmp/lib_list | while read a ; do
> echo ${a%/*}
> done | sort -u
/lib64
/usr/bin
/usr/lib64/python2.7
/usr/lib64/python2.7/encodings
/usr/lib64/python2.7/lib-dynload
/usr/lib64/python2.7/logging
/usr/lib64/python2.7/plat-linux2
/usr/lib64/python2.7/site-packages
/usr/lib64/python2.7/site-packages/duplicity
/usr/lib64/python2.7/site-packages/duplicity/backends
/usr/lib64/python2.7/site-packages/gtk-2.0
/usr/lib64/python27.zip
/usr/lib/python2.7/site-packages

tyl0re commented at 2014-09-04 11:56:

O.k Try 2:

FILES=strace -Ff -e open duply $DUPLY_PROFILE status 2>&1 1>/dev/null|grep -v '= -1'|grep -i open|grep -v "open resumed" |cut -d \" -f 2|sort -u

for name in $FILES; do
if [ -f $name ] || [ -L $name ]; then
DATEI=readlink -f $name
LIB=file $DATEI|grep "shared object"|cut -d \: -f 1
if [ "x$LIB" != "x" ]; then
echo $LIB
fi
fi
done

Files= (And then the ' is missing in the comment),seems to be cut out
Output on my System:
/lib/x86_64-linux-gnu/libacl.so.1.1.0
/lib/x86_64-linux-gnu/libattr.so.1.1.0
/lib/x86_64-linux-gnu/libbz2.so.1.0.4
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
/lib/x86_64-linux-gnu/libc-2.15.so
/lib/x86_64-linux-gnu/libdl-2.15.so
/lib/x86_64-linux-gnu/libexpat.so.1.5.2
/lib/x86_64-linux-gnu/libgcc_s.so.1
/lib/x86_64-linux-gnu/libm-2.15.so
/lib/x86_64-linux-gnu/libnsl-2.15.so
/lib/x86_64-linux-gnu/libnss_compat-2.15.so
/lib/x86_64-linux-gnu/libnss_dns-2.15.so
/lib/x86_64-linux-gnu/libnss_files-2.15.so
/lib/x86_64-linux-gnu/libnss_nis-2.15.so
/lib/x86_64-linux-gnu/libpthread-2.15.so
/lib/x86_64-linux-gnu/libreadline.so.6.2
/lib/x86_64-linux-gnu/libresolv-2.15.so
/lib/x86_64-linux-gnu/librt-2.15.so
/lib/x86_64-linux-gnu/libselinux.so.1
/lib/x86_64-linux-gnu/libssl.so.1.0.0
/lib/x86_64-linux-gnu/libtinfo.so.5.9
/lib/x86_64-linux-gnu/libusb-0.1.so.4.4.4
/lib/x86_64-linux-gnu/libutil-2.15.so
/lib/x86_64-linux-gnu/libz.so.1.2.3.4
/usr/lib/librsync.so.1.0.2
/usr/lib/libsigsegv.so.2.0.2
/usr/lib/python2.7/lib-dynload/datetime.so
/usr/lib/python2.7/lib-dynload/_heapq.so
/usr/lib/python2.7/lib-dynload/_io.so
/usr/lib/python2.7/lib-dynload/pyexpat.so
/usr/lib/python2.7/lib-dynload/resource.so
/usr/lib/python2.7/lib-dynload/termios.so
/usr/local/lib/python2.7/dist-packages/duplicity/_librsync.so

tyl0re commented at 2014-09-04 11:59:

This would do it if you need the phyiscal files, when the names of the links are needed:
#!/bin/bash
FILES=strace -Ff -e open duply mail status 2>&1 1>/dev/null|grep -v '= -1'|grep -i open|grep -v "open resumed" |cut -d \" -f 2|sort -u

for name in $FILES; do
if [ -f $name ] || [ -L $name ]; then
DATEI=readlink -f $name
LIB=file $DATEI|grep "shared object"|cut -d \: -f 1
if [ "x$LIB" != "x" ]; then
echo $name
fi
fi
done

Would do it

Files= (And then the ' is missing in the comment),seems to be cut out

Reiner030 commented at 2014-09-30 14:05:

Hello,

i have found some equal problem on/with:

# lsb_release -d
Description:    Debian GNU/Linux 7.6 (wheezy)
# rear -V
Relax-and-Recover 1.16.1 / Git
# duplicity -V
duplicity 0.6.18
# duply -V
  duply version 1.8.0
  (http://duply.net)

  Using installed duplicity version 0.6.18, python 2.7.3, gpg 1.4.12 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1', bash '4.2.37(1)-release (x86_64-pc-linux-gnu)'.

(duply/duplicity from wheezy backports)

Backup is done fine but restore hung with:

  • "duply missing - Executable & available in path ? (duply)"
    => /usr/bin/env is missing
  • "duplicity version check failed (please report, this is a bug)"
    "Using installed duplicity version (not found), /bin/duply: line 98: python: command not found, gpg 1.4.12 (home: /.gnupg), awk "GNU Awk 4.0.1, bash4.2.3781)-release (x86_64-pc-linux-gnu)'."
    ".... import keys..." ==> python is missing
  • "Temporary file space '/tmp/' from free space is smaller (501 MB)
    than one duplicity volume (1024MB).
    Hint: Free space or change TEMP_DIR setting.
    => perhaps VOLSIZE can be also extracted and used in ISO / tmp image building ?
# aptitude show duplicity |grep -e Package -e Depends -e Recommend -e Suggests
Package: duplicity
Depends: libc6 (>= 2.2.5), librsync1 (>= 0.9.6), python (>= 2.7), python (<
Recommends: rsync, python-paramiko
Suggests: python-boto, ncftp, python-pexpect (>= 2.3-1), python-cloudfiles,

but better install then please all backends ...
I have installed / tested them for our servers (running in different locations with different backup backends) with following packages:

  • duply
  • duplicity
  • python-boto
  • ncftp
  • python-pexpect
  • python-cloudfiles
  • lftp
  • python-gdata
  • tahoe-lafs
  • python-paramiko
  • python-gobject-2

( last two packaes have no references in Ubuntu/Debian yet:
http://www.rfc3092.net/2013/09/missing-modules-for-paramiko-and-gio-in-duplicity-foo/ )

# grep python-cloud /var/log/apt/history.log
Install: python-setuptools:amd64 (0.6.24-1, automatic), python-twisted-mail:amd64 (12.0.0-1, automatic), python-twisted-bin:amd64 (12.0.0-1, automatic), python-foolscap:amd64 (0.6.4-1, automatic), python-openssl:amd64 (0.13-2+deb7u1, automatic), tahoe-lafs:amd64 (1.9.2-1), python-twisted-lore:amd64 (12.0.0-1, automatic), libcrypto++9:amd64 (5.6.1-6, automatic), python-twisted-news:amd64 (12.0.0-1, automatic), python-nevow:amd64 (0.10.0-4, automatic), python-mock:amd64 (0.8.0-3, automatic), ncftp:amd64 (3.2.5-1.1), python-zope.interface:amd64 (3.6.1-3, automatic), python-twisted-conch:amd64 (12.0.0-1, automatic), python-twisted-words:amd64 (12.0.0-1, automatic), python-twisted-web:amd64 (12.0.0-1, automatic), python-twisted:amd64 (12.0.0-1, automatic), duplicity:amd64 (0.6.18-3), python-pycryptopp:amd64 (0.5.29-1, automatic), python-cloudfiles:amd64 (1.7.9.2-1), duply:amd64 (1.5.5.5-1), python-boto:amd64 (2.3.0-1), python-twisted-core:amd64 (12.0.0-1, automatic), python-simplejson:amd64 (2.5.2-1, automatic), python-gnupginterface:amd64 (0.3.2-9.1, automatic), python-twisted-names:amd64 (12.0.0-1, automatic), python-twisted-runner:amd64 (12.0.0-1, automatic), python-pexpect:amd64 (2.4-1), python-gdata:amd64 (2.0.17+dfsg-1), python-pkg-resources:amd64 (0.6.24-1, automatic), python-zfec:amd64 (1.4.5-2, automatic), librsync1:amd64 (0.9.7-9, automatic), python-crypto:amd64 (2.6-4+deb7u3, automatic), python-pyasn1:amd64 (0.1.3-1, automatic), python-paramiko:amd64 (1.7.7.1-3.1), python-gobject-2:amd64 (2.28.6-10)

Volume Size (is set):

# set the size of backup chunks to VOLSIZE MB instead of the default 25MB.
# VOLSIZE must be number of MB's to set the volume size to.
VOLSIZE=1024
DUPL_PARAMS="$DUPL_PARAMS --volsize $VOLSIZE "

Bests

Reiner

Reiner030 commented at 2014-10-02 12:30:

Hi,
the tempdir problem was because I forgot to give my test-VM more RAM/same as the origin instance.
I found out how easy it is to create a debian package from this repo ... thx ;)

With actual snapshot of 1st October and enough RAM I got still following missing files/error (same later for RESTORE):

Using profile '/etc/duply/rear-test'.
Using installed duplicity version (not found), python 2.7.3, gpg 1.4.12 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1', bash '4.2.37(1)-release (x86_64-pc-linux-gnu)'.

--- Start running command STATUS at 11:45:54.879 ---
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512
Traceback (most recent call last):
  File "/sbin/duplicity", line 45, in <module>
    from duplicity import collections
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 29, in <module>
    from duplicity import path
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 34, in <module>
    from duplicity import gpg
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 36, in <module>
    from sha import new as sha1
  File "/usr/lib/python2.7/sha.py", line 10, in <module>
    from hashlib import sha1 as sha
ImportError: cannot import name sha1
11:45:54.989 Task 'STATUS' failed with exit code '1'.

could have something to do with this (i have not nuch experiense with python yet)

# grep import /usr/lib/python2.7/hashlib.py
    >>> import hashlib
            import _sha
            import _md5
            import _sha256
            import _sha512
    import _hashlib
        import logging
# grep -rl -e sha256 -e sha512 -e md5 /usr/lib/python2.7/  | grep lib-dynload
/usr/lib/python2.7/lib-dynload/_hashlib.so

... but this file is already installed... there must be some other file(s) for the hashes...

tyl0re commented at 2014-10-11 13:23:

Hi made an Patch that works for me,
It Uses Strace and File to to find out which Librarys are used
and then add it
https://github.com/tyl0re/rear/blob/master/usr/share/rear/prep/DUPLICITY/default/25_find_all_libs.sh

Mayby Reiner030 can verify if its working for him
just put the FIle in
/usr/share/rear/prep/DUPLICITY/default/25_find_all_libs.sh
and try it again.

Reiner030 commented at 2014-10-13 13:39:

mmh, wasn't helping:

Build process:

...
2014-10-13 12:44:35 Including prep/GNU/Linux/24_include_multipath_tools.sh
2014-10-13 12:44:35 Including prep/DUPLICITY/default/25_find_all_libs.sh
2014-10-13 12:44:35 Including prep/GNU/Linux/28_include_systemd.sh
...

=> package need new package requirement: "strace" and perhaps also "coreutils" and "file" but I think that they are always installed in Debian/Ubuntu.

Restore process with new ISO image::

2014-10-13 12:36:34 Including verify/DUPLICITY/default/25_check_duply_profile.sh
Start duply v1.8.0, time is 2014-10-13 12:36:34.

INFO:

duplicity version check failed (please report, this is a bug)

Using profile '/etc/duply/rear-test'.
Using installed duplicity version (not found), python 2.7.3, gpg 1.4.12 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1', bash '4.2.37(1)-release (x86_64-pc-linux-gnu)'.
...
--- Start running command STATUS at 12:36:34.798 ---
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
  File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512
Traceback (most recent call last):
  File "/sbin/duplicity", line 45, in <module>
    from duplicity import collections
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 29, in <module>
    from duplicity import path
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 34, in <module>
    from duplicity import gpg
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 36, in <module>
    from sha import new as sha1
  File "/usr/lib/python2.7/sha.py", line 10, in <module>
    from hashlib import sha1 as sha
ImportError: cannot import name sha1
12:36:34.920 Task 'STATUS' failed with exit code '1'.

Manually run I got:

# strace -Ff -e open duply rear-test status 2>&1 1>/dev/null|grep -v '= -1'|grep -i open|grep -v "open resumed" |cut -d \" -f 2 |sort -u | grep -i -e md5 -e sha -e hash
/usr/lib/python2.7/dist-packages/Crypto/Hash/hashalgo.py
/usr/lib/python2.7/dist-packages/Crypto/Hash/hashalgo.pyc
/usr/lib/python2.7/dist-packages/Crypto/Hash/HMAC.py
/usr/lib/python2.7/dist-packages/Crypto/Hash/HMAC.pyc
/usr/lib/python2.7/dist-packages/Crypto/Hash/__init__.py
/usr/lib/python2.7/dist-packages/Crypto/Hash/__init__.pyc
/usr/lib/python2.7/dist-packages/Crypto/Hash/MD5.py
/usr/lib/python2.7/dist-packages/Crypto/Hash/MD5.pyc
/usr/lib/python2.7/dist-packages/Crypto/Hash/SHA256.py
/usr/lib/python2.7/dist-packages/Crypto/Hash/SHA256.pyc
/usr/lib/python2.7/dist-packages/Crypto/Hash/SHA.py
/usr/lib/python2.7/dist-packages/Crypto/Hash/SHA.pyc
/usr/lib/python2.7/dist-packages/Crypto/Random/Fortuna/SHAd256.py
/usr/lib/python2.7/dist-packages/Crypto/Random/Fortuna/SHAd256.pyc
/usr/lib/python2.7/hashlib.py
/usr/lib/python2.7/hashlib.pyc
/usr/lib/python2.7/lib-dynload/_hashlib.so
/usr/share/locale/en/LC_MESSAGES/coreutils.mo
/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo
/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo
/usr/share/locale/locale.alias

but backup/restore host have same content there...

Perhaps this is more a backend problem. I let install all (didn't take much space and later I need not take care when using one of he backends). Here I use/need the ssh backend:

Import of duplicity.backends.cloudfilesbackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.u1backend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
ssh: Connected (version 2.0, client OpenSSH_5.5p1)
ssh: Authentication (publickey) successful!
ssh: Secsh channel 1 opened.
ssh: [chan 1] Opened sftp connection (server version 3)
Main action: collection-status

... yes ... seems so .. => next comment ;)

Reiner030 commented at 2014-10-13 14:06:

a little to fast... but I think I have found the problem...

if I call

# python /usr/lib/python2.7/sha.py

on backup host it works fine but on restore host it throws above error messages.

=> here the diff of my duplicity call:

# diff libs_backup.txt libs_restore.txt
1a2
> /etc/localtime
10d10
< /usr/lib/locale/locale-archive
14a15,18
> /usr/lib/python2.7/atexit.py
> /usr/lib/python2.7/atexit.pyc
> /usr/lib/python2.7/bisect.py
> /usr/lib/python2.7/bisect.pyc
16a21,22
> /usr/lib/python2.7/collections.py
> /usr/lib/python2.7/collections.pyc
26a33,34
> /usr/lib/python2.7/encodings/ascii.py
> /usr/lib/python2.7/encodings/ascii.pyc
29,30d36
< /usr/lib/python2.7/encodings/utf_8.py
< /usr/lib/python2.7/encodings/utf_8.pyc
34a41,44
> /usr/lib/python2.7/heapq.py
> /usr/lib/python2.7/heapq.pyc
> /usr/lib/python2.7/keyword.py
> /usr/lib/python2.7/keyword.pyc
37a48,49
> /usr/lib/python2.7/logging/__init__.py
> /usr/lib/python2.7/logging/__init__.pyc
62a75,76
> /usr/lib/python2.7/threading.py
> /usr/lib/python2.7/threading.pyc
70a85,86
> /usr/lib/python2.7/weakref.py
> /usr/lib/python2.7/weakref.pyc
73,75d88
< /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
< /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
< /usr/local/lib/python2.7/dist-packages

=> minimum this seems the cause?
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0

=> yes... new failure message:

Traceback (most recent call last):
  File "/bin/duplicity", line 45, in <module>
    from duplicity import collections
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 29, in <module>
    from duplicity import path
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 36, in <module>
    from duplicity import librsync
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 29, in <module>
    import _librsync
ImportError: librsync.so.1: cannot open shared object file: No such file or directory

=> /usr/lib/x86_64-linux-gnu/librsync.so.1.0.2

... and creating symlinks for these 3 libs with *.so and *.so.1

=> works then ;)

Start duply v1.8.0, time is 2014-10-13 14:22:27.
Using profile '/etc/duply/rear-test'.
Using installed duplicity version 0.6.18, python 2.7.3, gpg 1.4.12 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1', bash '4.2.37(1)-release (x86_64-pc-linux-gnu)'.
...
Cleanup - Delete '/tmp/duply.3054.1413210147_*'(OK)

--- Start running command RESTORE at 14:22:28.042 ---
Using archive dir: /var/cache/duplicity/duply_rear-test
Using backup name: duply_rear-test
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Failed: libgobject-2.0.so.0: cannot open shared object file: No such file or directory
Import of duplicity.backends.u1backend Succeeded
...

(the complained lib is in an not used backend... )

tyl0re commented at 2014-10-13 18:39:

Hi Seems Git add Link + Libarys..
I changed it to add the link to $Library instead of die library. So Rear add both Links and librarys
Can you Test it again? (the new Version)

Reiner030 commented at 2014-10-13 22:26:

mmh, sadly there is now something other missing...
When I tried it with your script I got now a backend error:

Start duply v1.8.0, time is 2014-10-13 20:59:44.
Using profile '/etc/duply/rear-test'.
Using installed duplicity version 0.6.18, python 2.7.3, gpg 1.4.12 (Home: ~/.gnupg), awk 'GNU Awk 4.0.1', bash '4.2.37(1)-release (x86_64-pc-linux-gnu)'.
Encryption public key 'ECBEFE03' not found.
[...]
--- Start running command STATUS at 20:59:45.027 ---
Using archive dir: /var/cache/duplicity/duply_rear-test
Using backup name: duply_rear-test
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.u1backend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.ftpsbackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Import of duplicity.backends.cloudfilesbackend Succeeded
ssh: Connected (version 2.0, client OpenSSH_5.5p1)
Using temporary directory /tmp/duplicity-csYsQ0-tempdir
Backend error detail: Traceback (most recent call last):
  File "/sbin/duplicity", line 1404, in <module>
    with_tempdir(main)
  File "/sbin/duplicity", line 1397, in with_tempdir
    fn()
  File "/sbin/duplicity", line 1248, in main
    action = commandline.ProcessCommandLine(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 999, in ProcessCommandLine
    globals.backend = backend.get_backend(args[0])
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 158, in get_backend
    return _backends[pu.scheme](pu)
  File "/usr/lib/python2.7/dist-packages/duplicity/backends/sshbackend.py", line 140, in __init__
    raise BackendException("ssh connection to %s:%d failed: %s" % (parsed_url.hostname,portnumber,e))
BackendException: ssh connection to 192.168.10.41:22 failed: Unknown server 192.168.10.41

BackendException: ssh connection to 192.168.10.41:22 failed: Unknown server 192.168.10.41
20:59:45.455 Task 'STATUS' failed with exit code '23'.

I checked dns resolution / ssh connect and they are working normal. (ssh 192.168.10.41) so it must be a missing.library problem of the new ssh backend of duplicity now...
I check tomorrow strace (downgraded to last version to verify that it still works with manual copying of

/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
/usr/lib/x86_64-linux-gnu/librsync.so.1*

tyl0re commented at 2014-10-14 06:03:

The Old Version has had an bug i misswrote straces instead strace , so it havent done anythingsince check for binaries hasnt worked.

I think a Library is now added using, somthing that wasnt used before, where somthing is missing

Mayby some Python Modules are missing. I Added now all Script Strace finds, mayby you can try it again.

If its not working
Does "duply rear-test status" work?

What does
strace -Ff -e open duply rear-test status|grep -v '= -1'|grep -i open|grep -v "open resumed"
Shows on your machine you want to restore?

Reiner030 commented at 2014-10-14 16:44:

sorry, with last modification from today the script already stuck when initialzing network:

...
Running 62-routing.sh...
* * * Rescue System is ready * * *
/sbin/rear: line 131:/usr/share/rear/conf/default.conf: No such file or Directory
/sbin/rear: line 138: has_binary: command not found
ERROR: Required program 'pidof' missing, please check your PATH

I added 2>&1 redirection ;)

First with non-working version:

# strace -Ff -e open duply rear-test status 2>&1 |grep -v '= -1'|grep -i open|grep -v "open resumed" > strace.out

https://gist.github.com/Reiner030/2e5c54994b54e11d22a2#file-strace-output-rear-duply-test

Here the deduplicated file listing of failed openings:
https://gist.github.com/Reiner030/2e5c54994b54e11d22a2#file-gistfile1-txt

Then I copied these files manually to restore system (ready for all backends):

# scp /usr/lib/x86_64-linux-gnu/{libcrypto.so.1.0.0,libssl.so.1.0.0,librsync.so.1*,libgobject-2.0.so.0*,libgthread-2.0.so.0,libffi.so.5,libgio-2.0.so.0,libgmodule-2.0.so.0}* /usr/lib/libpyglib-2.0-python2.7.so.0* 192.168.10.216:/usr/lib/x86_64-linux-gnu/

and here duply status works fine.
Sorted unique output gave more then double of files:

RESCUE rear-test:~ # sed -e 's/^.*open("\([^"]*\)".*/\1/' strace.out | sort -u | wc -l
217
RESCUE rear-test:~ # sed -e 's/^.*open("\([^"]*\)".*/\1/' strace-working.out | sort -u | wc -l
503

https://gist.github.com/Reiner030/05ddefb1a5e5502536ea

And following file types:

# sed -e 's/^.*open("\([^"]*\)".*/\1/' strace-working.out | xargs file | sed -e "s/.*: *//" -e "s/\`.*'//" | sort | uniq -c 
/etc/magic, 4: Warning: using regular magic file `/usr/share/misc/magic'
      3 ' (No such file or directory)
    239 ASCII text
      5 ASCII text, with very long lines
      4 ISO-8859 text
     53 cannot open  (No such file or directory)
     60 character special
    592 data
      5 directory
     41 empty
      2 sticky directory
    462 symbolic link to

=> there is no filetype "script" mentioned... because there is no "#!/usr/bin/python" in first line ...
only ASCII text ...
But on origin host its "C source, ASCII text" / "C++ source, ASCII text":

# strace -Ff -e open duply zzz-puppet-optimus status 2>&1 |grep -v '= -1'|grep -i open|grep -v "open resumed" | sed -e 's/^.*open("\([^"]*\)".*/\1/' | xargs file | sed -e "s/.*: *//" -e "s/\`.*'//" | sort | uniq -c
     30 ASCII text
      5 ASCII text, with very long lines
      3 a /usr/bin/env bash script, ASCII text executable
     29 cannot open  (No such file or directory)
     56 character special
      4 C source, ASCII text
     10 C++ source, ASCII text
    158 data
     12 directory
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x086c854394888000ac3a2de5452f442db5e83ac5, stripped
      5 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x18b180f9d8f80887185dc3b54caf44541eaa39b9, stripped
      4 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x2eb5da4d615f5d47c1e41ee093e49269a007b196, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x40af5432efd1777fe16c160143782765a19e3038, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x49545971b62531aa2029267ebb86fdb6e5e39eb9, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x518025bf8956ab32da0ec4a3f640142fb399c5dd, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x65d0d64da20cd3170575f6b6a1a8815aeca7737f, stripped
      4 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x6c202a7ac89bfdf39da30ecd37df34576caa086d, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x71c10e114e06436e3e50505e78a2976768abb6c2, stripped
      4 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x9055c19c826f2b79ce2abb2da8b96cf67507aa9f, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x9413df9bed16b368c276019fed1fe1247042a4e8, stripped
      4 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0x9bd7b6a8658c422f451d0c7dcaa51d29b2ed404a, stripped
      4 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xabc2e4de6529d462a0d2565ddf51810088afb6da, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xb272d3a368a705aee38630dba8ec4bc54ffa7774, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xba7ef8fb0542ba4c1b33beef905d0dffdf68b293, stripped
      4 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xc65cb89065ae412c11a36e57145f205631fd049b, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xc81c9169d680b7c1d350329845ffcb7d32ca5fae, stripped
      8 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xcf2570488221e59ef684033986dd971dc2476737, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xd1bf89062b0106793f300596b32c0bcfffd018be, stripped
      2 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xd6ef69ffdafa45dd4aed4fab13a91e7148e6b90b, stripped
      4 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xd8ff7e88c133aebce82d00594d7c6a225a2980ac, stripped
      4 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xee4b6b3f778ee8ab58739c57733e09b17c61f64f, stripped
     37 empty
     15 GPG key public ring
      5 GPG key trust database version 3
      3 ' (No such file or directory)
     75 PDP-11 separate I&D executable not stripped
      1 PEM RSA private key
    365 python 2.7 byte-compiled
    188 Python script, ASCII text executable
      4 Python script, ISO-8859 text executable
      4 setgid directory
      2 sticky directory
    495 symbolic link to 
     18 timezone data, version 2, 1 gmt time flag, 1 std time flag, no leap seconds, no transition times, 1 abbreviation char

tyl0re commented at 2014-10-14 18:48:

You forgot the readlink
cat strace.out|grep -v '= -1'|grep -i open|grep -v "open resumed" | sed -e 's/^.open("([^"])"./\1/' | xargs -n 1 readlink -f |sort -u|xargs file | sed -e "s/.: //" -e "s/`.'//" | sort | uniq -c
Should show you the right Type of the files

Can you do the strace for the Original System ?
May by something is missing except Library

What does "file /usr/lib/python2.7/textwrap.py" say?

Reiner030 commented at 2014-10-14 19:31:

ok, I used only copy&paste from

What does
strace -Ff -e open duply rear-test status|grep -v '= -1'|grep -i open|grep -v "open resumed"
Shows on your machine you want to restore?

here from origin system:

# strace -Ff -e open duply rear-test status 2>&1 |grep -v '= -1'|grep -i open|grep -v "open resumed" | sed -e 's/^.*open("\([^"]*\)".*/\1/' | xargs -n 1 readlink -f |sort -u | xargs file  -e elf | sed -e "s/.*: *//" -e "s/\`.*'//" | sort | uniq -c | sort -n
      1 a /usr/bin/env bash script, ASCII text executable
      1 C source, ASCII text
      1 GPG key public ring
      1 GPG key trust database version 3
      1 ' (No such file or directory)
      1 PDP-11 separate I&D executable not stripped
      1 PEM RSA private key
      1 POSIX shell script, ASCII text executable
      1 setgid directory
      1 sticky directory
      1 timezone data, version 2, 1 gmt time flag, 1 std time flag, no leap seconds, no transition times, 1 abbreviation char
      1 UTF-8 Unicode text
      2 ASCII text, with very long lines
      2 Python script, ISO-8859 text executable
      3 character special
      4 data
      4 directory
      4 empty
     16 ASCII text
     20 C++ source, ASCII text
     51 ELF 64-bit LSB shared object, x86-64, version 1 (SYSV)
     56 cannot open  (No such file or directory)
    170 Python script, ASCII text executable
    198 python 2.7 byte-compiled

ah and:

# file /usr/lib/python2.7/textwrap.py
/usr/lib/python2.7/textwrap.py: Python script, ASCII text executable

tyl0re commented at 2014-10-14 21:20:

Found another Error,.... Its the first time using arrays in shell scripts.....
Can you try again.
Sorry i cant try it on my System it works....

Reiner030 commented at 2014-10-16 18:01:

Hi,
no problem... I have only the disadvantage that i must every time manually run VMware Sphere Client to start VMs and get into the boot menu ...

=> I have taken a longer step and created with Virtualbox and Vagrant a nice testing setup with a found ready-to-use image for it. The config/scripts I have prepared so that they are basically usable for multiple platforms/backend for testing.
https://gist.github.com/Reiner030/4b602289320f3c2ed4e7
Perhaps it make sense to offer/put it in a better way into the repository for testing purposes ?

You need

  • VirtualBox (www.virtualbox.org)
  • Vagrant (www.vagrantup.com)
  • your ssh pubkey line
  • one duply ssh backend private ssh key
  • one "remote" ssh server which let login duply for backing up.

My Test VM created

  • 70 MB ISO image and
  • 1,2 GB Backup file (because of much needed packages caused by doc generation in
    your repository => setup with $ADD_PACKAGES in script)

The installer process have some more problems because of missing Virtualbox drivers
(=> perhaps another task) but the duply error message is the same as on my system:

BackendException: ssh connection to 192.168.10.41:22 failed: Unknown server 192.168.10.41

  • 1st machine is "backup" which setup duply, rear and made the rear backup.
  • 2nd machine "recover" starts with gui mode on so that you can manually test the behavior.
    Sadly there seems no option to not provisioning the recover instance except calling vagrant directly by vagrant up recover --no-provision

Comments howto provision again and login into VMs within ssh are inside the script/Vagrantfile.

In the begin of script the git repository and debian mirror and duply backends modifications can be setup. If the test vm is of interest there could be added further rear backends, too.

Reiner030 commented at 2014-10-16 18:21:

ah, forgotten because for Duply/Duplicity with ssh backend not needed:
Duply has a fetch/restore problem with AWS... which are fixed in newer versions:
https://packages.debian.org/jessie/duply
https://packages.debian.org/wheezy-backports/duplicity

In my problematic testsystem I had already installed duply (1.8.0-1) with "default Wheezy" duplicity.
But the Test VM with duply (1.5.5.5-1) has same problems.

Reiner030 commented at 2014-10-16 19:09:

When testing rear/rear.git I found the solution for the "Unknown server" error...
=> /root/.ssh/known_hosts was not setup / here: not copied to ISO image ?

gdha commented at 2014-10-16 19:39:

interesting way of automating testing. Why don't you write an article about
it and we will be honoured to link to it from our rear pages.
Gratien

On Thu, Oct 16, 2014 at 9:09 PM, Reiner notifications@github.com wrote:

When testing rear/rear.git I found the solution for the "Unknown server"
error...
=> /root/.ssh/known_hosts was not setup / here: not copied to ISO image ?

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/issues/426#issuecomment-59413694.

Reiner030 commented at 2014-10-16 21:05:

Our PHP developer uses this way of local dev stage website testing (former with puppet and now with salt). The nice thing is that you can use testing parallel - you need only different folders for each repository/branch you want to test.

The above error message "Unknown server" is also in the main repository; so I made a manual workaround for it ... and works now fine... Thanks. ;)

(except the grub/initramfs problem of https://github.com/rear/rear/issues/473)

tyl0re commented at 2014-10-17 08:19:

Do you have used the newst Version of my script ?, Sould i make a pull request (When it was working)?

Mayby sou should try
CLONE_USERS=( root)
in the config

gdha commented at 2014-10-17 08:41:

I think if you would add in script ./prep/DUPLICITY/default/05_prep_duplicity.sh under COPY_AS_IS the /root/.ssh that would have all the required keys (maybe to much)?

tyl0re commented at 2014-10-17 09:10:

it could be to much in some siuations. I Use http Upload for the iso. So the keys would be tranferes unencrypted then,since they are also in /root/.ssh . Its no Problem for me, but i think there could be an issue for someone

Reiner030 commented at 2014-10-17 21:32:

@tyl0re yes I checked out your repository for my checks, so it works now nice and can be committed.

@tyl0re / @gdha The user/ssh keys I need for provisioning the restore VM is the vagrant user.
I would try testing the CLONE_USERS=(vagrant) then for this setup.

But my provisioning script still needs external private duply key/public ssh line for user/known_hosts for duply because without these files I can't create the backup itself in the testing machine ;)

Reiner030 commented at 2014-10-20 23:19:

ok, with merge this task is completed - thanks (I think only @tyl0re can close this issue).
I would open another issue for the possible Virtualbox / Vagrant integration.

[Export of Github issue for rear/rear.]