#483 Issue closed: drbd problem¶
Labels: enhancement, bug, fixed / solved / done
tbsky opened issue at 2014-10-18 15:35:¶
hi:
I use drbd on lvm at scientific linux 6.5 x86_64. drbd rpm package from
ELRepo.
under linux, the same lvm logic volume can be accessed by several path.
so "/dev/rootvg/my-lv" and "/dev/mapper/rootvg-my--lv" are the same
thing. rear may confused about this when analyze the disk layout.
for example, I have a drbd locate on /dev/rootvg/kvm-wiki. rear analyzed
result below:
disklayout.conf:
lvmvol /dev/rootvg kvm-wiki 2048 16777216
drbd /dev/drbd9 wiki /dev/rootvg/kvm-wiki
diskdeps.conf:
/dev/mapper/rootvg-kvm--wiki /dev/rootvg
/dev/drbd9 /dev/rootvg/kvm-wiki
disktodo.conf:
todo /dev/mapper/rootvg-kvm--wiki lvmvol
todo /dev/drbd9 drbd
rear assume "/dev/drbd9" is depend on "/dev/rootvg/kvm-wik", but it only knows about "/dev/mapper/rootvg-kvm--wiki" when re-create layout. so "/dev/drbd9" can not satisfy dependency when doing "rear recover".
I can not find a simple and proper way to fix rear code. I think maybe rear should analyze the soft links of the device, since the same problem may happened on anything above lvm, not just drbd.
the workarround is to make sure drbd configuration use the correct format that rear will use. in my case, I need to modify the drbd configuration, use "/dev/mapper/rootvg-kvm--wiki" instead of "/dev/rootvg/kvm-wiki". "rear recover" will then satisfied about the dependency and re-create drbd.
there is one thing to note about drbd >= 8.4.5, the drbd-utils are now an independent package. so we need more tools when re-create drbd. in my case, "drbdadm" will call "drbdadm-84" and "drbdsetup-84". so I need to add these two programs to rear "$PROGS". or the recover will fail.
tbsky commented at 2014-10-19 06:39:¶
hi:
I got more problems when dealing with drbd with rear. I am still
fighting with them. the first problem is critical. I have drbd device
"drbd1" and "drbd10", rear use code below which will can not distinguish
them so cause many problems:
read drbd disk resource device junk < <(grep "^drbd $1" "$LAYOUT_FILE")
I find the same bug are shared by many other scripts, .like "12_include_raid_code.sh", "16_include_luks_code.sh" and more. but "10_include_partition_code.sh" is fine. so the bug maybe introduced at rear later versions.
second and third problem logs like below:
+++ Print 'Creating DRBD resource bot-ubuntu-12-04'
+++ test 1
+++ echo -e 'Creating DRBD resource bot-ubuntu-12-04'
+++ dd if=/dev/zero of=/dev/mapper/rootvg-kvm--bot--ubuntu--12--04 bs=1M count=20
20+0 records in
20+0 records out
20971520 bytes (21 MB) copied, 0.0918644 s, 228 MB/s
+++ sync
+++ drbdadm create-md bot-ubuntu-12-04
You want me to create a v08 style flexible-size internal meta data block.
There appears to be a v08 flexible-size internal meta data block
already in place on /dev/rootvg/kvm-bot-ubuntu-12-04 at byte offset 34359734272
NOT initializing bitmap
+++ drbdadm attach bot-ubuntu-12-04
2: Failure: (127) Device minor not allocated
additional info from kernel:
unknown minor
Command 'drbdsetup-84 attach 2 /dev/rootvg/kvm-bot-ubuntu-12-04 /dev/rootvg/kvm-bot-ubuntu-12-0
4 internal --fencing=resource-and-stonith --al-extents=3389 --disk-flushes=no --disk-barrier=no
' terminated with exit code 10
2014-10-19 19:19:11 An error occurred during layout recreation.
rear use "dd" to try to clean existing drbd metadata. but drbd meta is at the end of device, not at the begining. another problem is "drbdadm attach" failed. but I am not familiar with this command and I don't need it, so I will not trace the problem further for now
first and sceond problem patch below (not including third problem "drbdadm attach" fix):
--- 15_include_drbd_code.sh.orig 2014-05-12 14:37:21.000000000 +0800
+++ 15_include_drbd_code.sh 2014-10-19 14:14:50.019529211 +0800
@@ -3,7 +3,7 @@
# This requires DRBD configuration present!
create_drbd() {
local drbd disk resource device junk
- read drbd disk resource device junk < <(grep "^drbd $1" "$LAYOUT_FILE")
+ read drbd disk resource device junk < <(grep "^drbd $1 " "$LAYOUT_FILE")
cat >> "$LAYOUT_CODE" <<EOF
if [ ! -e /proc/drbd ] ; then
@@ -13,9 +13,7 @@
mkdir -p /var/lib/drbd
LogPrint "Creating DRBD resource $resource"
-dd if=/dev/zero of=$device bs=1M count=20
-sync
-drbdadm create-md $resource
+drbdadm -- --force create-md $resource
EOF
the biggest problem I am facing now is rear can not recognize drbd resource which has more then one device. rear assumes every drbd resource has only one drbd device. it seems a fundamental issue. I will try to fxi it or rear can not be used in our environment. that would be sad..
tbsky commented at 2014-10-19 11:34:¶
hi:
I have workaround the multiple volumes in one drbd resource problem.
it's not a proper fix. the drbd volumes will be "create-md" multiple
times when "rear recover". and "drbdadm primary" or "drbdadm attach" is
not happy when running multiple times with same resource. but I don't
need them so finally rear is working for me. and since "rear recover" is
not a ervery-day job, so I can live with it.
I also found that in my environment it's a little hard to use
"/dev/mapper/*" for drbd configuration. so I also workaround rear code
to produce "/dev/mapper/vg-lv" layout from "/dev/vg/lv" layout.
workarround code below. it's not a proper patch, just workaround for
current drbd problem.
--- 25_drbd_layout.sh.orig 2012-02-23 23:57:55.000000000 +0800
+++ 25_drbd_layout.sh 2014-10-19 18:23:34.705786559 +0800
@@ -4,9 +4,26 @@
Log "Saving DRBD configuration."
for resource in $(drbdadm sh-resources) ; do
- dev=$(drbdadm sh-dev $resource)
- disk=$(drbdadm sh-ll-dev $resource)
+ dev=( $(drbdadm sh-dev $resource) )
+ disk=( $(drbdadm sh-ll-dev $resource) )
- echo "drbd $dev $resource $disk" >> $DISKLAYOUT_FILE
+ for i in ${!dev[*]}; do
+ vol_dev=${dev[$i]}
+ vol_disk=${disk[$i]}
+
+ # map disk to rear way (/dev/mapper)
+ if drbd_dm=`readlink $vol_disk`; then
+ for dm in /dev/mapper/*; do
+ if lvm_dm=`readlink $dm`; then
+ if [ "$drbd_dm" = "$lvm_dm" ]; then
+ vol_disk=$dm
+ break
+ fi
+ fi
+ done
+ fi
+
+ echo "drbd $vol_dev $resource $vol_disk" >> $DISKLAYOUT_FILE
+ done
done
fi
gdha commented at 2014-10-21 11:45:¶
@tbsky a pull request would be nice.
@jhoekx Jeroen, what is your feedback on the proposed solution?
jhoekx commented at 2014-10-21 12:12:¶
I don't have any drbd systems running, so I can't test easily. As long as the old functionality is not broken and this solves a problem, I'm happy :-)
For translating the names, did you try the get_device_name function?
That one should be able to receive any kind of device name and produce
the name used everywhere in Rear.
I don't know why we didn't use the --force option to create-md. Do you happend to know if it always existed? Otherwise we might have to have the two options and enable --force inside a feature flag...
tbsky commented at 2014-10-21 17:42:¶
hi jhoekx:
I will check "get_device_name" and see what to do this weekend.
"--force" is not a create-md option, it will be passed to underlying
sub-program (drbdmeta) by create-md, so the usage is a little tricky.
you won't see that option when you "man drbdadm". you need to "man
drbdmeta" to know it. so maybe you guys missed that.
about the drbd version, I didn't know exactly what feature comes with the what version. I am using latest version of drbd84, and I can take a look at latest version of drbd83, but I am afraid I can not check other earlier versions.
tbsky commented at 2014-10-24 15:56:¶
hi jhoekx:
just check the function "get_device_name". I think the idea is great,
but unfortunately that function is too simple and seems only cover two
case: cciss and dm-X. it didn't cover lvm naming like /dev/vg/lv.
I am a little confused. since I think the naming scheme /dev/vg/lv is
popular, I think anything above that will be broken, not just drbd?
tbsky commented at 2014-10-24 16:43:¶
hi jhoekx:
I tried to use /dev/vg/lv to create filesystem and swap and put then at
/etc/fstab. but rear use other ways to detect these without fstab, so
they are fine. so far I can see only drbd is affected. but I think the
correct way should be fix get_device_name so it can handle more cases.
I will try to do that..
tbsky commented at 2014-10-27 15:45:¶
hi:
I fixed the code, so most of them can work correctly both under drbd 8.3
and 8.4. it can also support multiple volumes of drbd 8.4. there is one
strange issue about "drbdadm attach" which works under 8.3, but failed
under 8.4.4 and 8.4.5. I think it is a drbdadm bug. if I run "drbdadm -d
attach xxxx" under 8.4.5, it failed because it forgot to create resource
and device first. under 8.3.16 everything is ok. we can workarround this
but I don't know if it is necessary. maybe 8.4.6 will fix this bug if
report upstream.
gdha commented at 2014-10-30 15:10:¶
committed 3 pull requests of @tbsky
waiting on feedback to see whether all issues are resolved?
tbsky commented at 2014-10-30 16:06:¶
hi gdha:
yes all my drbd issues are resolved, since I don't need "drbdadm
attach". I don't know what kind of situation of "drbdadm attach" will be
used. since under that state drbd can not be accessed(rear only try
attach it but not make it primary), maybe jhoekx knows better.
I already write drbd email list about "drbdadm attach" error under 8.4.5. but didn't get response yet. I will try to write again.
tbsky commented at 2014-11-04 06:04:¶
hi:
drbdadm upstream reply that 8.4 "drbdadm attach" not working is a
feature, not bug. so we need to tune the code. reply at
http://lists.linbit.com/pipermail/drbd-user/2014-November/021686.html
I will try to test what command can make both 8.3/8.4 happy..
[Export of Github issue for rear/rear.]