#502 PR `closed`: add support for modular custom function cf--$stage--my_function¶

tbsky opened issue at 2014-11-02 05:19:¶

hi:
I need to add a small custom script to rear. thanks for rear modular design, I can easily add my script to something like "/usr/share/rear/backup/NETFS/default/66_my_backup.sh" and "/usr/share/rear/restore/NETFS/default/66_my_restore.sh" .

now I want to further enhance the modular design, try to put all my scripts to /etc/rear/local.conf so it is easier to maintain and separate my scripts from official rear system.

I try to use a "custom function" idea, so I can add my script to /etc/rear/local.conf and rear will automatically find them and include them. so my /etc/rear/local.conf would look like below:

function cf--backup--66_make_drbd_vm_backup {
/share/bin/drbd-vm-backup -b -t drbd-single -d $TMP_DIR/isofs/drbd-vm-backup
}
function cf--restore--66_restore_drbd_vm_backup {
drbd-vm-backup -r -t drbd-single -d $BUILD_DIR/outputfs/drbd-vm-backup
}

with the patch I can enjoy rear modular design and maintain all the custom change to /etc/rear/local.conf. I try this in my system and is seems work well. but I don't really understand all the rear internals.so please review and fix any non-proper codes.

schlomo commented at 2014-11-02 09:46:¶

Hi @tbsky,

this is a really interesting idea and I would like to help you to achieve your goal, though I don't feel well with this implementation because this implementation sufferes IMHO from the following problems:

No separation between code and config (Yes, I know you want to have that, but how would you write a unit test for it?)
The whole topic of SIMULATE is now spread over Source() and SourceStage() where before it was just in Source().
Sorting custom functions into the list of scripts uses a different method than sorting scripts, this might lead to future troubles
This code only plugs scripts into stages but not into different OUTPUT, BACKUP and OS* dimensions. It would be nice to find a solution that would manage that as well with the same code.

If to follow this path then I would suggest that you add the functions to the scripts array so that they will be sorted together with the real scripts. Then you would extend the Source function to detect if it is given a script or a function and do the right thing.

Can you please help me to understand why you must maintain your extra code in configuration? Would you do the same with extra Apache stuff or would you plug that into the appropriate subdir in /etc/httpd?

Can it be that your problem is more a problem of deployment than a problem of having it all in the same file? Maybe you can share with us how you deploy ReaR? How do you deploy the ReaR configuration?

Kind Regards,
Schlomo

tbsky commented at 2014-11-02 15:53:¶

hi schlomo:
thanks a lot for your information! I am sorry I am not a native English speaker. so I can only try my best to describe my thought :)

my environment:
hard disks -> mdadm raid -> lvm -> drbd -> libvirtd vm (I am using drbd device as kvm raw disk). this is a cluster node, so cluster data is synced at drbd level and vm is controlled by cluster software. the real application which running user services is at the vm level, so I just backup the vm data in the past. about the host environment, it was just too complicated to backup,so I gave up, until I met rear.

rear can backup the host os and layout of mdadm,lvm,drbd and restore all these correctly. that is amazing! now only the vm left. I can just skip that, since this part are really hard to handle correctly. if you analyze the content of vm, there maybe windows vm with ntfs or linux vm with lvm+ext4fs.

my thought:
rear won't handle every situation, so it leaves some options for the users. in my case, I need to define
backup method and some include/exclude parameters. then if I want to backup the vm in rear, I need to write my own custom script.

in my feeling, the script which go to /usr/share/rear should be official/general usage script. there should be some other place to put the custom/specific script. but currently I saw rear only offer pre-restore/post-restore parameter. so if you need to hook other parts of rear, you need put your code the the "official" /usr/share/rear place, which in my feeling that place belongs to developer, not to end user.

take an example for my thought. our company also use another software "Freepbx" which is a pbx software. it will call custom setup/script everywhere, so you can inject your custom setup and code into the system. it even provide you config files so you can "overwrite" some of the official code. and in fact I did need that feature to overwrite the official code for some special case. and I can do it without touch the "Freepbx" official code.

so I think if rear can have some custom hook is a good thing. the patch try to do minimal changes to get the feature and maintain compatibility. I think the official way maybe need to think about where end-users can hook their code, like "pre-restore" or "post-restore". but the amazing part is in current rear design, you can almost hook your code into every place, just there is no official way. you need to code your custom script as the official scripts.

I also think the custom code is not necessary to take the same treatment as official code. there maybe no need about "function debug" or "test unit". since it's the code written by end user for their specific needs. in my case, the custom "script" is an one liner which calls the real script I wrote for backup/restore the vm (my script is only doing dd now) . as you can see in my example local.conf.

maybe my need is special. I can already archive my need in current rear design. but maintain everything in one config file is a benefit to me. so I don't need to care the location of my custom scripts which mix with official rear scripts. I don't really concern about mix code and configuration in one file, since rear config file is actually bash code.

schlomo commented at 2014-11-02 19:08:¶

So how do you deploy ReaR and that local.conf?

Actually the "official" way to extend ReaR is exactly as you found out:
Simply drop your additions into the appropriate ReaR subdirectories. Thats
why it is so open and modular, so that people like you can easily add their
own stuff without any need to modify files from ReaR.

If you use packages like RPM or DEB there also can't be any confusion
between ReaR files (from rear package) and your own addons.

Your setup sound really impressive and I still think that the answer to
your question lies in the deployment and configuration management.

However, if it makes you happy: Why not send us a patch to add a
PRE_RESTORE and POST_RESTORE or PRE_BACKUP and POST_BACKUP variable,
whatever you need right now? Similar to the POST_RECOVERY_SCRIPT variable.
You could simply clone that.

Sometimes beeing more specific is actually simpler than beeing more
generic. I am not sure if our users would understand this concept of
injecting specially named functions.

On 2 November 2014 16:53, tbsky notifications@github.com wrote:

hi schlomo:
thanks a lot for your information! I am sorry I am not a native English
speaker. so I can only try my best to describe my thought :)

my environment:
hard disks -> mdadm raid -> lvm -> drbd -> libvirtd vm (I am using drbd
device as kvm raw disk). this is a cluster node, so cluster data is synced
at drbd level and vm is controlled by cluster software. the real
application which running user services is at the vm level, so I just
backup the vm data in the past. about the host environment, it was just too
complicated to backup,so I gave up, until I met rear.

rear can backup the host os and layout of mdadm,lvm,drbd and restore all
these correctly. that is amazing! now only the vm left. I can just skip
that, since this part are really hard to handle correctly. if you analyze
the content of vm, there maybe windows vm with ntfs or linux vm with
lvm+ext4fs.

my thought:
rear won't handle every situation, so it leaves some options for the
users. in my case, I need to define
backup method and some include/exclude parameters. then if I want to
backup the vm in rear, I need to write my own custom script.

in my feeling, the script which go to /usr/share/rear should be official/general usage script. there should be some other place to put the custom/specific script. but currently I saw rear only offer pre-restore/post-restore parameter. so if you need to hook other parts of rear, you need put your code the the "official" /usr/share/rear place, which in my feeling that place belongs to developer, not to end user.

take an example for my thought. our company also use another software
"Freepbx" which is a pbx software. it will call custom setup/script
everywhere, so you can inject your custom setup and code into the system.
it even provide you config files so you can "overwrite" some of the
official code. and in fact I did need that feature to overwrite the
official code for some special case. and I can do it without touch the
"Freepbx" official code.

so I think if rear can have some custom hook is a good thing. the patch
try to do minimal changes to get the feature and maintain compatibility. I
think the official way maybe need to think about where end-users can hook
their code, like "pre-restore" or "post-restore". but the amazing part is
in current rear design, you can almost hook your code into every place,
just there is no official way. you need to code your custom script as the
official scripts.

I also think the custom code is not necessary to take the same treatment
as official code. there maybe no need about "function debug" or "test
unit". since it's the code written by end user for their specific needs. in
my case, the custom "script" is an one liner which calls the real script I
wrote for backup/restore the vm (my script is only doing dd now) . as you
can see in my example local.conf.

maybe my need is special. I can already archive my need in current rear design. but maintain everything in one config file is a benefit to me. so I don't need to care the location of my custom scripts which mix with official rear scripts. I don't really concern about mix code and configuration in one file, since rear config file is actually bash code.

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/pull/502#issuecomment-61411745.

tbsky commented at 2014-11-03 04:22:¶

hi schlomo:
thanks a lot for your confirm. I am just providing an idea to inject end-user code. as you said "drop your additions into the appropriate ReaR subdirectories" is the official way. then I will just stay with it. as you said, there is no such difference.

the "PRE_BACKUP" or "POST_BACKUP" won't help a lot, since code may need to inject at some specific stage. but maybe a "PRE" and "POST" variable for every "official" script? so if I need to inject some code at some stage, I can just use that variable to inject my code. "Freepbx" use that method. I don't know if it will be considered too lousy.

about " how do you deploy ReaR and that local.conf". I am using ScientificLinux(RHEL) 6.5 and rear 1.6.1 comes with EPEL (but now using git master) . i just edit /etc/rear/local.conf and do "rear mkbackup". my complete /etc/rear/local.conf with custom function below:

BACKUP=NETFS
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/share/cluster/iso/*' '/share/rear-tmp/*')
PROGS=( "${PROGS[@]}" '/lib/drbd/drbdadm-84' '/lib/drbd/drbdsetup-84' 'pv' '/share/bin/drbd-vm-backup')
ISO_MAX_SIZE=25000
BACKUP_URL=iso://backup/
OUTPUT=ISO
OUTPUT_URL=null
ISO_DIR=/share/rear-tmp

function cf--backup--66_make_drbd_vm_backup {
        /share/bin/drbd-vm-backup -b -t drbd-single -p gzip -d $TMP_DIR/isofs/drbd-vm-backup
}

function cf--restore--66_restore_drbd_vm_backup {
        drbd-vm-backup -r -t drbd-test,drbd-single -p gzip -c -d $BUILD_DIR/outputfs/drbd-vm-backup
}

without custom function, i will put custom backup/restore script to rear directory (maybe make a rpm for it) and define some more variable at local.conf. (since I don't want to put different custom scripts to different cluster node, so I want to use configuration files to control the behavior). my custom script will look like below.

add script parameters to /etc/rear/local.conf

DRBD_BACKUP=drbd-single
DRBD_BACKUP_PARAMETER="-p gzip"
DRBD_RESTORE_PARAMETER="-p gzip"

66_make_drbd_vm_backup.sh

if [ -n "$DRBD_BACKUP" ];then
        /share/bin/drbd-vm-backup -b -t $DRBD_BACKUP $DRBD_BACKUP_PARAMETER -d $TMP_DIR/isofs/drbd-vm-backup
fi

66_restore_drbd_vm.sh

if [ -n "$DRBD_BACKUP" ];then
        drbd-vm-backup -r -t $DRBD_BACKUP $DRBD_RESTORE_PARAMETER -c -d $BUILD_DIR/outputfs/drbd-vm-backup
fi

as you can see, custom function will be clear and simple in my case. but I can live without it :)
and if rear would accept make a "PRE" and "POST" variable into every official script, things will be simple and clear for me too.

schlomo commented at 2014-11-03 07:20:¶

The recommended way in your scenario would be to create an RPM, e.g. called
"my-rear" which would ship the configuration as /etc/rear/site.conf
(instead of local.conf) and it would ship your extra scripts and simply
depend on rear. That way you just install "my-rear" and everything else
works automatically.

That is actually how I deploy everything. Just put it into a custom RPM.

If you want to submit a patch that would allow wrapping every ReaR script
with a suitably named pre_* or post_* function then I would be happy to
review it (Hint: It should go into the Source function)

On 3 November 2014 05:22, tbsky notifications@github.com wrote:

hi schlomo:
thanks a lot for your confirm. I am just providing an idea to inject
end-user code. as you said "drop your additions into the appropriate ReaR
subdirectories" is the official way. then I will just stay with it. as you
said, there is no such difference.

the "PRE_BACKUP" or "POST_BACKUP" won't help a lot, since code may need to
inject at some specific stage. but maybe a "PRE" and "POST" variable for
every "official" script? so if I need to inject some code at some stage, I
can just use that variable to inject my code. "Freepbx" use that method. I
don't know if it will be considered too lousy.

about " how do you deploy ReaR and that local.conf". I am using
ScientificLinux(RHEL) 6.5 and rear 1.6.1 comes with EPEL (but now using git
master) . i just edit /etc/rear/local.conf and do "rear mkbackup". my
complete /etc/rear/local.conf with custom function below:

BACKUP=NETFS
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/share/cluster/iso/' '/share/rear-tmp/')
PROGS=( "${PROGS[@]}" '/lib/drbd/drbdadm-84' '/lib/drbd/drbdsetup-84' 'pv' '/share/bin/drbd-vm-backup')
ISO_MAX_SIZE=25000
BACKUP_URL=iso://backup/
OUTPUT=ISO
OUTPUT_URL=null
ISO_DIR=/share/rear-tmp

function cf--backup--66_make_drbd_vm_backup {
/share/bin/drbd-vm-backup -b -t drbd-single -p gzip -d $TMP_DIR/isofs/drbd-vm-backup
}

function cf--restore--66_restore_drbd_vm_backup {
drbd-vm-backup -r -t drbd-test,drbd-single -p gzip -c -d $BUILD_DIR/outputfs/drbd-vm-backup
}

without custom function, i will put custom backup/restore script to rear
directory (maybe make a rpm for it) and define some more variable at
local.conf. (since I don't want to put different custom scripts to
different cluster node, so I want to use configuration files to control the
behavior). my custom script will look like below.

add script parameters to /etc/rear/local.conf

DRBD_BACKUP=drbd-single
DRBD_BACKUP_PARAMETER="-p gzip"
DRBD_RESTORE_PARAMETER="-p gzip"

66_make_drbd_vm_backup.sh

if [ -n "$DRBD_BACKUP" ];then
/share/bin/drbd-vm-backup -b -t $DRBD_BACKUP $DRBD_BACKUP_PARAMETER -d $TMP_DIR/isofs/drbd-vm-backup
fi

66_restore_drbd_vm.sh

if [ -n "$DRBD_BACKUP" ];then
drbd-vm-backup -r -t $DRBD_BACKUP $DRBD_RESTORE_PARAMETER -c -d $BUILD_DIR/outputfs/drbd-vm-backup
fi

as you can see, custom function will be clear and simple in my case. but I
can live without it :)
and if rear would accept make a "PRE" and "POST" variable into every
official script, things will be simple and clear for me too.

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/pull/502#issuecomment-61439295.

tbsky commented at 2014-11-03 08:07:¶

hi schlomo:

I am glad that you would take a review with the "pre" & "post" script idea. I will try to do it asap, I think it will make my life easier :)

my current linux machines has all kind of configuration, so I plan to tune local.conf to adopt the situation. but if I can find a configuration that suit all cases, I think I will take your route: make a rpm and deploy it!

thanks again for your patience and kindly help!!

gdha commented at 2014-11-04 14:23:¶

@tbsky is the pull request obsolete now?

tbsky commented at 2014-11-04 15:16:¶

hi gdha:
yes I think it is obsolete now. how to I handle it? just close it?

schlomo commented at 2014-11-04 15:36:¶

Yes, just close it, maybe with a reference to the new one.

On 4 November 2014 16:16, tbsky notifications@github.com wrote:

hi gdha:
yes I think it is obsolete now. how to I handle it? just close it?

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/pull/502#issuecomment-61654596.

tbsky commented at 2014-11-04 15:58:¶

ok. close it and reference to https://github.com/rear/rear/pull/503 for further discussion :)

[Export of Github issue for rear/rear.]

#502 PR closed: add support for modular custom function cf--$stage--my_function¶