#522 PR `merged`: DRLM integration with ReaR¶

Labels: enhancement

didacog opened issue at 2014-12-29 15:57:¶

DRLM (Disaster Recovery Linux Manager) integration with ReaR¶

These are the required configurations in order to configure ReaR to work properly with DRLM:

ReaR client required configurations for DRLM:¶

Create DRLM user on the client side:

useradd -d /home/drlm -c "DRLM User" -m -s /bin/bash drlm
passwd drlm

**NOTE: password will be blocked, the connection is only allowed using ssh keys.

Disable password aging for drlm user:

chage -I -1 -m 0 -M 99999 -E -1 drlm

Copy ssh key from DRLM Server to the new client (login as root on DRLM server):

ssh-copy-id drlm@client_name

SUDO DRLM user configuration:¶

On the client side (if not present):

mkdir /etc/sudoers.d
chmod 750 /etc/sudoers.d
echo "#includedir /etc/sudoers.d" >> /etc/sudoers

Create SUDOERS DRLM user rules (in old SUDO versions you may need to configure DRLM rule directly in /etc/sudoers):

vi /etc/sudoers.d/drlm

     Cmnd_Alias DRLM = /usr/sbin/rear -dDv mkrescue, \ 
 /usr/sbin/rear -dDv mkbackup, \ 
 /usr/sbin/rear -d mkrescue, \ 
 /usr/sbin/rear -d mkbackup, \ 
     /usr/sbin/rear -D mkrescue, \ 
     /usr/sbin/rear -D mkbackup, \ 
     /usr/sbin/rear -v mkrescue, \ 
     /usr/sbin/rear -v mkbackup, \ 
     /usr/sbin/rear mkrescue, \ 
     /usr/sbin/rear mkbackup, \ 
 /usr/sbin/rear dump

     drlm    ALL=(root)      NOPASSWD: DRLM

chmod 440 /etc/sudoers.d/drlm

Disable password login:

passwd -l drlm

DRLM Required /etc/rear/site.conf settings:¶

DRLM_MANAGED=y

(is set to (n) by default on /usr/share/rear/conf/default.conf)

schlomo commented at 2014-12-29 16:41:¶

Very nice indeed. Thanks a lot for participating in ReaR!

Why not simply modify OUTPUT_PREFIX instead of adding DLRM_NAME to 81_create_pxelinux_cfg.sh? At a quick glance it seems like DLRM_NAME would be added twice: Once in OUTPUT_URL and another time in 81_create_pxelinux_cfg.sh

Also, OUTPUT_PREFIX is used in several other places as well but you don't patch all of them, too.

Are there other patches following that use the DLRM_* variables? If not, then you could probably achieve the same without patching ReaR itself but by using those variables in the local.conf or site.conf to avoid writing the same thing several times.

Also, we love all variables to default to a sane value. If the intended use case of DLRM_NAME is actually HOSTNAME, then it should be set to that by default. Or simply use OUTPUT_PREFIX which is set to HOSTNAME by default already.

What would be the use case of setting another DRLM_NAME?

didacog commented at 2014-12-29 23:34:¶

In DRLM, a NFS share is a mounted DR image only exported to a defined DRLM client.
Every image is mounted and exported from /DRLM/STORE/[DRLM_NAME]

In this DR image we are storing the rescue image under PXE folder and backup under BKP folder.
This is why we need OUTPUT_PREFIX=PXE and NETFS_PREFIX=BKP in order to ReaR can create those directories on the NFS share.

/DRLM/STORE is the tftproot where are the pxelinux.cfg directory and pxelinux.0 image, and every DRLM client has it's own mountpoint.

This is the reason why we need $DRLM_NAME/$OUTPUT_PREFIX in 81_create_pxelinux_cfg.sh in order to give the correct path to kernel and initrd.

We set DRLM_NAME because maybe the user will define DRLM clients by it's own name ex: rear01, rear02,... instead of the real hostname of the server. This is one of the reasons for differentiate the HOSTNAME and DRLM_NAME.

DRLM_NAME can be usually the same that HOSTNAME but is not a DRLM requirement.

schlomo commented at 2014-12-30 09:28:¶

I see. Why not simply modify the OUTPUT_PREFIX and NETFS_PREFIX accordingly?

I would strongly suggest to drop the DRLM_NAME variable altogether and prevent the users from shooting themselves in the foot by using different names for host and application. This is a setup for failure. And where would such a mapping be maintained? That is why I think that you and your users would be better off to avoid that extra level of complexity and keep things simple.

In general my opinion on supporting DRLM would be to create a new BACKUP and OUTPUT method called DRLM that modifies the other ReaR variables to suit DRLM. And then simply have a DRLM_SERVER variable that points to the DRLM server.

That way a user would only say:

BACKUP=DRLM
OUTPUT=DRLM
DRLM_SERVER=fooserver.domain.tld

and be done. Seems much more user friendly and less error prone to me.

If the way how the DRLM server expects the data changes then you simply update those BACKUP and OUTPUT modules in ReaR and the user does not change his configuration.

Another thing: What is the purpose of the DRLM_MANAGED variable? It is not used in ReaR.

schlomo commented at 2015-01-04 18:39:¶

I got another idea how to better integrate your changes with the general ReaR concept:

Add code that would use the DRLM_SERVER variable to activate DRLM mode and modify OUTPUT_PREFIX and other variables. If DRLM_SERVER is not set, then nothing happens. That way the user has to configure only one thing (DRLM_SERVER) to activate and configure DRLM support.

Doing that will make you much more compatible with how ReaR works internally and will make sure that future development on other parts of ReaR will not need to know about DRLM.

The other thing is, I am still not happy with the concept of a DRLM_NAME variable and giving the user the option to name the client differently. Do you have right now a requirement for this? If not then I would kindly ask you to postpone this part till the need actually arises. If you have a concrete requirement I would love to learn more about it. (Maybe instead of setting it manually better have some code that queries the DRLM_SERVER with the hostname, MAC address etc. to find out what the DRLM_NAME should be.)

What do you think?

gdha commented at 2015-01-25 15:46:¶

@didacog End of January is the target to accept new pull requests as need a week or two to validate the next rear release-1.17. So, you have still one week to prepare your next pull request.

schlomo commented at 2015-01-28 14:35:¶

@didacog please notify me when you update the pull request.

didacog commented at 2015-01-28 23:59:¶

@schlomo @gdha,
I've updated the pull request with last changes.

Please let me know if any modifications are needed.

schlomo commented at 2015-01-29 08:12:¶

Hello @didacog,

this looks already very good. Small stuff:

Can you please update your branch to reduce the amount of white space changes and keep our indentation.
Can you please use [[ ]] instead of [ ].
Checking for all your DRLM_* variables should happen in a single [[ ]] block (see help [[) and the variables must be protected with " quotes, otherwise.
Please change the comment for the OUTPUT_PREFIX_PXE to something like "The remote file system layout for OUTPUT=PXE can be modified to accomodate different TFTP server layouts".

Some questions:

Are you sure about using HTTPS without certificate verification? You basically throw away all the benefits of actually using HTTPS and could use plain HTTP instead. If you want to protect the password "on the wire" then you can also use curl --digest (See man page). Better yet, dynamically issue a token on the DRLM server which is used as a one-time-password. Since ReaR can only be used when triggered from the DRLM server the DRLM server already knows that it plans to run ReaR on a given machine and can issue such a token and authorize the token to retrieve the ReaR configuration for that specific ReaR run. The token could also be the key in a database under which the ReaR config for this run is stored. And so on, lots of cool things you can do with that. You can then use the client name as user and the token as password. Or encode both client name and token into an "access key". Since you pass the username and password via command line to ReaR you could equally well pass such a one time token. That would actually prevent the admins from using ReaR without DRLM much more efficiently than a static username/password protection which will for example be visible in the sudo logs or also in the ReaR logs.
Using POST to retrieve information is somewhat unusual and not REST-like (See Wikipedia or for example http://code.tutsplus.com/tutorials/a-beginners-guide-to-http-and-rest--net-16340 for a short intro about how the HTTP verbs are usually used). Information retrieval is usually done with GET to indicate that the state on the server does not change through this action and because repeated calls to the same URL yield the same result.

However, since those are really your areas and of no concern to ReaR in general these topics will not prevent this merge.

Regards,
Schlomo

didacog commented at 2015-01-29 08:50:¶

Hello @schlomo,

We'll take a look to your suggestions and update the pull request. :)

Regards,
Didac

didacog commented at 2015-01-29 15:42:¶

Hello @schlomo,

Regarding to questions made:

HTTPS is not only to protect the password, also the configuration information that is sent. filesystems excluded/included, VGs, ...

Another option could be to initialize an empty variable like "DRLM_SSL_KEY=" in default.conf and distribute the key, and its path in the site.conf like DRLM_MANAGED. Using "curl --cert $ DRLM_SSL_KEY -u user: pass ... URL" instead of (-k)

Referring to send by POST method, we aren't working in REST-like mode because there isn't any app behind.
We have CGI's written in bash that allow us to interact via HTTP between rear and DRLM by standard protocol.

Now all we serve is a REAR configuration file, but we will use this model for other tasks, such as sending rear log output to drlm, list available backups on DRLM from ReaR, ...
with POST method, we can maintain the same standard way for any HTTP request between rear and DRLM, download a configuration or a list of backups or send logs or file ...

DRLM like ReaR, is written completely in BASH and if we don't find limitations impossible to remedy with bash and standard OS tools/protocols, we intend to continue to maintain bash as the only programming language.

I'll correct code indention, spacing,... today evening! ;)

Regards,
Didac

schlomo commented at 2015-01-29 17:06:¶

Hi @didacog,

as I said - your HTTP protocol is your responsability. IMHO using Bash is no reason for not doing REST-like properly...

Looking forward to your next update.

Regards,
Schlomo

didacog commented at 2015-01-30 02:24:¶

Hi @schlomo I've updated the pull request.

Thanks for your suggestions, have made possible much better integration than our initial approach.

Kind Regards,
Didac

schlomo commented at 2015-01-30 08:33:¶

@didacog Please check if this still works for you. I added a check for curl (how did this work for you during recovery if curl was not part of the rescue system???).

Thanks a lot for contributing to the ReaR ecosystem, your http://www.drlm.org website looks very nice!

@gdha I added DRLM to the release notes but I am not sure if it is in the right place.

didacog commented at 2015-01-30 13:28:¶

@schlomo @gdha
Many thanks to you! sincerely appreciate your support.

I added curl to REQUIRED_PROGS array by hand in default.conf for the tests and I completely forgot to take it in mind for the code in pull request. :_(

Many thanks for adding it in the code! :)

I may need to recover some sleep from last weeks... :P

Kind Regards,
Didac

schlomo commented at 2015-01-30 13:35:¶

We don't want to require curl for everybody. Since REQUIRED_PROGS is checked in prep/default/95_check_missing_programs.sh we cannot use that in the DRLM case because in prep/default/01_*.sh you already need curl to download further configuration. That is why I added a dedicated check to the code. Adding it to PROGS should then make sure that it will be also there in the rescue system.

gdha commented at 2015-01-30 13:49:¶

Indeed Schlomo is right:
./usr/share/rear/prep/default/01_set_drlm_env.sh:PROGS=( "${PROGS[@]}" curl
)

On Fri, Jan 30, 2015 at 2:35 PM, Schlomo Schapiro notifications@github.com
wrote:

We don't want to require curl for everybody. Since REQUIRED_PROGS is
checked in prep/default/95_check_missing_programs.sh we cannot use that in
the DRLM case because in prep/default/01__.sh you already *need_ curl to
download further configuration. That is why I added a dedicated check to
the code. Adding it to PROGS should then make sure that it will be also
there in the rescue system.

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/pull/522#issuecomment-72201036.

didacog commented at 2015-01-30 14:08:¶

Yes yes, understood. I added it there, just for my local tests and forgot to see where to put it on a correct site. On tests I only wanted that it works on recovery in a fast way, and I forgot. :P

schlomo commented at 2015-01-30 14:45:¶

No worries. Thats why we work together.

Another question: Is the DRLM code on GitHub the "latest" and matches the
code now in ReaR? I ask because the last DRLM release is 1.0 and from
October and you mentioned that the HTTPS feature is more recent. I wanted
to take a look at the server side of DRLM.

On 30 January 2015 at 15:08, Didac Oliveira notifications@github.com
wrote:

Yes yes, understood. I added it there, just for my local tests and forgot
to see where to put it on a correct site. On tests I only wanted that it
works on recovery in a fast way, and I forgot. :P

—
Reply to this email directly or view it on GitHub
https://github.com/rear/rear/pull/522#issuecomment-72206091.

didacog commented at 2015-01-30 14:50:¶

To all ReaR team,

On behalf of all DRLM team members, Thank you! for all your support and suggestions.

Have made it possible that our integration with rear finally looks pretty well and far better than our first proposal :P

Last month was crazy for us, in the middle of various projects on our customers, preparing DRLM website, the documentation, our wishes about integrating with rear 1.17, without sleeping a lot... and finally we got it! we are very glad! :)

Regarding this issue we'll postpone our work on DRLM 2.0 until we release 1.1 version with the new HTTP service well handled from drlm. We expect to have it during February.

As schlomo has seen, DRLM website is online at http://drlm.org and our initial Docs at http://docs.drlm.org.

We hope you liked our presentation video.. :D

Kind Regards
DRLM Team

didacog commented at 2015-01-30 14:56:¶

@schlomo we'll update ASAP our docs in order to have proper howto of manual config of the HTTP service until DRLM 1.1 release on February.

Regards,
Didac

[Export of Github issue for rear/rear.]

#522 PR merged: DRLM integration with ReaR¶

didacog opened issue at 2014-12-29 15:57:¶

DRLM (Disaster Recovery Linux Manager) integration with ReaR¶

ReaR client required configurations for DRLM:¶

SUDO DRLM user configuration:¶

DRLM Required /etc/rear/site.conf settings:¶

schlomo commented at 2014-12-29 16:41:¶

didacog commented at 2014-12-29 23:34:¶

schlomo commented at 2014-12-30 09:28:¶

schlomo commented at 2015-01-04 18:39:¶

gdha commented at 2015-01-25 15:46:¶

schlomo commented at 2015-01-28 14:35:¶

didacog commented at 2015-01-28 23:59:¶

schlomo commented at 2015-01-29 08:12:¶

didacog commented at 2015-01-29 08:50:¶

didacog commented at 2015-01-29 15:42:¶

schlomo commented at 2015-01-29 17:06:¶

didacog commented at 2015-01-30 02:24:¶

schlomo commented at 2015-01-30 08:33:¶

didacog commented at 2015-01-30 13:28:¶

schlomo commented at 2015-01-30 13:35:¶

gdha commented at 2015-01-30 13:49:¶

didacog commented at 2015-01-30 14:08:¶

schlomo commented at 2015-01-30 14:45:¶

didacog commented at 2015-01-30 14:50:¶

didacog commented at 2015-01-30 14:56:¶

#522 PR `merged`: DRLM integration with ReaR¶