#742 Issue closed: Test suite for rear

Labels: enhancement, needs sponsorship

phracek opened issue at 2015-12-16 11:35:

Would it possible to create a test suite for rear?
Or even rear already include it, but I did not find them.

schlomo commented at 2015-12-16 11:54:

YES, we all want this.

So far we have make validate and @aussendorf built something for their Bareos product.

phracek commented at 2015-12-18 08:18:

@aussendorf I am trying to thing how to test ReaR. Any suggestions or examples?

I guess that it would be awesome to have it and test at least basic functionality.

jhoekx commented at 2015-12-18 08:53:

@dagwieers and I have had a plan for this for a few years already, but we never got around to actually doing it. Dag already has a server for it.

I'm running a DR test every week automatically as an Ansible playbook. Building on that, our plan was to:

  • create a master ReaR image that has all the necessary tooling
  • boot it
  • upload and run a script that creates the test layout
  • select a distribution tar and extract it over the layout
  • boot the newly created system
  • install the ReaR version under test
  • make a ReaR image
  • recover the system

Thoughts?

Furthermore I was also thinking about a rear test workflow to run unit tests, but given that tools on different distributions have a different output, you would end up in mocking hell. I think there is more value in end-to-end tests.

aussendorf commented at 2015-12-18 10:57:

Hello,
I've integrated a ReaR test into our overall Bareos CI tests. We have VM (xen) templates for every distribution we support. Whenever we build new Bareos packages, we automatically clone those VMs, and install, configure, run backup/restore, install debug-packages, create traceback-files, de-install, and some other stuff.

For ReaR I've included a fully automated test in our framework that:

  • creates a rear image (iso)
  • takes that iso as boot-image for vm (via xen api)
  • boots ReaR system (to do this I needed the patch, to configure the default option in isolinux.cfg)
  • connect by ssh and run "rear recover"
  • remove iso
  • reboot
  • do some sanity checks.

Everything controlled using jenkins.

The scripts are integrated into a broader scripting system so it does not make much sense to post them partially.

dagwieers commented at 2015-12-23 21:42:

Ok, @jhoekx and me sat together today to look into automated testing and we did 2 things today.

  • We integrated the 'make validate' step as part of a TravisCI hook, so that commits automatically get syntax checked by Bash.
  • We started a separate project, named 'rear-test' (lacking a better name) with the goal to have automated backup and restore tests using different methods.

This last part is unfinished yet, it currently only includes a single Ansible playbook to create a CentOS 7 image. The ultimate goal is to include Ansible playbooks that use prior generated images to set up infrastructure (if needed) and perform the backup and restore tests on pre-made configuration files.

We intend to have OS-independent playbooks for testing, so that they can be used for different Linux distributions. Since each backup and restore method test only needs a specific Rear configuration, it should be straightforward to reuse this for different OS images. So we do intend to have SLES, Ubuntu, Arch and whatever interests users to contribute. However our aim now is to ensure we have the proper framework set up first.

Once we have these tests implemented, I expect some opportunities to improve Rear for automated testing so it becomes easier to find specific error-conditions (that may or may not be fatal to Rear). Because it is essential that we can find issues in an automated way.

schlomo commented at 2015-12-24 08:26:

@dagwieers & @jhoekx thank you so much! This is really great news.

I have only one suggestion: Maybe we should rename the repo to rear-integration-tests or something else that is a bit more self-explanatory? I am only worried that people would think at their first glance that this is just some test and not for real. Or that it contains the bleeding edge test stuff...

jhoekx commented at 2015-12-24 08:52:

Ok, renamed to rear-integration-tests.

gdha commented at 2016-02-18 17:21:

Issue #38 was the original request (we can close that one) and we better use this issue for further discussions etc.

gdha commented at 2016-08-23 10:38:

@rear/owners set the milestone to Rear future as that describes it much better - it is something that might be available in the future (perhaps sponsored?)

gdha commented at 2017-07-19 16:05:

We have https://github.com/gdha/rear-automated-testing in place to test daily ReaR builds (we do this on request for customers - yes those who pay for it)

gdha commented at 2018-01-12 08:02:

The "Relax-and-Recover Automated Test" project already proofed it works sufficient to test new versions of ReaR. More enhancements are needed, but as mentioned before we will only implement new features when these are sponsored. With this I think we can close this topic.

[Export of Github issue for rear/rear.]