#3014 Issue closed: RFC: enabling automatic backup and restore test

Labels: enhancement, fixed / solved / done, ReaR Project

pcahyna opened issue at 2023-06-19 15:00:

@lzaoral and @antonvoznia have created a fully automated test that performs a backup, boots in to the rescue environment, recovers the system, and reboots into the recovered system. The test can be triggered on all pull requests, using Packit (the service that currently merely rebuilds RPMs).

The test scripts are currently in @antonvoznia's repository: https://github.com/antonvoznia/rear-testing . I would like to enable the tests on the main ReaR repository and run them on all PRs. The test structure is such that the scripts are assumed to be in a separate repository and in the main repository itself there is merely a some configuration (test plan) that describes which tests to run. If we are to keep the structure, we should import the scripts into a repository under our organization (if we run them by default, they should be under our control and not in some private repo).

We then need to choose a name for the test repository. rear-testing or rear-tests or whatever seems to me a bit too generic - one may have different kinds of tests (there is already a tests subdirectory with different tests, although they are not much used AFAICT). The distinctive feature of the new tests is that they are described using the TMT (Test Management Tool, https://github.com/teemtee/tmt ) metadata. For this reason, I would like to call the new repository rear-tmt-tests or similar. If tests written using a different framework appear, they would go into a different repo to keep things organized.

Thoughts?

If there are no objections, I would like to import and enable the new test quite soon - in a week or so.

If anyone is interested in technical details of this test, it is described in @antonvoznia 's diploma thesis - https://dspace.cvut.cz/handle/10467/107238 , which has already resulted in our ShellCheck configuration ( #2976 ).

jsmeix commented at 2023-06-20 11:13:

@pcahyna
Thank You!
An automatic backup and restore test is very much needed.
I have no objections to import and enable the new test soon.

As far as I read in
https://dspace.cvut.cz/bitstream/handle/10467/107238/F3-DP-2023-Voznia-Anton-dip_thesis-2.pdf
"Chapter 4 ReaR testing in infrastructure"
the code (in particular all those 'rl...' functions) in
https://github.com/antonvoznia/rear-testing
seem to be from
https://github.com/beakerlib
but - at least at first glance - I don't find documentation there
so currently for me this is mostly a "black box"
where I cannot review anything.

I have a question:
At least at first glance I could not find how the original system
and the replacement system are configured?
I mean in particular
how to specify the storage layout on the original system
and the disks with their sizes on the replacement system
so that we could test recovery of various storage layouts
on various more or less identical replacement systems.

pcahyna commented at 2023-06-20 11:25:

As far as I read in https://dspace.cvut.cz/bitstream/handle/10467/107238/F3-DP-2023-Voznia-Anton-dip_thesis-2.pdf "Chapter 4 ReaR testing in infrastructure" the code (in particular all those 'rl...' functions) in https://github.com/antonvoznia/rear-testing seem to be from https://github.com/beakerlib but - at least on first glance - I don't find documentation there so currently for me this is mostly a "black box" where I cannot review anything.

Yes, Beakerlib is the test framework used by the test script. Documentation is here: https://github.com/beakerlib/beakerlib/wiki/man

I have a question: At least on first glance I could not find how the original system and the replacement system are configured? I mean in particular how to specify the storage layout on the original system and the disks with their sizes on the replacement system so that we could test recovery of various storage layouts on various more or less identical replacement systems.

This would be part of the hardware requirements, i.e. one would request a machine with e.g. multiple disks using the test metadata. One then could also add some storage preparation steps before the test itself, e.g. to create another Volume Group and filesystems on the additional disks. Unfortunately, support for additional hardware in the public infrastructure used to run the tests (Testing Farm) is little or nonexistent, but I am confident it will improve. So far, the configuration is very basic - only one disk, I think it even lacks LVM.

Providing a different replacement system to test stuff like migration mode is an intriguing idea. So far I have not attempted anything like that, because the original machine is used also as the machine to be recovered. Doing this would basically require reserving two machines, with access to some shared storage for the backup.

jsmeix commented at 2023-06-20 11:33:

It is perfectly right to start with what is currently available
(i.e. a simple system with recovery on the original machine)
and later try to enhance things step by step
as far as possible with reasonable effort.

Ah!
Nowadays documentation may no longer be in 'doc' but in 'wiki' :-)

Strange:
I cannot find a link to
https://github.com/beakerlib/beakerlib/wiki/man
anywhere on
https://github.com/beakerlib/beakerlib
Am I blind?

pcahyna commented at 2023-06-20 11:55:

Strange:
I cannot find a link to
https://github.com/beakerlib/beakerlib/wiki/man
anywhere on
https://github.com/beakerlib/beakerlib
Am I blind?

Strange indeed. Since I have Beakerlib installed, I just type man beakerlib when I need to read the manual page - it has also the link to the online version. I suspect this is the case for most beakerlib users and that's why they have not cared so much about accessibility of the online docs.
@sopos would there be a place for a link to the docs somewhere where it would be easy to discover, please?

sopos commented at 2023-06-21 08:24:

Right, there's no link from the README.md, there's a "tab" Wiki on the top, however. Moreover, we are in process of moving it to the readthedocs so I plan to make some updates including links, anyway.

pcahyna commented at 2023-06-22 14:16:

By the way:
there is already a repository for integration tests, https://github.com/rear/rear-integration-tests. its description reads: "The purpose of this project is to create Ansible playbooks to perform automated backup and restore tests for different Linux distributions." The purpose seems similar to what we are doing, but the approach is very different - these tests use a VM that they create, spawn and control from outside. It is all written as an Ansible playbook: https://github.com/rear/rear-integration-tests/blob/master/os/centos7.yml . This approach is more flexible, but is not suitable for automated testing in CI, because the machines that we get from available CI infrastructures do not support virtualization ( although this may change, https://github.com/actions/runner-images/issues/183 ), so one can not spawn and control a test VM this way.
The repository also has tests written in Beakerlib by my colleagues (similarly to the proposed backup and restore test) in the "tests" directory https://github.com/rear/rear-integration-tests/tree/master/tests, https://github.com/rear/rear-integration-tests/pull/2 . The tests do not seem to have anything to do with the original scope of the repository (they are pure backup tests, not full backup and recovery) and are not integrated with the Ansible playbooks in any way.

Given that there is already a repository that contains Beakerlib tests, maybe I should import the new test there, rather than creating a new repository? In the future this would allow us to enable the other Beakerlib tests under https://github.com/rear/rear-integration-tests/tree/master/tests quite easily if we find them valuable.

github-actions commented at 2023-08-22 02:01:

Stale issue message

pcahyna commented at 2023-09-01 12:04:

Tests themselves are in https://github.com/rear/rear-integration-tests/pull/3 . @gdha can you please have a look? I don't have commit access to that repository.

pcahyna commented at 2023-10-05 06:34:

Thank you @gdha for your assistance, finally done 🎉

[Export of Github issue for rear/rear.]