#3289 Issue `closed`: OBS signing key has expired (The gpg key signing file 'repomd.xml' has expired)¶

Labels: enhancement, support / question, fixed / solved / done, ReaR Project

thomasmerz opened issue at 2024-07-22 10:49:¶

https://github.com/rear/rear/issues/2309 is back:

Warning: The gpg key signing file 'repomd.xml' has expired.
  Repository:       Relax-and-Recover snapshot packages (15.5)
  Key Fingerprint:  985D 26DB 1764 72E4 5FFE 5FF5 1B4D D1E2 FFC8 DD5F
  Key Name:         Archiving:Backup:Rear OBS Project <Archiving:Backup:Rear@build.opensuse.org>
  Key Algorithm:    RSA 2048
  Key Created:      Thu 17 Mar 2022 04:08:05 PM CET
  Key Expires:      Sat 25 May 2024 05:08:05 PM CEST (EXPIRED)
  Rpm Name:         gpg-pubkey-ffc8dd5f-62334ed5

ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.7 / 2022-07-13
If your ReaR version is not the current version, explain why you can't upgrade: see title of issue 😉
OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"): openSUSE Leap 15.5

jsmeix commented at 2024-07-22 12:40:¶

@thomasmerz
as always I have no idea how that OBS key stuff actually works.
To me all that is obscure and behaves inexplicably.

So I need details how I could reproduce your issue.
Ideally exact command lines that I could type in
and what their output is on your system
so I could compare your output with mine.
I do not use graphical GUI tools so I could not
reproduce issues with graphical GUI tools.

FYI
what I did so far
(basically blind attempts to "see" something):

https://build.opensuse.org/projects/Archiving:Backup:Rear/signing_keys

Expires on    2026-09-30

which matches

# osc signkey Archiving:Backup:Rear | gpg
...
pub   rsa2048 2020-01-17 [SC] [expires: 2026-09-30]
      985D26DB176472E45FFE5FF51B4DD1E2FFC8DD5F

# osc signkey Archiving:Backup:Rear:15.5 | gpg
...
pub   rsa2048 2020-01-17 [SC] [expires: 2026-09-30]
      985D26DB176472E45FFE5FF51B4DD1E2FFC8DD5F

# osc signkey --extend Archiving:Backup:Rear
<status code="ok" />

But contradictingly

# curl https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/repodata/repomd.xml.key | gpg
...
pub   rsa2048 2020-01-17 [SC] [expired: 2024-05-25]
      985D26DB176472E45FFE5FF51B4DD1E2FFC8DD5F

but this somehow matches what
http://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/repodata/
shows

Name                      Last Modified
...
repomd.xml.key            10/19/2023

But on June 03 2024 I had already triggered a rebuild
of Archiving:Backup:Rear rear-2.7 for all build targets:
https://github.com/rear/rear/issues/3235#issuecomment-2144590757

???

jsmeix commented at 2024-07-22 12:45:¶

I tried a rebuild:

# osc rebuild Archiving:Backup:Rear rear-2.7 15.5
ok

# osc results -v Archiving:Backup:Rear rear-2.7 | grep '^15.5'
15.5  x86_64   rear-2.7  building: building on i04-ch1d:2
15.5  ppc64le  rear-2.7  building: building on obs-power9-15:10

# osc results -v Archiving:Backup:Rear rear-2.7 | grep '^15.5'
15.5  x86_64   rear-2.7  outdated (was: finished: unchanged)
15.5  ppc64le  rear-2.7  succeeded

# osc results -v Archiving:Backup:Rear rear-2.7 | grep '^15.5'
15.5  x86_64   rear-2.7  succeeded
15.5  ppc64le  rear-2.7  succeeded

# curl https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/repodata/repomd.xml.key | gpg
...
pub   rsa2048 2020-01-17 [SC] [expired: 2024-05-25]
      985D26DB176472E45FFE5FF51B4DD1E2FFC8DD5F

thomasmerz commented at 2024-07-22 12:56:¶

I was just doing some zypper commands when I saw this "has expired" message.

I'm using the same repo for OpenSUSE 15.5 as your example above:

$ zypper lr Archiving_Backup_Rear_Snapshot
Alias          : Archiving_Backup_Rear_Snapshot
Name           : Relax-and-Recover snapshot packages (15.5)
URI            : https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/
Enabled        : Yes
GPG Check      : (r ) Yes
Priority       : 99 (default priority)
Autorefresh    : Off
Keep Packages  : Off
Type           : rpm-md
GPG Key URI    : https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/repodata/repomd.xml.key
Path Prefix    :
Parent Service :
Keywords       : ---
Repo Info Path : /etc/zypp/repos.d/Archiving_Backup_Rear_Snapshot.repo
MD Cache Path  : /var/cache/zypp/raw/Archiving_Backup_Rear_Snapshot

After your rebuild and my zypper ref -f there's a newer certificat 👍🏼

$ curl https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/repodata/repomd.xml.key 2>/dev/null | gpg
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa2048 2020-01-17 [SC] [expires: 2026-09-30]
      985D26DB176472E45FFE5FF51B4DD1E2FFC8DD5F
uid           Archiving:Backup:Rear OBS Project <Archiving:Backup:Rear@build.opensuse.org>

BUT… 😢

Signature verification failed for file 'repomd.xml' from repository 'Relax-and-Recover snapshot packages (15.5)'.

    Note: Signing data enables the recipient to verify that no modifications occurred after the data
    were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
    and in extreme cases even to a system compromise.

    Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
    whole repo.

    Warning: This file was modified after it has been signed. This may have been a malicious change,
    so it might not be trustworthy anymore! You should not continue unless you know it's safe.

    Note: This might be a transient issue if the server is in the midst of receiving new data. The
    data file and its signature are two files which must fit together. In case the request hit the
    server in the midst of updating them, the signature verification might fail. After a few
    minutes, when the server has updated its data, it should work again.

Signature verification failed for file 'repomd.xml' from repository 'Relax-and-Recover snapshot packages (15.5)'. Continue? [yes/no] (no): yes
Retrieving repository 'Relax-and-Recover snapshot packages (15.5)' metadata .................................................................[error]
Repository 'Relax-and-Recover snapshot packages (15.5)' is invalid.
[Archiving_Backup_Rear_Snapshot|https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/] Valid metadata not found at specified URL
History:
 - File './repodata/591e88d87481f355baaf2df44fe8352331dc11dbc8991d07fba266e0682e8de0-primary.xml.gz' not found on medium 'https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/'

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'Relax-and-Recover snapshot packages (15.5)' because of the above error.

Even after "yes" every time: "Signature verification failed".

jsmeix commented at 2024-07-22 13:04:¶

Got that beast!

See in my previous
https://github.com/rear/rear/issues/3289#issuecomment-2242869945

# osc results -v Archiving:Backup:Rear rear-2.7 | grep '^15.5'
15.5  x86_64   rear-2.7  outdated (was: finished: unchanged)
15.5  ppc64le  rear-2.7  succeeded

That finished: unchanged triggered something in my mind:

I think I vaguely remember that OBS does not re-publish
a rebuilt package, when the build result is unchanged
(i.e. osc rebuild rebuilds but that does not enforce
to also re-publish it).

So I tried what happens when I artificially change
something so that the build result gets changed.

I did in rear.spec spelling fixes in comments 'rear' -> 'ReaR'
see
https://build.opensuse.org/package/revisions/Archiving:Backup:Rear/rear-2.7
and the actual changes
https://build.opensuse.org/package/rdiff/Archiving:Backup:Rear/rear-2.7?linkrev=base&rev=4

This automatically triggered a rebuild of
Archiving:Backup:Rear rear-2.7 for all build targets
and now - after that rebuild finished - I get

# curl https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/repodata/repomd.xml.key | gpg
...
pub   rsa2048 2020-01-17 [SC] [expires: 2026-09-30]
      985D26DB176472E45FFE5FF51B4DD1E2FFC8DD5F

which now matches what

osc signkey Archiving:Backup:Rear | gpg

shows, cf. my above
https://github.com/rear/rear/issues/3289#issuecomment-2242860385

thomasmerz commented at 2024-07-22 13:14:¶

Great! Thanks a lot ❤️ You fixed this issue:

$ zypper ref -f 1
Forcing raw metadata refresh
Retrieving repository 'Relax-and-Recover snapshot packages (15.5)' metadata ..................................................................[done]
Forcing building of repository cache
Building repository 'Relax-and-Recover snapshot packages (15.5)' cache .......................................................................[done]
Specified repositories have been refreshed.



$ curl https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/repodata/repomd.xml.key 2>/dev/null | gpg
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa2048 2020-01-17 [SC] [expires: 2026-09-30]
      985D26DB176472E45FFE5FF51B4DD1E2FFC8DD5F
uid           Archiving:Backup:Rear OBS Project <Archiving:Backup:Rear@build.opensuse.org>

jsmeix commented at 2024-07-22 13:16:¶

Regarding zypper:

I do not yet use a Archiving:Backup:Rear repository
so I start here from scratch on my Leap 15.5 system:

(I show full output here unless marked with '...')

# zypper ar http://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/Archiving:Backup:Rear.repo
Adding repository 'Relax-and-Recover (15.5)' ...........[done]
Repository 'Relax-and-Recover (15.5)' successfully added

URI         : https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/
Enabled     : Yes
GPG Check   : Yes
Autorefresh : No
Priority    : 99 (default priority)

Repository priorities are without effect. All enabled repositories share the same priority.

# zypper lr
Repository priorities are without effect. All enabled repositories share the same priority.

#  | Alias                 | Name                     | Enabled | GPG Check | Refresh
---+-----------------------+--------------------------+---------+-----------+--------
 1 | Archiving_Backup_Rear | Relax-and-Recover (15.5) | Yes     | ( p) Yes  | No
...

# zypper refresh Archiving_Backup_Rear

New repository or package signing key received:

  Repository:       Relax-and-Recover (15.5)
  Key Fingerprint:  985D 26DB 1764 72E4 5FFE 5FF5 1B4D D1E2 FFC8 DD5F
  Key Name:         Archiving:Backup:Rear OBS Project <Archiving:Backup:Rear@build.opensuse.org>
  Key Algorithm:    RSA 2048
  Key Created:      Mon Jul 22 14:11:05 2024
  Key Expires:      Wed Sep 30 14:11:05 2026
  Rpm Name:         gpg-pubkey-ffc8dd5f-669e4c59

    Note: Signing data enables the recipient to verify that
    no modifications occurred after the data were signed.
    Accepting data with no, wrong or unknown signature
    can lead to a corrupted system and in extreme cases
    even to a system compromise.

    Note: A GPG pubkey is clearly identified by its
    fingerprint. Do not rely on the key's name.
    If you are not sure whether the presented key is
    authentic, ask the repository provider or check
    their web site. Many providers maintain a web page
    showing the fingerprints of the GPG keys they are using.

Do you want to reject the key, trust temporarily, or trust always? [r/t/a/?] (r): a
Retrieving repository 'Relax-and-Recover (15.5)' metadata ....[done]
Building repository 'Relax-and-Recover (15.5)' cache .........[done]
Specified repositories have been refreshed.

# zypper refresh Archiving_Backup_Rear
Repository 'Relax-and-Recover (15.5)' is up to date.                                                                                                                                  
Specified repositories have been refreshed.

# rpm -qi gpg-pubkey-ffc8dd5f-669e4c59
Name        : gpg-pubkey
Version     : ffc8dd5f
Release     : 669e4c59
Architecture: (none)
Install Date: Mon Jul 22 15:10:56 2024
Group       : Public Keys
Size        : 0
License     : pubkey
Signature   : (none)
Source RPM  : (none)
Build Date  : Mon Jul 22 14:11:05 2024
Build Host  : localhost
Relocations : (not relocatable)
Packager    : Archiving:Backup:Rear OBS Project <Archiving:Backup:Rear@build.opensuse.org>
Summary     : gpg(Archiving:Backup:Rear OBS Project <Archiving:Backup:Rear@build.opensuse.org>)
Description :
-----BEGIN PGP PUBLIC KEY BLOCK-----
...
-----END PGP PUBLIC KEY BLOCK-----

Distribution: (none)

jsmeix commented at 2024-07-22 13:24:¶

@thomasmerz
thank you for your prompt reply that it now also works for you.

I noticed that your Archiving:Backup:Rear repository
is named Relax-and-Recover snapshot packages
and also other values contain 'snapshot'
but I had removed the Archiving:Backup:Rear:Snapshot
repository some time ago, see
https://github.com/rear/rear/issues/3056
so I assume your Archiving:Backup:Rear repository
is somewhat old and perhaps somewhat outdated.

Perhaps things work better when you remove that
old and possibly somewhat outdated repository
from zypper and (re)-install it from scratch
as I did it above in my
https://github.com/rear/rear/issues/3289#issuecomment-2242937020

But this is only an offhanded proposal.
I don't know your system environment
and/or your respositories setup so perhaps
better don't touch it more than actually needed?

thomasmerz commented at 2024-07-22 13:45:¶

You fixed it, you got the praise 👏🏼
My repo was already latest/correct, but the name was something old. Thanks for your hint I had to check with your "URI: https://download.opensuse.org/repositories/Archiving:/Backup:/Rear/15.5/"

jsmeix commented at 2024-07-22 14:24:¶

:-)
actually I did not fix it (i.e. I did not correct a bug)
but I learned now how to work around this bug in OBS:

It seems OBS does not re-build or re-publish a package
when metadata changed (at least not when expiration date
of a key changed).

I think when metadata of a published package changed,
then OBS should normally (i.e. except exceptions)
automatcally re-build and re-publish the package.

I think such an enhanced re-build and re-publish behaviour
would be in line with the current automated re-build
of a package which is:
When the package sources changed but also when
another package changed that is needed to build
(i.e. when something changed that gets installed by OBS
into its build system), simply put: when the build system
of a package changed, then the package gets automatically
re-built by OBS.

Bottom line (or "lesson learned"):
I think for all ReaR source packages in OBS
I should do some artificial source change once a year
to trigger a complete re-build and re-publish.