Package signing and verification

Disclaimer: This post focuses on Debian-based and Fedora/RHEL-based distributions and packaging.

Everybody using a GNU/Linux distribution most likely knows that packages used by the given distribution are somehow signed and such signatures are somehow verified. Usually, this knowledge comes with the first requirement to import some key when an extra package repository is being added to the system (the standard repositories of a distribution use keys that are present and trusted by default). While users don’t usually pay much attention to the key import process and the particular key used, these keys and signatures are actually parts of the critical security mechanisms in their systems.

Purpose

Why are these signatures a critical security mechanism? And what are they exactly? Package signatures are digital signatures¹ and as such, they prove two things:

• authenticity, i.e. that the signed data is coming from the particular originator, and
• integrity, i.e. that the signed data was not tampered with.

In the case of package signing, they can be used to verify that the software (or data, in general) contained in the package comes from the given entity and that the contents are the same as when the signature was created, i.e. without extra content added later and without any modifications, introducing security issues, for example. The entity can be a particular human (sometimes required), a company, or a service. However, a part of best practices is that the signatures are not created automatically as part of a build process, but rather as an explicit step done by a human, only on packages that are properly tested and approved for release. This is a practice the CFEngine team has been following as well.

The key

To be more precise, a successful verification proves that the given signature was created with a particular key. Whoever can access the key and the data can create a valid signature. As mentioned above, it can be a human, a build system, or a package repository system. A particular key can represent a person, a company, or a service (or a particular user using the service). In all cases, it is the key that is trusted. This means that it is important to pay enough attention to making sure that it is the particular key desired to be trusted. Ideally, the key should be coming from a different place and channel than the signed data. Such a setup requires an attacker to control two sources/channels if they want to provide fraud packages. In all cases, the key’s fingerprint and metadata should be verified before it is imported and thus trusted.

For example, when installing with yum, after adding the CFEngine Community Edition repository and pointing it to the particular key, you get a prompt to confirm the fingerprint and import the key:

sudo yum install cfengine-community

Importing GPG key 0x9B920A5E:
 Userid     : "CFEngine Packages (Key for signing releases) <contact@northern.tech>"
 Fingerprint: F2EE 2A0E 76A6 D469 8AE5 301A 7420 C142 9B92 0A5E
 From       : https://cfengine-package-repos.s3.amazonaws.com/pub/gpg.key
Is this ok [y/N]:

and if we check the instructions for using the repository, it says that the fingerprint of the key used for CFEngine Community packages is

F2EE 2A0E 76A6 D469 8AE5  301A 7420 C142 9B92 0A5E

Cross-checking the fingerprint in the yum prompt and on the website allows the user to verify that the key associated with the repository is actually the expected (and trusted) one. Once accepted, the key is imported and marked as trusted in the system and yum then verifies the digital signature of the downloaded package before installing it.

What is being signed

In the above command, yum fetches the repository metadata, determines which packages it needs to download to install cfengine-community, downloads the package(s), downloads the particular key, and shows the prompt. If rejected or if yum doesn’t know which key to import, the package installation fails with an error. However, if we try a similar thing on a Debian-based system and skip the step to import the key, this is the error we get from:

sudo apt-get update

Err:6 https://cfengine-package-repos.s3.amazonaws.com/pub/apt/packages stable InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7420C1429B920A5E
E: The repository 'https://cfengine-package-repos.s3.amazonaws.com/pub/apt/packages stable InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.

We can see a significant difference in the above. While yum checks the signature of the downloaded package, apt-get checks the signature of the repository metadata. From a user perspective, as long as everything works, it is all the same – import a key and have signatures verified. However, the big difference is that while in the case of Fedora/RHEL-based distributions using RPM packages, individual packages each have a signature, on Debian-based distributions, it’s the repository metadata that is signed. In particular, the Release file in the repository is signed, which contains a checksum of the Packages file, which provides information about packages present in the repository together with checksums of the individual packages. The trust and verification mechanisms are different than in the case of RPM-based distributions. However, even in the case of Debian-based distributions each package is verified by comparing the checksum of the downloaded package with the checksum declared in the metadata (Packages) and the checksum of this piece of metadata is compared to the checksum in the Release file. And since this piece of metadata is trusted as long as the key it is signed with is trusted, there exists a chain of trust from the key to every individual package in the repository.

Standalone packages

As written above, when using a repository, there’s no functional difference between the approach to signatures in the Fedora/RHEL-based and Debian-based distributions. However, what happens when a standalone package, i.e. an .rpm (RPM) or .deb (DEB) file is being installed? There is no repository and thus on a Debian-based distribution, there is no way to verify the particular package (there is no signature). One would expect that the result must be that dpkg -i ./some_package.deb (or apt install ./some_package.deb) installs the file just fine and rpm -i ./some_package.rpm (or yum install ./some_package.rpm) reject the file because the file is signed and the key is not available. However, it’s not so easy. By default, the result is the same, the package is installed in both cases. rpm -i only gives a warning like this:

sudo rpm -i ./cfengine-nova-3.24.1-1.el9.x86_64.rpm

warning: ./cfengine-nova-3.24.1-1.el9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 9b920a5e: NOKEY

but the package is installed. The recommended approach is to use rpm -K to verify the signature of the file before installing it.²

Beyond default (practices)

The above describes the default behavior of the tools and, indirectly, the default (packaging) practices. Yet, the area of packaging, even when limited to Fedora/RHEL-based and Debian-based distributions, is wide, with many alternatives.

Forcing rpm to do package verification

For example, to follow up on the last case mentioned above, there is a way to force rpm to actually reject such packages:

sudo echo '%_pkgverify_level all' >> /etc/rpm/macros
sudo rpm -i ./cfengine-nova-3.24.1-1.el9.x86_64.rpm

warning: ./cfengine-nova-3.24.1-1.el9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 9b920a5e: NOKEY
    package cfengine-nova-3.24.1-1.el9.x86_64 does not verify: Header V3 RSA/SHA256 Signature, key ID 9b920a5e: NOKEY

The same warning is printed, but this time it is followed by a statement that the package does not verify and the package is not installed. yum (or dnf) has the localpkg_gpgcheck configuration option to enforce the same behavior³

DEB package signatures

The Securing Debian Manual explains how the standard scheme for signatures works, the way described above. At the end of the page, however, it also describes an Alternative per-package signing scheme in which individual packages are signed, similarly to how the default RPM package signing schema works. In fact, this is the approach the CFEngine team is using to allow verification of individual DEB packages before installation, which is a non-trivial process requiring extra tools and a custom XML config file.

RPM repo signatures

For completeness, we need to mention that yum and dnf have the repo_gpgcheck configuration option which, when enabled, will trigger signature verification of repository metadata, similarly to how apt and apt-get do it in Debian-based distributions.

Avoiding signatures

As described in this blog post, there are multiple different approaches to how signatures can be used to verify packages. Or more precisely, how to verify their authenticity and integrity. They have their pros and cons and limitations and all require a key to be verified and imported/trusted. To avoid their complexities and to require the user to carefully check and accept a key and to provide an easy verification of standalone packages, a commonly used alternative is to provide digests (fingerprints) of strong cryptographic checksums of the published packages. A strong cryptographic checksum (hash) ensures that the package contents were not tampered with. And if the source of the checksum digests can be trusted, ideally being a separate channel from the one used when downloading the package, comparing the checksum digest (fingerprint) of a downloaded package with the expected and trusted value can work as both integrity and authenticity verification. CFEngine release announcement emails use this practice to give users an alternative to signature-based package verification.

Summary

This blog post provides an overview of how package verification using digital signatures works in Fedora/RHEL-based and Debian-based GNU/Linux distributions. The schemes used are surprisingly different and the default practices and behavior of the default and commonly used tools can be surprising too. Thus the text also provides suggestions for how to enforce verification of packages in all scenarios and what to pay attention to in order to avoid undermining this critical part of the security mechanisms in modern GNU/Linux systems.