Update artifact signing key management process
Now that the SKS keyserver network is no more, and there's no convenient way to share third-party key signatures, we need to adjust our key management and rollover process accordingly. Change-Id: I7008706aae06b6e4a16db2dd85a8c7f91530cd50
This commit is contained in:
Jeremy Stanley
parent
867675d13d
commit
738f42760a
@ -5,14 +5,14 @@
|
||||
Signing System
|
||||
##############
|
||||
|
||||
Our standard signing automation leverages an OpenPGP signing subkey,
|
||||
encrypted as a Zuul secret, to create detached signatures for
|
||||
release artifacts (tarballs, wheels, et cetera) and to sign and push
|
||||
Git tags as part of our managed release automation. The master key
|
||||
corresponding to this subkey is replaced near the start of each new
|
||||
development cycle and set to expire soon after the cycle is
|
||||
scheduled to conclude (with enough overlap to allow for graceful
|
||||
replacement).
|
||||
Our standard signing automation model leverages an OpenPGP signing
|
||||
subkey, encrypted as a Zuul secret, to create detached signatures
|
||||
for release artifacts (tarballs, wheels, et cetera) and to sign and
|
||||
push Git tags as part of managed release automation. For OpenStack's
|
||||
releases, the master key corresponding to this subkey is replaced
|
||||
near the start of each new development cycle and set to expire soon
|
||||
after the cycle is scheduled to conclude (with enough overlap to
|
||||
allow for graceful replacement).
|
||||
|
||||
|
||||
At a Glance
|
||||
@ -23,49 +23,44 @@ At a Glance
|
||||
<https://opendev.org/openstack/project-config/src/branch/master/zuul.d/secrets.yaml>`_
|
||||
:Roles:
|
||||
* `add-gpgkey
|
||||
<https://docs.openstack.org/infra/zuul-jobs/roles.html#role-add-gpgkey>`_
|
||||
<https://docs.openstack.org/infra/zuul-jobs/general-roles.html#role-add-gpgkey>`_
|
||||
* `sign-artifacts
|
||||
<https://docs.openstack.org/infra/zuul-jobs/roles.html#role-sign-artifacts>`_
|
||||
<https://docs.openstack.org/infra/zuul-jobs/general-roles.html#role-sign-artifacts>`_
|
||||
|
||||
|
||||
Key Management Overview
|
||||
=======================
|
||||
|
||||
The signing system is implemented as a set of Zuul v3 jobs; these
|
||||
The signing system is implemented as a set of Zuul jobs; these
|
||||
utilize the signing subkey via an encrypted Zuul secret imported
|
||||
into a job's ``~/.gnupg/secring.gpg`` at runtime. It's used by jobs
|
||||
to create detached signatures of release artifacts and to sign Git
|
||||
tags under release management automation.
|
||||
tags in release management automation.
|
||||
|
||||
|
||||
Storage
|
||||
-------
|
||||
|
||||
While the signing subkey is installed unencrypted on some job nodes,
|
||||
so that it can be used unattended by job automation, the
|
||||
corresponding master key is kept symmetrically encrypted in the root
|
||||
home directory of the Infra systems management bastion instead. At
|
||||
the time of key creation a revocation certificate is also generated,
|
||||
for which Infra root sysadmins are encouraged to retrieve and keep
|
||||
local copies in case control over or access to the original master
|
||||
key is lost. In the future, the master key and revocation
|
||||
certificate may be distributed across our root team rather than kept
|
||||
in one place (for example using Shamir's secret sharing scheme
|
||||
similar to what `the Debian Project does for its archive keys
|
||||
<https://ftp-master.debian.org/keys.html>`).
|
||||
so that it can be used unattended by job automation, for OpenStack
|
||||
the corresponding master key is kept symmetrically encrypted in the
|
||||
root home directory of the Infra systems management bastion. At the
|
||||
time of key creation a revocation certificate is also generated, for
|
||||
which sysadmins are encouraged to retrieve and keep local copies in
|
||||
case control over or access to the original master key is lost.
|
||||
|
||||
|
||||
Rotation
|
||||
--------
|
||||
|
||||
The master key is rotated at the start of each development cycle
|
||||
(usually shortly after cycle-trailing deliverables are released),
|
||||
signed by a majority of Infra root sysadmins before being put into
|
||||
service, and has an expiration date set for a month after the end of
|
||||
the targeted development cycle. The newly-created key gets signed by
|
||||
the old, and this signature pushed to the public keyserver network.
|
||||
New key fingerprints are also submitted to the openstack/releases
|
||||
repository, for publication on the releases.openstack.org Web site.
|
||||
For OpenStack, the master key is rotated at the start of each
|
||||
development cycle (usually shortly after cycle-trailing deliverables
|
||||
are released), signed by the previous key before being put into
|
||||
service, and has an expiration date set for at least a month after
|
||||
the end of the targeted development cycle (or best guess, often
|
||||
longer for safety). New key fingerprints are also submitted to the
|
||||
openstack/releases repository, for publication on the
|
||||
releases.openstack.org Web site.
|
||||
|
||||
|
||||
Revocation
|
||||
@ -93,25 +88,10 @@ our management bastion host::
|
||||
# installed both the gnupg and gnupg-curl packages. Set your umask
|
||||
# to 077, create a /root/signing.gnupg directory and place this
|
||||
# configuration file in it.
|
||||
#
|
||||
# Retrieve and validate the HKPS key for the SKS keyservers this way:
|
||||
#
|
||||
# wget -P ~/signing.gnupg/ \
|
||||
# https://sks-keyservers.net/sks-keyservers.netCA.pem{,.asc}
|
||||
# gpg --homedir signing.gnupg --recv-key \
|
||||
# 0x94CBAFDD30345109561835AA0B7F8B60E3EDFAE3
|
||||
# gpg --homedir signing.gnupg --verify \
|
||||
# ~/signing.gnupg/sks-keyservers.netCA.pem{.asc,}
|
||||
#
|
||||
# You'll need to list them in the accompanying dirmngr.conf file.
|
||||
|
||||
# Receive, send and search for keys in the SKS keyservers pool using
|
||||
# HKPS (OpenPGP HTTP Keyserver Protocol via TLS/SSL).
|
||||
keyserver hkps://hkps.pool.sks-keyservers.net
|
||||
|
||||
# Ignore keyserver URLs specified in retrieved/refreshed keys
|
||||
# so they don't direct you to update from non-HKPS sources.
|
||||
keyserver-options no-honor-keyserver-url
|
||||
keyserver hkps://keys.openpgp.org
|
||||
|
||||
# Display key IDs in a more accurate 16-digit hexidecimal format
|
||||
# and add 0x at the beginning for clarity.
|
||||
@ -122,28 +102,10 @@ our management bastion host::
|
||||
list-options show-uid-validity
|
||||
verify-options show-uid-validity
|
||||
|
||||
And this is the content of the ``/root/signing.gnupg/dirmngr.conf`` file on
|
||||
our management bastion host::
|
||||
|
||||
# Set the path to the public certificate for the
|
||||
# sks-keyservers.net CA used to verify connections to servers in
|
||||
# the accompanying gpg.conf file.
|
||||
hkp-cacert /root/signing.gnupg/sks-keyservers.netCA.pem
|
||||
|
||||
|
||||
Generation
|
||||
----------
|
||||
|
||||
Key generation should happen reasonably far in advance of expiration
|
||||
of the old key (at least a month), so as to provide ample time for a
|
||||
majority of our root sysadmins to attest to the key and provide
|
||||
warning to the rest of the community of the upcoming transition. Of
|
||||
course, if this is being done to replace a revoked key, this
|
||||
timeline should be accelerated as much as possible to provide
|
||||
continuity of service so use your best judgement on a balance of
|
||||
sufficient attestation and warning (same-day turnaround is
|
||||
preferred).
|
||||
|
||||
Make sure we start with a restrictive umask so that files and
|
||||
directories we write from this point forward are only accessible by
|
||||
the root user:
|
||||
@ -356,7 +318,14 @@ separately:
|
||||
.. code-block:: shell-session
|
||||
|
||||
root@bridge:~# gpg --homedir signing.gnupg --send-keys 0x120D3C23C6D5584D
|
||||
sending key 0x120D3C23C6D5584D to hkps server hkps.pool.sks-keyservers.net
|
||||
gpg: sending key 0x120D3C23C6D5584D to hkps://keys.openpgp.org
|
||||
|
||||
Check the infra-root inbox (the address associated with the key) for
|
||||
a notification about the key upload to keys.openpgp.org and follow
|
||||
the URL within it. Once there, click the button to send a
|
||||
verification message. Now check the inbox again and follow the URL
|
||||
provided in the new message which should arrive. Once that's done,
|
||||
the key will be searchable on the keyserver.
|
||||
|
||||
The rest of this process shouldn't happen until we're ready for the
|
||||
signing system to transition to our new key. In a typical,
|
||||
@ -373,7 +342,7 @@ GnuPG directory:
|
||||
root@bridge:~# umask 077
|
||||
root@bridge:~# mkdir temporary.gnupg
|
||||
root@bridge:~# gpg --homedir signing.gnupg \
|
||||
> --output temporary.gnupg/secret-subkeys
|
||||
> --output temporary.gnupg/secret-subkeys \
|
||||
> --export-secret-subkeys 0xC0224DB5F541FB68\!
|
||||
root@bridge:~# gpg --homedir temporary.gnupg \
|
||||
> --import temporary.gnupg/secret-subkeys
|
||||
@ -408,8 +377,11 @@ from the master key) must be reset to an empty string in the new
|
||||
temporary copy. Here we override the default pinentry mode to
|
||||
loopback as a workaround for other pinentry frontends refusing to
|
||||
accept an empty passphrase (unfortunately the prompting and feedback
|
||||
from the loopback pinentry leaves something to be desired). This is
|
||||
again done using an interactive key editor session:
|
||||
from the loopback pinentry leaves something to be desired). Note
|
||||
that the first password prompt will be for the original key's
|
||||
password, but the second password prompt should be left blank thus
|
||||
removing that password. This is again done using an interactive key
|
||||
editor session:
|
||||
|
||||
.. code-block:: shell-session
|
||||
|
||||
@ -457,7 +429,7 @@ as a secret to Zuul for use by release jobs.
|
||||
root@bridge:~# wget https://opendev.org/zuul/zuul/raw/branch/master/tools/encrypt_secret.py
|
||||
root@bridge:~# python3 encrypt_secret.py --tenant openstack \
|
||||
> --infile temporary.gnupg/for-zuul --outfile temporary.gnupg/zuul.yaml \
|
||||
> https://zuul.openstack.org openstack/project-config
|
||||
> https://zuul.opendev.org openstack/project-config
|
||||
writing RSA key
|
||||
Public key length: 4096 bits (512 bytes)
|
||||
Max plaintext length per chunk: 470 bytes
|
||||
@ -483,9 +455,8 @@ public master key:
|
||||
|
||||
.. code-block:: shell-session
|
||||
|
||||
root@bridge:~# ( gpg --fingerprint \
|
||||
> 0x120d3c23c6d5584d6fc2464664dbb05acc5e7c28
|
||||
> gpg --armor --export-options export-clean,export-minimal \
|
||||
root@bridge:~# ( gpg --fingerprint --list-sigs \
|
||||
> 0x120d3c23c6d5584d6fc2464664dbb05acc5e7c28 ; gpg --armor \
|
||||
> --export 0x120d3c23c6d5584d6fc2464664dbb05acc5e7c28 ) > \
|
||||
> 0x120d3c23c6d5584d6fc2464664dbb05acc5e7c28.txt
|
||||
|
||||
@ -501,89 +472,18 @@ of the prior key as the start date for the new one).
|
||||
Attestation
|
||||
-----------
|
||||
|
||||
We need a majority (if not all) of our current root sysadmins to
|
||||
verify and attest to the authenticity of our artifact signing key,
|
||||
because it represents a system maintained by our team rather than
|
||||
representing some particular individual and so anyone else attesting
|
||||
to this key can really only do so transitively through us. This
|
||||
should be done soon after a new key is minted (preferably the same
|
||||
week) so that others in the community who wish to extend the web of
|
||||
trust around the key based on our attestations (for example, release
|
||||
managers or team leads) have an opportunity to do so before it's put
|
||||
into production.
|
||||
Since the collapse of the SKS keyserver network, no popular
|
||||
keyservers publish third-party key signatures any longer (the
|
||||
pollution thereof was a big part of SKS's downfall). As such, until
|
||||
something like a caff-style signature approval mechanism is
|
||||
integrated into the OpenPGP keyserver infrastructure or some
|
||||
alternative comes along, there's little point in generating our own
|
||||
individual key signatures we can't easily distribute to users. For
|
||||
now, we simply make sure the signature made by the previous key is
|
||||
included in the export we publish from our own sites so that users
|
||||
can have some path to confirm new keys.
|
||||
|
||||
Start by logging into the management bastion and examining the
|
||||
fingerprint of the key as it exists on disk:
|
||||
|
||||
.. code-block:: shell-session
|
||||
|
||||
me@bridge:~$ sudo gpg --homedir /root/signing.gnupg --fingerprint \
|
||||
> --list-keys "OpenStack Infra (Some Cycle)"
|
||||
pub ed25519/0x120D3C23C6D5584D 2016-07-07 [expires: 2017-02-02]
|
||||
Key fingerprint = 120D 3C23 C6D5 584D 6FC2 4646 64DB B05A CC5E 7C28
|
||||
uid [ultimate] OpenStack Infra (Some Cycle) <infra-root@openstack.org>
|
||||
sub cv25519/0x1F215B56867C5D9A 2016-07-07 [expires: 2017-02-02]
|
||||
sub ed25519/0xC0224DB5F541FB68 2016-07-07
|
||||
|
||||
Now on your own system where your OpenPGP key resides, retrieve the
|
||||
key, compare the fingerprint from above, and if they match, sign it
|
||||
and push the signature back to the keyserver network:
|
||||
|
||||
.. code-block:: shell-session
|
||||
|
||||
me@home:~$ gpg2 --recv-keys 0x120D3C23C6D5584D
|
||||
gpg: requesting key 0x120D3C23C6D5584D from hkps server hkps.pool.sks-keyservers.net
|
||||
gpg: key 0x120D3C23C6D5584D: public key "OpenStack Infra (Some Cycle) <infra-root@openstack.org>" imported
|
||||
gpg: 3 marginal(s) needed, 1 complete(s) needed, classic trust model
|
||||
gpg: depth: 0 valid: 3 signed: 31 trust: 0-, 0q, 0n, 0m, 0f, 3u
|
||||
gpg: depth: 1 valid: 31 signed: 46 trust: 30-, 0q, 0n, 0m, 1f, 0u
|
||||
gpg: next trustdb check due at 2016-11-30
|
||||
gpg: Total number processed: 1
|
||||
gpg: imported: 1
|
||||
me@home:~$ gpg2 --fingerprint 0x120D3C23C6D5584D
|
||||
pub ed25519/0x120D3C23C6D5584D 2016-07-07 [expires: 2017-02-02]
|
||||
Key fingerprint = 120D 3C23 C6D5 584D 6FC2 4646 64DB B05A CC5E 7C28
|
||||
uid [ full ] OpenStack Infra (Some Cycle) <infra-root@openstack.org>
|
||||
sub cv25519/0x1F215B56867C5D9A 2016-07-07 [expires: 2017-02-02]
|
||||
sub ed25519/0xC0224DB5F541FB68 2016-07-07
|
||||
me@home:~$ gpg2 --sign-key 0x120D3C23C6D5584D
|
||||
|
||||
pub ed25519/0x120D3C23C6D5584D created: 2016-07-07 expires: 2017-02-02 usage: SC
|
||||
trust: unknown validity: full
|
||||
sub cv25519/0x1F215B56867C5D9A created: 2016-07-07 expires: 2017-02-02 usage: E
|
||||
sub ed25519/0xC0224DB5F541FB68 created: 2016-07-07 expires: never usage: S
|
||||
[ full ] (1). OpenStack Infra (Some Cycle) <infra-root@openstack.org>
|
||||
|
||||
|
||||
pub ed25519/0x120D3C23C6D5584D created: 2016-07-07 expires: 2017-02-02 usage: SC
|
||||
trust: unknown validity: full
|
||||
Primary key fingerprint: 120D 3C23 C6D5 584D 6FC2 4646 64DB B05A CC5E 7C28
|
||||
|
||||
OpenStack Infra (Some Cycle) <infra-root@openstack.org>
|
||||
|
||||
This key is due to expire on 2017-02-02.
|
||||
Are you sure that you want to sign this key with your
|
||||
key "My Name <me@example.org>" (0xAB54A98CEB1F0AD2)
|
||||
|
||||
Really sign? (y/N) y
|
||||
|
||||
+-----------------------------------------------------------------------+
|
||||
| Please enter the passphrase to unlock the secret key for the OpenPGP |
|
||||
| certificate: |
|
||||
| "My Name <me@example.org>" |
|
||||
| ID 0xAB54A98CEB1F0AD2, |
|
||||
| created 2008-09-10. |
|
||||
| |
|
||||
| |
|
||||
| Passphrase **********************____________________________________ |
|
||||
| |
|
||||
| <OK> <Cancel> |
|
||||
+-----------------------------------------------------------------------+
|
||||
|
||||
me@home:~$ gpg2 --send-keys 0x120D3C23C6D5584D
|
||||
gpg: sending key 0x120D3C23C6D5584D to hkps server hkps.pool.sks-keyservers.net
|
||||
|
||||
Also, please retrieve a copy of the
|
||||
Still, please retrieve a copy of the
|
||||
``/root/signing.gnupg/some.revoke.asc`` fallback revocation
|
||||
certificate (``some`` to be replaced with the lower-cased release
|
||||
name) from the management bastion and keep it stashed somewhere
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user