opendev/system-config
Code Issues Proposed changes
ded27cbb5d
system-config/playbooks/roles/letsencrypt-request-certs/tasks/acme.yaml

29 lines
1.1 KiB
YAML
Raw Normal View History

letsencrypt support This change contains the roles and testing for deploying certificates on hosts using letsencrypt with domain authentication. From a top level, the process is implemented in the roles as follows: 1) letsencrypt-acme-sh-install This role installs the acme.sh tool on hosts in the letsencrypt group, along with a small custom driver script to help parse output that is used by later roles. 2) letsencrypt-request-certs This role runs on each host, and reads a host variable describing the certificates required. It uses the acme.sh tool (via the driver) to request the certificates from letsencrypt. It populates a global Ansible variable with the authentication TXT records required. If the certificate exists on the host and is not within the renewal period, it should do nothing. 3) letsencrypt-install-txt-record This role runs on the adns server. It installs the TXT records generated in step 2 to the acme.opendev.org domain and then refreshes the server. Hosts wanting certificates will have pre-provisioned CNAME records for _acme-challenge.host.opendev.org pointing to acme.opendev.org. 4) letsencrypt-create-certs This role runs on each host, reading the same variable as in step 2. However this time the acme.sh tool is run to authenticate and create the certificates, which should now work correctly via the TXT records from step 3. After this, the host will have the full certificate material. Testing is added via testinfra. For testing purposes requests are made to the staging letsencrypt servers and a self-signed certificate is provisioned in step 4 (as the authentication is not available during CI). We test that the DNS TXT records are created locally on the CI adns server, however. Related-Spec: https://review.openstack.org/587283 Change-Id: I1f66da614751a29cc565b37cdc9ff34d70fdfd3f
2019-02-14 08:10:51 +11:00
- name: 'Build arguments for letsencrypt acme.sh driver for: {{ cert.key }}'
set_fact:
# NOTE(ianw): note the domains are passed in one string (between
# ") as it makes argument parsing a little easier in the driver.sh
Generate ssl check list directly from letsencrypt variables This autogenerates the list of ssl domains for the ssl-cert-check tool directly from the letsencrypt list. The first step is the install-certcheck role that replaces the puppet-ssl_cert_check module that does the same. The reason for this is so that during gate testing we can test this on the test bridge.openstack.org server, and avoid adding another node as a requirement for this test. letsencrypt-request-certs is updated to set a fact letsencrypt_certcheck_domains for each host that is generating a certificate. As described in the comments, this defaults to the first host specified for the certificate and the listening port can be indicated (if set, this new port value is stripped when generating certs as is not necessary for certificate generation). The new letsencrypt-config-certcheck role runs and iterates all letsencrypt hosts to build the final list of domains that should be checked. This is then extended with the letsencrypt_certcheck_additional_domains value that covers any hosts using certificates not provisioned by letsencrypt using this mechanism. These additional domains are pre-populated from the openstack.org domains in the extant check file, minus those openstack.org domain certificates we are generating via letsencrypt (see letsencrypt-create-certs/handlers/main.yaml). Additionally, we update some of the certificate variables in host_vars that are listening on port !443. As mentioned, bridge.openstack.org is placed in the new certcheck group for gate testing, so the tool and config file will be deployed to it. For production, cacti is added to the group, which is where the tool currently runs. The extant puppet installation is disabled, pending removal in a follow-on change. Change-Id: Idbe084f13f3684021e8efd9ac69b63fe31484606
2020-05-18 14:43:16 +10:00
acme_args: '"{% for domain in cert.value %}-d {{ domain.split(":")[0] }} {% endfor %}"'
letsencrypt support This change contains the roles and testing for deploying certificates on hosts using letsencrypt with domain authentication. From a top level, the process is implemented in the roles as follows: 1) letsencrypt-acme-sh-install This role installs the acme.sh tool on hosts in the letsencrypt group, along with a small custom driver script to help parse output that is used by later roles. 2) letsencrypt-request-certs This role runs on each host, and reads a host variable describing the certificates required. It uses the acme.sh tool (via the driver) to request the certificates from letsencrypt. It populates a global Ansible variable with the authentication TXT records required. If the certificate exists on the host and is not within the renewal period, it should do nothing. 3) letsencrypt-install-txt-record This role runs on the adns server. It installs the TXT records generated in step 2 to the acme.opendev.org domain and then refreshes the server. Hosts wanting certificates will have pre-provisioned CNAME records for _acme-challenge.host.opendev.org pointing to acme.opendev.org. 4) letsencrypt-create-certs This role runs on each host, reading the same variable as in step 2. However this time the acme.sh tool is run to authenticate and create the certificates, which should now work correctly via the TXT records from step 3. After this, the host will have the full certificate material. Testing is added via testinfra. For testing purposes requests are made to the staging letsencrypt servers and a self-signed certificate is provisioned in step 4 (as the authentication is not available during CI). We test that the DNS TXT records are created locally on the CI adns server, however. Related-Spec: https://review.openstack.org/587283 Change-Id: I1f66da614751a29cc565b37cdc9ff34d70fdfd3f
2019-02-14 08:10:51 +11:00
- name: Run acme.sh driver for certificate issue
shell:
cmd: |
letsencrypt : don't use staging in the gate Currently we connect to the LE staging environment with acme.sh during CI to get the DNS-01 tokens (but we never follow-through and actually generate the certificate, as we have nowhere to publish the tokens). We've known for a while that LE staging isn't really meant to be used by CI like this, and recent instability has made the issue pronounced. This modifies the driver script to generate fake tokens which work to ensure all the DNS processing, etc. is happening correctly. I have put this behind a flag so the letsencrypt job still does this however. I think it is worth this job actually calling acme.sh to validate this path; this shouldn't be required too often. Change-Id: I7c0b471a0661aa311aaa861fd2a0d47b07e45a72
2021-10-06 07:36:59 +11:00
/opt/acme.sh/driver.sh {{ 'issue-selfsign' if letsencrypt_self_generate_tokens else 'issue' }} {{ acme_args }}
letsencrypt support This change contains the roles and testing for deploying certificates on hosts using letsencrypt with domain authentication. From a top level, the process is implemented in the roles as follows: 1) letsencrypt-acme-sh-install This role installs the acme.sh tool on hosts in the letsencrypt group, along with a small custom driver script to help parse output that is used by later roles. 2) letsencrypt-request-certs This role runs on each host, and reads a host variable describing the certificates required. It uses the acme.sh tool (via the driver) to request the certificates from letsencrypt. It populates a global Ansible variable with the authentication TXT records required. If the certificate exists on the host and is not within the renewal period, it should do nothing. 3) letsencrypt-install-txt-record This role runs on the adns server. It installs the TXT records generated in step 2 to the acme.opendev.org domain and then refreshes the server. Hosts wanting certificates will have pre-provisioned CNAME records for _acme-challenge.host.opendev.org pointing to acme.opendev.org. 4) letsencrypt-create-certs This role runs on each host, reading the same variable as in step 2. However this time the acme.sh tool is run to authenticate and create the certificates, which should now work correctly via the TXT records from step 3. After this, the host will have the full certificate material. Testing is added via testinfra. For testing purposes requests are made to the staging letsencrypt servers and a self-signed certificate is provisioned in step 4 (as the authentication is not available during CI). We test that the DNS TXT records are created locally on the CI adns server, however. Related-Spec: https://review.openstack.org/587283 Change-Id: I1f66da614751a29cc565b37cdc9ff34d70fdfd3f
2019-02-14 08:10:51 +11:00
args:
chdir: /opt/acme.sh/
register: acme_output
letsencrypt: split staging and self-signed generation We currently only have letsencrypt_test_only as a single flag that sets tests to use the letsencrypt staging environment and also generates a self-signed certificate. However, for initial testing we actually want to fully generate certificates on hosts, but using the staging environment (i.e. *not* generate self-signed certs). Thus we need to split this option into two, so the gate tests still use staging+self-signed, but in-progress production hosts can just using the staging flag. These variables are split, and graphite01.opendev.org is made to create staging certificates. Also remove some debugging that is no longer necessary. Change-Id: I08959ba904f821c9408d8f363542502cd76a30a4
2019-04-10 07:31:26 +10:00
environment:
LETSENCRYPT_STAGING: '{{ "1" if letsencrypt_use_staging else "0" }}'
letsencrypt support This change contains the roles and testing for deploying certificates on hosts using letsencrypt with domain authentication. From a top level, the process is implemented in the roles as follows: 1) letsencrypt-acme-sh-install This role installs the acme.sh tool on hosts in the letsencrypt group, along with a small custom driver script to help parse output that is used by later roles. 2) letsencrypt-request-certs This role runs on each host, and reads a host variable describing the certificates required. It uses the acme.sh tool (via the driver) to request the certificates from letsencrypt. It populates a global Ansible variable with the authentication TXT records required. If the certificate exists on the host and is not within the renewal period, it should do nothing. 3) letsencrypt-install-txt-record This role runs on the adns server. It installs the TXT records generated in step 2 to the acme.opendev.org domain and then refreshes the server. Hosts wanting certificates will have pre-provisioned CNAME records for _acme-challenge.host.opendev.org pointing to acme.opendev.org. 4) letsencrypt-create-certs This role runs on each host, reading the same variable as in step 2. However this time the acme.sh tool is run to authenticate and create the certificates, which should now work correctly via the TXT records from step 3. After this, the host will have the full certificate material. Testing is added via testinfra. For testing purposes requests are made to the staging letsencrypt servers and a self-signed certificate is provisioned in step 4 (as the authentication is not available during CI). We test that the DNS TXT records are created locally on the CI adns server, however. Related-Spec: https://review.openstack.org/587283 Change-Id: I1f66da614751a29cc565b37cdc9ff34d70fdfd3f
2019-02-14 08:10:51 +11:00
letsencrypt: force renewal on certificate change There is a bug, or misfeature, in acme.sh using dns manual mode where it will not renew the certificate when new domains are added to an existing certificate. It appears to generate the TXT record requests correctly, but then when we renew the certificate it thinks it is not time and skips it. This is filed upstream with [1] however we can work around it, and generally be better anyway. For each letsencrypt host, during certificate request we build up the "acme_txt_required" key which is a list of TXT record tuples. Currently we keep the challenge domain in the first entry, which is not useful (all our hosts have the same challenge domain, amce.opendev.org). Modify this to be the certificate key from the host config. To be clear; when a host has letsencrypt_certs: hostname-cert-main: hostname.opendev.org altname.opendev.org hostname-cert-secondary: secondary.opendev.org secondaryalt.opendev.org acme_txt_required when renewing all certs will end up looking like: [ (hostname-cert-main, <txt1>), (hostname-cert-main, <txt2>), (hostname-cert-secondary, <txt3>), (hostname-cert-secondary, <txt3>>) ] In the certificate creation path, we walk "acme_txt_required" and take the unique 0-value entries; this gives us the list of keys in "letsencrypt_certs" which were actually updated. We can then force renewal for these certs, because we know they changed in some way that requires reissuing them (within renewal time, or new domains). This isn't just a work-around, it is generically better too. Previously if any cert on host required an update, we would try to update them all. This would be a no-op; acme.sh would just skip doing anything; but now we don't even have to call into the renewal if we know nothing has changed. [1] https://github.com/acmesh-official/acme.sh/issues/2763 Change-Id: I1e82c64217d46d7e1acc0111dff4db2f0062c42a
2020-02-28 11:49:06 +11:00
# NOTE(ianw): The output of the driver is
#
# challenge-domain:TXT-key
#
# We don't care about the challenge-domain part (we have set all
# _acme-challenge.hostname.o.o records as CNAMES to acme.opendev.org).
# Record the config key along with the TXT record; later we use it to
# check which config keys have been updated and need a refresh.
#
letsencrypt support This change contains the roles and testing for deploying certificates on hosts using letsencrypt with domain authentication. From a top level, the process is implemented in the roles as follows: 1) letsencrypt-acme-sh-install This role installs the acme.sh tool on hosts in the letsencrypt group, along with a small custom driver script to help parse output that is used by later roles. 2) letsencrypt-request-certs This role runs on each host, and reads a host variable describing the certificates required. It uses the acme.sh tool (via the driver) to request the certificates from letsencrypt. It populates a global Ansible variable with the authentication TXT records required. If the certificate exists on the host and is not within the renewal period, it should do nothing. 3) letsencrypt-install-txt-record This role runs on the adns server. It installs the TXT records generated in step 2 to the acme.opendev.org domain and then refreshes the server. Hosts wanting certificates will have pre-provisioned CNAME records for _acme-challenge.host.opendev.org pointing to acme.opendev.org. 4) letsencrypt-create-certs This role runs on each host, reading the same variable as in step 2. However this time the acme.sh tool is run to authenticate and create the certificates, which should now work correctly via the TXT records from step 3. After this, the host will have the full certificate material. Testing is added via testinfra. For testing purposes requests are made to the staging letsencrypt servers and a self-signed certificate is provisioned in step 4 (as the authentication is not available during CI). We test that the DNS TXT records are created locally on the CI adns server, however. Related-Spec: https://review.openstack.org/587283 Change-Id: I1f66da614751a29cc565b37cdc9ff34d70fdfd3f
2019-02-14 08:10:51 +11:00
- set_fact:
letsencrypt: force renewal on certificate change There is a bug, or misfeature, in acme.sh using dns manual mode where it will not renew the certificate when new domains are added to an existing certificate. It appears to generate the TXT record requests correctly, but then when we renew the certificate it thinks it is not time and skips it. This is filed upstream with [1] however we can work around it, and generally be better anyway. For each letsencrypt host, during certificate request we build up the "acme_txt_required" key which is a list of TXT record tuples. Currently we keep the challenge domain in the first entry, which is not useful (all our hosts have the same challenge domain, amce.opendev.org). Modify this to be the certificate key from the host config. To be clear; when a host has letsencrypt_certs: hostname-cert-main: hostname.opendev.org altname.opendev.org hostname-cert-secondary: secondary.opendev.org secondaryalt.opendev.org acme_txt_required when renewing all certs will end up looking like: [ (hostname-cert-main, <txt1>), (hostname-cert-main, <txt2>), (hostname-cert-secondary, <txt3>), (hostname-cert-secondary, <txt3>>) ] In the certificate creation path, we walk "acme_txt_required" and take the unique 0-value entries; this gives us the list of keys in "letsencrypt_certs" which were actually updated. We can then force renewal for these certs, because we know they changed in some way that requires reissuing them (within renewal time, or new domains). This isn't just a work-around, it is generically better too. Previously if any cert on host required an update, we would try to update them all. This would be a no-op; acme.sh would just skip doing anything; but now we don't even have to call into the renewal if we know nothing has changed. [1] https://github.com/acmesh-official/acme.sh/issues/2763 Change-Id: I1e82c64217d46d7e1acc0111dff4db2f0062c42a
2020-02-28 11:49:06 +11:00
acme_txt_required: '{{ acme_txt_required + [(cert.key, item.split(":")[1])] }}'
letsencrypt support This change contains the roles and testing for deploying certificates on hosts using letsencrypt with domain authentication. From a top level, the process is implemented in the roles as follows: 1) letsencrypt-acme-sh-install This role installs the acme.sh tool on hosts in the letsencrypt group, along with a small custom driver script to help parse output that is used by later roles. 2) letsencrypt-request-certs This role runs on each host, and reads a host variable describing the certificates required. It uses the acme.sh tool (via the driver) to request the certificates from letsencrypt. It populates a global Ansible variable with the authentication TXT records required. If the certificate exists on the host and is not within the renewal period, it should do nothing. 3) letsencrypt-install-txt-record This role runs on the adns server. It installs the TXT records generated in step 2 to the acme.opendev.org domain and then refreshes the server. Hosts wanting certificates will have pre-provisioned CNAME records for _acme-challenge.host.opendev.org pointing to acme.opendev.org. 4) letsencrypt-create-certs This role runs on each host, reading the same variable as in step 2. However this time the acme.sh tool is run to authenticate and create the certificates, which should now work correctly via the TXT records from step 3. After this, the host will have the full certificate material. Testing is added via testinfra. For testing purposes requests are made to the staging letsencrypt servers and a self-signed certificate is provisioned in step 4 (as the authentication is not available during CI). We test that the DNS TXT records are created locally on the CI adns server, however. Related-Spec: https://review.openstack.org/587283 Change-Id: I1f66da614751a29cc565b37cdc9ff34d70fdfd3f
2019-02-14 08:10:51 +11:00
loop: '{{ acme_output.stdout_lines }}'
Copy Permalink