From 589803c186697d1a2212422c48d5acf8e767dc7f Mon Sep 17 00:00:00 2001
From: James Kirsch <generalfuzz@gmail.com>
Date: Thu, 2 Jul 2020 13:40:37 -0700
Subject: [PATCH] Update TLS documentation
Updated TLS documentation to reflect new features and configuration
options added in Ussuri.
Change-Id: I74550eaf394287b14fc521293cc4b5ea8074192c
Partially-Implements: blueprint add-ssl-internal-network
---
doc/source/admin/tls.rst | 371 +++++++++++++++++++++++++++++----------
1 file changed, 282 insertions(+), 89 deletions(-)
diff --git a/doc/source/admin/tls.rst b/doc/source/admin/tls.rst
index eae7241fe8..eefca82477 100644
--- a/doc/source/admin/tls.rst
+++ b/doc/source/admin/tls.rst
@@ -2,127 +2,320 @@
TLS
===
-An additional endpoint configuration option is to enable or disable
-TLS protection for the internal and/or external VIP. TLS allows a client to
-authenticate the OpenStack service endpoint and allows for encryption of the
-requests and responses.
+This guide describes how to configure Kolla Ansible to deploy OpenStack with
+TLS enabled. Enabling TLS on the provided internal and/or external VIP address
+allows OpenStack clients to authenticate and encrypt network communication
+with OpenStack services.
-The configuration variables that control TLS networking are:
+When an OpenStack service exposes an API endpoint, Kolla Ansible will
+configure HAProxy for that service to listen on the internal and/or
+external VIP address. The HAProxy container load-balances requests on
+the VIPs to the nodes running the service container.
-- kolla_enable_tls_external
-- kolla_external_fqdn_cert
-- kolla_enable_tls_internal
-- kolla_internal_fqdn_cert
+There are two different layers of TLS configuration for OpenStack APIs:
+
+1. Enabling TLS on the internal and/or external VIP, so communication between
+ an OpenStack client and the HAProxy listening on the VIP is secure.
+2. Enabling TLS on the backend network, so communication between HAProxy and
+ the backend API services is secure.
.. note::
- If TLS is enabled only on the internal or the external network
- the kolla_internal_vip_address and kolla_external_vip_address must
- be different.
+ TLS authentication is based on certificates that have been signed by
+ trusted Certificate Authorities. Examples of commercial CAs are Comodo,
+ Symantec, GoDaddy, and GlobalSign. Letsencrypt.org is a CA that will
+ provide trusted certificates at no charge. If using a trusted CA is not
+ possible for your project, you can use a private CA, e.g. Hashicorp Vault,
+ to create a certificate for your domain, or see
+ :ref:`admin-tls-generating-a-private-ca` to use a Kolla Ansible generated
+ private CA.
- If there is only a single network configured in your network topology
- (opposed to configuring seperate internal and external networks), TLS
- can be enabled using only the internal network configuration variables.
+Quick Start
+~~~~~~~~~~~
-The default for TLS is disabled, to enable TLS networking:
+.. note::
+
+ The certificates generated by Kolla Ansible use a simple Certificate
+ Authority setup and are not suitable for a production deployment. Only
+ certificates signed by a trusted Certificate Authority should be used in a
+ production deployment.
+
+To deploy OpenStack with TLS enabled for the external, internal and backend
+APIs, configure the following in ``globals.yml``:
.. code-block:: yaml
- kolla_enable_tls_external: "yes"
- kolla_external_fqdn_cert: "{{ kolla_certificates_dir }}/mycert.pem"
+ kolla_enable_tls_internal: "yes"
+ kolla_enable_tls_external: "yes"
+ kolla_enable_tls_backend: "yes"
+ kolla_copy_ca_into_containers: "yes"
- and/or
+If deploying on Debian or Ubuntu:
- kolla_enable_tls_internal: "yes"
- kolla_internal_fqdn_cert: "{{ kolla_certificates_dir }}/mycert-internal.pem"
+.. code-block:: yaml
+ openstack_cacert: "/etc/ssl/certs/ca-certificates.crt"
-.. note::
+If on CentOS or RHEL:
- TLS authentication is based on certificates that have been
- signed by trusted Certificate Authorities. Examples of commercial
- CAs are Comodo, Symantec, GoDaddy, and GlobalSign. Letsencrypt.org
- is a CA that will provide trusted certificates at no charge. Many
- company's IT departments will provide certificates within that
- company's domain. If using a trusted CA is not possible for your
- situation, you can use `OpenSSL <https://www.openssl.org>`__
- to create your own company's domain or see the section below about
- kolla generated self-signed certificates.
+.. code-block:: yaml
-Two certificate files are required to use TLS securely with authentication.
-These two files will be provided by your Certificate Authority. These
-two files are the server certificate with private key and the CA certificate
-with any intermediate certificates. The server certificate needs to be
-installed with the kolla deployment and is configured with the
-``kolla_external_fqdn_cert`` or ``kolla_internal_fqdn_cert`` parameter.
-If the server certificate provided is not already trusted by the client,
-then the CA certificate file will need to be distributed to the client.
+ openstack_cacert: "/etc/pki/tls/certs/ca-bundle.crt"
-When using TLS to connect to a public endpoint, an OpenStack client will
-have settings similar to this:
-
-.. code-block:: shell
-
- export OS_PROJECT_DOMAIN_ID=default
- export OS_USER_DOMAIN_ID=default
- export OS_PROJECT_NAME=demo
- export OS_USERNAME=demo
- export OS_PASSWORD=demo-password
- export OS_AUTH_URL=https://mykolla.example.net:5000
- # os_cacert is optional for trusted certificates
- export OS_CACERT=/etc/pki/ca/mykolla-cacert.crt
- export OS_IDENTITY_API_VERSION=3
-
-Self-Signed Certificates
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. note::
-
- Self-signed certificates should never be used in production.
-
-It is not always practical to get a certificate signed by a well-known
-trust CA, for example a development or internal test kolla deployment. In
-these cases it can be useful to have a self-signed certificate to use.
-
-For convenience, the ``kolla-ansible`` command will generate the necessary
-certificate files based on the information in the ``globals.yml``
-configuration file:
+The Kolla Ansible ``certificates`` command generates a private test Certificate
+Authority, and then uses the CA to sign the generated certificates for the
+enabled VIP(s) to test TLS in your OpenStack deployment. Assuming you are
+using the ``multinode`` inventory:
.. code-block:: console
- kolla-ansible certificates
+ kolla-ansible -i ~/multinode certificates
-The certificate file haproxy.pem will be generated and stored in the
-``/etc/kolla/certificates/`` directory, and the CA cert will be in the
-``/etc/kolla/certificates/ca/`` directory.
+TLS Configuration for internal/external VIP
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The configuration variables that control TLS for the internal and/or external
+VIP are:
+
+- ``kolla_enable_tls_external``
+- ``kolla_enable_tls_internal``
+- ``kolla_internal_fqdn_cert``
+- ``kolla_external_fqdn_cert``
+
+.. note::
+
+ If TLS is enabled only on the internal or external network, then
+ ``kolla_internal_vip_address`` and ``kolla_external_vip_address`` must
+ be different.
+
+ If there is only a single network configured in your topology
+ (as opposed to separate internal and external networks), TLS
+ can only be enabled using the internal network configuration variables.
+
+The default state for TLS networking is disabled. To enable external TLS
+encryption:
+
+.. code-block:: yaml
+
+ kolla_enable_tls_external: "yes"
+
+To enable internal TLS encryption:
+
+.. code-block:: yaml
+
+ kolla_enable_tls_internal: "yes"
+
+Two certificate files are required to use TLS securely with authentication,
+which will be provided by your Certificate Authority:
+
+* server certificate with private key
+* CA certificate with any intermediate certificates
+
+The combined server certificate and private key needs to be provided to Kolla
+Ansible, with the path configured via ``kolla_external_fqdn_cert`` or
+``kolla_internal_fqdn_cert``. These paths default to ``{{
+kolla_certificates_dir }}/haproxy.pem`` and ``{{ kolla_certificates_dir
+}}/haproxy-internal.pem`` respectively, where ``kolla_certificates_dir`` is
+``/etc/kolla/certificates`` by default.
+
+If the server certificate provided is not already trusted by clients, then
+the CA certificate file will need to be distributed to the clients. This is
+discussed in more detail in :ref:`admin-tls-openrc` and
+:ref:`admin-tls-ca-in-containers`.
+
+.. _admin-tls-openrc:
+
+Configuring the OpenStack Client for TLS
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The location for the CA certificate for the ``admin-openrc.sh`` file is
+configured with the ``kolla_admin_openrc_cacert`` variable, which is not set by
+default. This must be a valid path on all hosts where ``admin-openrc.sh`` is
+used.
+
+When TLS is enabled on a VIP, and ``kolla_admin_openrc_cacert`` is set to
+``/etc/pki/tls/certs/ca-bundle.crt``, an OpenStack client will have settings
+similar to this configured by ``admin-openrc.sh``:
+
+.. code-block:: shell
+
+ export OS_PROJECT_DOMAIN_NAME=Default
+ export OS_USER_DOMAIN_NAME=Default
+ export OS_PROJECT_NAME=admin
+ export OS_TENANT_NAME=admin
+ export OS_USERNAME=admin
+ export OS_PASSWORD=demoPassword
+ export OS_AUTH_URL=https://mykolla.example.net:5000
+ export OS_INTERFACE=internal
+ export OS_ENDPOINT_TYPE=internalURL
+ export OS_MISTRAL_ENDPOINT_TYPE=internalURL
+ export OS_IDENTITY_API_VERSION=3
+ export OS_REGION_NAME=RegionOne
+ export OS_AUTH_PLUGIN=password
+ # os_cacert is optional for trusted certificates
+ export OS_CACERT=/etc/pki/tls/certs/ca-bundle.crt
+
+.. _admin-tls-ca-in-containers:
Adding CA Certificates to the Service Containers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To copy CA certificate files to the service containers
+To copy CA certificate files to the service containers:
.. code-block:: yaml
- kolla_copy_ca_into_containers: "yes"
+ kolla_copy_ca_into_containers: "yes"
-When ``kolla_copy_ca_into_containers`` is configured to "yes", the
-CA certificate files in /etc/kolla/certificates/ca will be copied into
+When ``kolla_copy_ca_into_containers`` is configured to "yes", the CA
+certificate files in ``/etc/kolla/certificates/ca`` will be copied into
service containers to enable trust for those CA certificates. This is required
for any certificates that are either self-signed or signed by a private CA,
-and are not already present in the service image trust store.
+and are not already present in the service image trust store. Kolla will
+install these certificates in the container system wide trust store when the
+container starts.
-All certificate file names will have the "kolla-customca-" prefix prepended to
-it when it is copied into the containers. For example, if a certificate file is
-named "internal.crt", it will be named "kolla-customca-internal.crt" in the
-containers.
+All certificate file names will have the ``kolla-customca-`` prefix prepended
+to them when they are copied into the containers. For example, if a certificate
+file is named ``internal.crt``, it will be named
+``kolla-customca-internal.crt`` in the containers.
-For Debian and Ubuntu containers, the certificate files will be copied to
-the ``/usr/local/share/ca-certificates/`` directory.
+For Debian and Ubuntu containers, the certificate files will be copied to the
+``/usr/local/share/ca-certificates/`` directory.
-For Centos and Red Hat Linux containers, the certificate files will be copied
-to the ``/etc/pki/ca-trust/source/anchors/`` directory.
+For CentOS and RHEL containers, the certificate files will be copied to the
+``/etc/pki/ca-trust/source/anchors/`` directory.
-In addition, the ``openstack_cacert`` should be configured with the path to
-the cacert in the container. For example, if the self-signed certificate task
-was used and the deployment is on ubuntu, the path would be:
-"/etc/pki/ca-trust/source/anchors/kolla-customca-haproxy-internal.crt"
+In both cases, valid certificates will be added to the system trust store -
+``/etc/ssl/certs/ca-certificates.crt`` on Debian and Ubuntu, and
+``/etc/pki/tls/certs/ca-bundle.crt`` on CentOS and RHEL.
+
+Configuring a CA bundle
+~~~~~~~~~~~~~~~~~~~~~~~
+
+OpenStack services do not always trust CA certificates from the system trust
+store by default. To resolve this, the ``openstack_cacert`` variable should be
+configured with the path to the CA Certificate in the container.
+
+To use the system trust store on Debian or Ubuntu:
+
+.. code-block:: yaml
+
+ openstack_cacert: /etc/ssl/certs/ca-certificates.crt
+
+For CentOS or RHEL:
+
+.. code-block:: yaml
+
+ openstack_cacert: /etc/pki/tls/certs/ca-bundle.crt
+
+Back-end TLS Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Enabling TLS on the backend services secures communication between the
+HAProxy listing on the internal/external VIP and the OpenStack
+services. It also enables secure end-to-end communication between OpenStack
+services that support TLS termination. The OpenStack services that support
+backend TLS termination in Ussuri are: Keystone, Glance, Heat, Placement,
+Horizon, Barbican, and Cinder.
+
+The configuration variables that control back-end TLS for service endpoints
+are:
+
+- ``kolla_enable_tls_backend``
+- ``kolla_tls_backend_cert``
+- ``kolla_tls_backend_key``
+- ``haproxy_backend_cacert``
+- ``haproxy_backend_cacert_dir``
+
+The default state for back-end TLS is disabled. To enable TLS for the back-end
+communication:
+
+.. code-block:: yaml
+
+ kolla_enable_tls_backend: "yes"
+
+It is also possible to enable back-end TLS on a per-service basis. For example,
+to enable back-end TLS for Keystone, set ``keystone_enable_tls_backend`` to
+``yes``.
+
+The default values for ``haproxy_backend_cacert`` and
+``haproxy_backend_cacert_dir`` should suffice if the certificate is in the
+system trust store. Otherwise, they should be configured to a location of the
+CA certificate installed in the service containers.
+
+Each backend service requires a certificate and private key. In many cases it
+is necessary to use a separate certificate and key for each host, or even
+per-service. The following precedence is used for the certificate:
+
+* ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}/{{ project_name
+ }}-cert.pem``
+* ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}-cert.pem``
+* ``{{ kolla_certificates_dir }}/{{ project_name }}-cert.pem``
+* ``{{ kolla_tls_backend_cert }}``
+
+And for the private key:
+
+* ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}/{{ project_name
+ }}-key.pem``
+* ``{{ kolla_certificates_dir }}/{{ inventory_hostname }}-key.pem``
+* ``{{ kolla_certificates_dir }}/{{ project_name }}-key.pem``
+* ``{{ kolla_tls_backend_key }}``
+
+The default for ``kolla_certificates_dir`` is ``/etc/kolla/certificates``.
+
+``kolla_tls_backend_cert`` and ``kolla_tls_backend_key``, default to ``{{
+kolla_certificates_dir }}/backend-cert.pem`` and ``{{ kolla_certificates_dir
+}}/backend-key.pem`` respectively.
+
+``project_name`` is the name of the OpenStack service, e.g. ``keystone`` or
+``cinder``.
+
+.. note::
+ The back-end TLS cert/key can be the same certificate that is used for the
+ VIP, as long as those certificates are configured to allow requests from
+ both the VIP and internal networks.
+
+By default, the TLS certificate will be verified as trustable by the
+OpenStack services. Although not recommended for production, it is possible to
+disable verification of the backend certificate:
+
+.. code-block:: yaml
+
+ kolla_verify_tls_backend: "no"
+
+.. _admin-tls-generating-a-private-ca:
+
+Generating a Private Certificate Authority
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note::
+
+ The certificates generated by Kolla Ansible use a simple Certificate
+ Authority setup and are not suitable for a production deployment. Only
+ certificates signed by a trusted Certificate Authority should be used in a
+ production deployment.
+
+It's not always practical to get a certificate signed by a trusted CA. In a
+development or internal test OpenStack deployment, it can be useful to
+generate certificates locally to enable TLS.
+
+For convenience, the ``kolla-ansible`` command will generate the necessary
+certificate files based on the information in the ``globals.yml``
+configuration file and the inventory file:
+
+.. code-block:: console
+
+ kolla-ansible -i multinode certificates
+
+The ``certificates`` role performs the following actions:
+
+1. Generates a test root Certificate Authority
+2. Generates the internal/external certificates which are signed by the root
+ CA.
+3. If back-end TLS is enabled, generate the back-end certificate signed by
+ the root CA.
+
+The combined certificate and key file ``haproxy.pem`` (which is the default
+value for ``kolla_external_fqdn_cert``) will be generated and stored in the
+``/etc/kolla/certificates/`` directory, and a copy of the CA certificate
+(``root.crt``) will be stored in the ``/etc/kolla/certificates/ca/`` directory.