From ee68bc385f7b7494c4f19ea9adfba1990f6acc1f Mon Sep 17 00:00:00 2001
From: Mark Goddard <mark@stackhpc.com>
Date: Tue, 21 Feb 2017 08:53:02 -0500
Subject: [PATCH] Improve developer documentation for Bifrost
This change fills out the Bifrost developer documentation and makes a
few corrections.
Change-Id: I14a6eca58c440677d5de638cbc677e9ec05738af
Closes-Bug: #1661013
Co-Authored-By: Sayantani Goswami <sayantani.goswami@intel.com>
---
doc/bifrost.rst | 407 ++++++++++++++++++++++++++++++++----------------
1 file changed, 273 insertions(+), 134 deletions(-)
diff --git a/doc/bifrost.rst b/doc/bifrost.rst
index a379de04af..82af61d946 100644
--- a/doc/bifrost.rst
+++ b/doc/bifrost.rst
@@ -2,31 +2,87 @@
Bifrost Guide
=============
+From the bifrost developer documentation:
-Prep host
-=========
+ Bifrost (pronounced bye-frost) is a set of Ansible playbooks that automates
+ the task of deploying a base image onto a set of known hardware using
+ Ironic. It provides modular utility for one-off operating system
+ deployment with as few operational requirements as reasonably possible.
-Clone kolla
------------
+Kolla uses bifrost as a mechanism for bootstrapping an OpenStack control plane
+on a set of baremetal servers. Kolla provides a container image for bifrost.
+Kolla-ansible provides a playbook to configure and deploy the bifrost
+container, as well as building a base OS image and provisioning it onto the
+baremetal nodes.
-::
+Hosts in the System
+===================
- git clone https://github.com/openstack/kolla
- cd kolla
+In a system deployed by bifrost we define a number of classes of hosts.
-set up kolla dependencies :doc:`quickstart`
+Control host
+ The control host is the host on which kolla and kolla-ansible will be
+ installed, and is typically where the cloud will be managed from.
+Deployment host
+ The deployment host runs the bifrost deploy container and is used to
+ provision the cloud hosts.
+Cloud hosts
+ The cloud hosts run the OpenStack control plane, compute and storage
+ services.
+Bare metal compute hosts:
+ In a cloud providing bare metal compute services to tenants via Ironic,
+ these hosts will run the bare metal tenant workloads. In a cloud with only
+ virtualised compute this category of hosts does not exist.
-Fix hosts file
---------------
+.. note::
-Docker bind mounts ``/etc/hosts`` into the container from a volume
-This prevents atomic renames which will prevent ansible from fixing
-the ``/etc/hosts`` file automatically.
+ In many cases the control and deployment host will be the same, although
+ this is not mandatory.
-To enable bifrost to be bootstrapped correctly add the deployment
-hosts hostname to 127.0.0.1 line for example:
+.. note::
-::
+ Bifrost supports provisioning of bare metal nodes. While kolla-ansible is
+ agnostic to whether the host OS runs on bare metal or is virtualised, in a
+ virtual environment the provisioning of VMs for cloud hosts and their base
+ OS images is currently out of scope.
+
+Cloud Deployment Procedure
+==========================
+
+Cloud deployment using kolla and bifrost follows the following high level
+steps:
+
+#. Install and configure kolla and kolla-ansible on the control host.
+#. Deploy bifrost on the deployment host.
+#. Use bifrost to build a base OS image and provision cloud hosts with this
+ image.
+#. Deploy OpenStack services on the cloud hosts provisioned by bifrost.
+
+Preparation
+===========
+
+Prepare the Control Host
+------------------------
+
+Follow the **Install dependencies** section of the :doc:`quickstart` guide
+instructions to set up kolla and kolla-ansible dependencies. Follow the
+instructions in either the **Install kolla for development** section or the
+**Install kolla for deployment or evaluation** section to install kolla and
+kolla-ansible.
+
+Prepare the Deployment Host
+---------------------------
+
+RabbitMQ requires that the system's hostname resolves to the IP address that it
+has been configured to use, which with bifrost will be ``127.0.0.1``. Bifrost
+will attempt to modify ``/etc/hosts`` on the deployment host to ensure that
+this is the case. Docker bind mounts ``/etc/hosts`` into the container from a
+volume. This prevents atomic renames which will prevent Ansible from fixing
+the
+``/etc/hosts`` file automatically.
+
+To enable bifrost to be bootstrapped correctly add an entry to ``/etc/hosts``
+resolving the deployment host's hostname to ``127.0.0.1``, for example::
ubuntu@bifrost:/repo/kolla$ cat /etc/hosts
127.0.0.1 bifrost localhost
@@ -40,50 +96,70 @@ hosts hostname to 127.0.0.1 line for example:
ff02::3 ip6-allhosts
192.168.100.15 bifrost
+Build a Bifrost Container Image
+===============================
-Enable source build type
-========================
+This section provides instructions on how to build a container image for
+bifrost using kolla.
-Via config file
----------------
+Enable Source Build Type
+------------------------
-::
+Currently kolla only supports the source install type for the bifrost image.
+Configuration File
+~~~~~~~~~~~~~~~~~~
+
+If required, generate a default configuration file for ``kolla-build``::
+
+ cd kolla
tox -e genconfig
-Modify ``kolla-build.conf`` as follows.
-Set ``install_type`` to ``source``
-
-::
+Modify ``kolla-build.conf``, setting ``install_type`` to ``source``::
install_type = source
Command line
-------------
+~~~~~~~~~~~~
-Alternatively if you do not wish to use the ``kolla-build.conf``
-you can enable a source build by appending ``-t source`` to
-your ``kolla-build`` or ``tools/build.py`` command.
+Alternatively, instead of using ``kolla-build.conf``, a source build can be
+enabled by appending ``--type source`` to the ``kolla-build`` or
+``tools/build.py`` command.
-Build container
-===============
+Build Container
+---------------
Development
------------
+~~~~~~~~~~~
::
+ cd kolla
tools/build.py bifrost-deploy
Production
-----------
+~~~~~~~~~~
::
kolla-build bifrost-deploy
+.. note::
+
+ By default kolla-build will build all containers using CentOS as the base
+ image. To change this behavior, use the following parameter with
+ ``kolla-build`` or ``tools/build.py`` command::
+
+ --base [ubuntu|centos|oraclelinux]
+
+Configure and Deploy a Bifrost Container
+========================================
+
+This section provides instructions for how to configure and deploy a container
+running bifrost services.
+
Prepare Kolla-Ansible Inventory
-===============================
+-------------------------------
Kolla-ansible will deploy bifrost on the hosts in the ``bifrost`` Ansible
group. In the ``all-in-one`` and ``multinode`` inventory files, a ``bifrost``
@@ -100,85 +176,131 @@ services deployed by kolla including OpenStack Ironic, MariaDB, RabbitMQ and
(optionally) OpenStack Keystone. These services should not be deployed on the
host on which bifrost is deployed.
-Prepare bifrost configs
-=======================
+Prepare Kolla-Ansible Configuration
+-----------------------------------
-Create servers.yml
-------------------
+Follow the instructions in :doc:`quickstart` to prepare kolla-ansible's global
+configuration file ``globals.yml``. For bifrost, the
+``bifrost_network_interface`` variable should be set to the name of the
+interface that will be used to provision bare metal cloud hosts if this is
+different than ``network_interface``. For example to use ``eth1``:
-The ``servers.yml`` will describing your physical nodes and list IPMI
-credentials. See bifrost dynamic inventory examples for more details.
+.. code-block:: yaml
-For example ``/etc/kolla/config/bifrost/servers.yml``
+ bifrost_network_interface: eth1
+
+Note that this interface should typically have L2 network connectivity with the
+bare metal cloud hosts in order to provide DHCP leases with PXE boot options.
+
+Since bifrost only supports the source image type, ensure that this is
+reflected in ``globals.yml``
+
+.. code-block:: yaml
+
+ kolla_install_type: source
+
+Prepare Bifrost Configuration
+-----------------------------
+
+Kolla ansible custom configuration files can be placed in a directory given by
+the ``node_custom_config`` variable, which defaults do ``/etc/kolla/config``.
+Bifrost configuration files should be placed in this directory or in a
+``bifrost`` subdirectory of it (e.g. ``/etc/kolla/config/bifrost``). Within
+these directories the files ``bifrost.yml``, ``servers.yml`` and ``dib.yml``
+can be used to configure Bifrost.
+
+Create a Bifrost Inventory
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The file ``servers.yml`` defines the bifrost hardware inventory that will be
+used to populate Ironic. See the `bifrost dynamic inventory examples
+<https://github.com/openstack/bifrost/tree/master/playbooks/inventory>`_ for
+further details.
+
+For example, the following inventory defines a single node managed via the
+Ironic ``agent_ipmitool`` driver. The inventory contains credentials required
+to access the node's BMC via IPMI, the MAC addresses of the node's NICs, an IP
+address to configure the node's configdrive with, a set of scheduling
+properties and a logical name.
.. code-block:: yaml
---
cloud1:
- uuid: "31303735-3934-4247-3830-333132535336"
- driver_info:
- power:
- ipmi_username: "admin"
- ipmi_address: "192.168.1.30"
- ipmi_password: "root"
- nics:
- -
- mac: "1c:c1:de:1c:aa:53"
- -
- mac: "1c:c1:de:1c:aa:52"
- driver: "agent_ipmitool"
- ipv4_address: "192.168.1.10"
- properties:
- cpu_arch: "x86_64"
- ram: "24576"
- disk_size: "120"
- cpus: "16"
- name: "cloud1"
+ uuid: "31303735-3934-4247-3830-333132535336"
+ driver_info:
+ power:
+ ipmi_username: "admin"
+ ipmi_address: "192.168.1.30"
+ ipmi_password: "root"
+ nics:
+ -
+ mac: "1c:c1:de:1c:aa:53"
+ -
+ mac: "1c:c1:de:1c:aa:52"
+ driver: "agent_ipmitool"
+ ipv4_address: "192.168.1.10"
+ properties:
+ cpu_arch: "x86_64"
+ ram: "24576"
+ disk_size: "120"
+ cpus: "16"
+ name: "cloud1"
-adjust as appropriate for your deployment
+The required inventory will be specific to the hardware and environment in use.
-Create bifrost.yml
-------------------
-By default kolla mostly use bifrosts default playbook values.
-Parameters passed to the bifrost install playbook can be overridden by
-creating a ``bifrost.yml`` file in the kolla custom config directory or in a
-bifrost sub directory.
-For example ``/etc/kolla/config/bifrost/bifrost.yml``
+Create Bifrost Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-::
+The file ``bifrost.yml`` provides global configuration for the bifrost
+playbooks. By default kolla mostly uses bifrost's default variable values.
+For details on bifrost's variables see the bifrost documentation.
- mysql_service_name: mysql
- ansible_python_interpreter: /var/lib/kolla/venv/bin/python
- network_interface: < add you network interface here >
- # uncomment below if needed
- # dhcp_pool_start: 192.168.2.200
- # dhcp_pool_end: 192.168.2.250
- # dhcp_lease_time: 12h
- # dhcp_static_mask: 255.255.255.0
+For example:
-Create Disk Image Builder Config
---------------------------------
-By default kolla mostly use bifrosts default playbook values when
-building the baremetal os image. The baremetal os image can be customised
-by creating a ``dib.yml`` file in the kolla custom config directory or in a
-bifrost sub directory.
-For example ``/etc/kolla/config/bifrost/dib.yml``
+.. code-block:: yaml
-::
+ mysql_service_name: mysql
+ ansible_python_interpreter: /var/lib/kolla/venv/bin/python
+ enabled_drivers: agent_ipmitool,agent_ipminative
+ # uncomment below if needed
+ # dhcp_pool_start: 192.168.2.200
+ # dhcp_pool_end: 192.168.2.250
+ # dhcp_lease_time: 12h
+ # dhcp_static_mask: 255.255.255.0
- dib_os_element: ubuntu
+Create Disk Image Builder Configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The file ``dib.yml`` provides configuration for bifrost's image build
+playbooks. By default kolla mostly uses bifrost's default variable values when
+building the baremetal OS and deployment images, and will build an
+**Ubuntu-based** image for deployment to nodes. For details on bifrost's
+variables see the bifrost documentation.
+
+For example to use the ``debian`` Disk Image Builder OS element:
+
+.. code-block:: yaml
+
+ dib_os_element: debian
+
+See the `diskimage-builder documentation
+<http://docs.openstack.org/developer/diskimage-builder/>`__ for more details.
Deploy Bifrost
-=========================
+--------------
-Ansible
--------
+The bifrost container can be deployed either using kolla-ansible or manually.
+
+Kolla-Ansible
+~~~~~~~~~~~~~
Development
___________
::
+ cd kolla-ansible
tools/kolla-ansible deploy-bifrost
Production
@@ -189,16 +311,19 @@ __________
kolla-ansible deploy-bifrost
Manual
-------
+~~~~~~
Start Bifrost Container
_______________________
+
::
- docker run -it --net=host -v /dev:/dev -d --privileged --name bifrost_deploy kolla/ubuntu-source-bifrost-deploy:3.0.1
+ docker run -it --net=host -v /dev:/dev -d \
+ --privileged --name bifrost_deploy \
+ kolla/ubuntu-source-bifrost-deploy:3.0.1
-Copy configs
-____________
+Copy Configuration Files
+________________________
.. code-block:: console
@@ -207,39 +332,40 @@ ____________
docker cp /etc/kolla/config/bifrost/bifrost.yml bifrost_deploy:/etc/bifrost/bifrost.yml
docker cp /etc/kolla/config/bifrost/dib.yml bifrost_deploy:/etc/bifrost/dib.yml
-Bootstrap bifrost
+Bootstrap Bifrost
_________________
::
docker exec -it bifrost_deploy bash
-Generate ssh key
-~~~~~~~~~~~~~~~~
+Generate an SSH Key
+___________________
::
ssh-keygen
-Source env variables
-~~~~~~~~~~~~~~~~~~~~
+Bootstrap and Start Services
+____________________________
-::
-
- cd /bifrost
- . env-vars
- . /opt/stack/ansible/hacking/env-setup
- cd playbooks/
-
-
-Bootstrap and start services
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. code-block:: console
- ansible-playbook -vvvv -i /bifrost/playbooks/inventory/localhost /bifrost/playbooks/install.yaml -e @/etc/bifrost/bifrost.yml
+ cd /bifrost
+ ./scripts/env-setup.sh
+ . env-vars
+ cat > /etc/rabbitmq/rabbitmq-env.conf << EOF
+ HOME=/var/lib/rabbitmq
+ EOF
+ ansible-playbook -vvvv \
+ -i /bifrost/playbooks/inventory/target \
+ /bifrost/playbooks/install.yaml \
+ -e @/etc/bifrost/bifrost.yml \
+ -e @/etc/bifrost/dib.yml \
+ -e skip_package_install=true
-Check ironic is running
-=======================
+Validate the Deployed Container
+===============================
.. code-block:: console
@@ -257,80 +383,93 @@ Running "ironic node-list" should return with no nodes, for example
+------+------+---------------+-------------+--------------------+-------------+
+------+------+---------------+-------------+--------------------+-------------+
-
Enroll and Deploy Physical Nodes
================================
-Ansible
--------
+Once we have deployed a bifrost container we can use it to provision the bare
+metal cloud hosts specified in the inventory file. Again, this can be done
+either using kolla-ansible or manually.
+
+Kolla-Ansible
+-------------
Development
-___________
+~~~~~~~~~~~
::
tools/kolla-ansible deploy-servers
Production
-__________
+~~~~~~~~~~
::
kolla-ansible deploy-servers
-
Manual
------
+
.. code-block:: console
docker exec -it bifrost_deploy bash
cd /bifrost
. env-vars
export BIFROST_INVENTORY_SOURCE=/etc/bifrost/servers.yml
- ansible-playbook -vvvv -i inventory/bifrost_inventory.py enroll-dynamic.yaml -e "ansible_python_interpreter=/var/lib/kolla/venv/bin/python" -e network_interface=<provisioning interface>
+ ansible-playbook -vvvv \
+ -i /bifrost/playbooks/inventory/bifrost_inventory.py \
+ /bifrost/playbooks/enroll-dynamic.yaml \
+ -e "ansible_python_interpreter=/var/lib/kolla/venv/bin/python" \
+ -e @/etc/bifrost/bifrost.yml
docker exec -it bifrost_deploy bash
cd /bifrost
. env-vars
export BIFROST_INVENTORY_SOURCE=/etc/bifrost/servers.yml
- ansible-playbook -vvvv -i inventory/bifrost_inventory.py deploy-dynamic.yaml -e "ansible_python_interpreter=/var/lib/kolla/venv/bin/python" -e network_interface=<prvisioning interface> -e @/etc/bifrost/dib.yml
+ ansible-playbook -vvvv \
+ -i /bifrost/playbooks/inventory/bifrost_inventory.py \
+ /bifrost/playbooks/deploy-dynamic.yaml \
+ -e "ansible_python_interpreter=/var/lib/kolla/venv/bin/python" \
+ -e @/etc/bifrost/bifrost.yml
-At this point ironic should clean down your nodes and install the default
-os image.
+At this point Ironic should clean down the nodes and install the default
+OS image.
-Advanced configuration
+Advanced Configuration
======================
-Bring your own image
+Bring Your Own Image
--------------------
+
TODO
-Bring your own ssh key
+Bring Your Own SSH Key
----------------------
-To use your own ssh key after you have generated the ``passwords.yml`` file
-update the private and public keys under bifrost_ssh_key.
+
+To use your own SSH key after you have generated the ``passwords.yml`` file
+update the private and public keys under ``bifrost_ssh_key``.
Known issues
============
SSH daemon not running
----------------------
-By default sshd is installed in the image but may not be enabled.
-If you encounter this issue you will have to access the server physically in
-recovery mode to enable the ssh service. If your hardware supports it, this
-can be done remotely with ipmitool and serial over lan. For example
+
+By default ``sshd`` is installed in the image but may not be enabled. If you
+encounter this issue you will have to access the server physically in recovery
+mode to enable the ``sshd`` service. If your hardware supports it, this can be
+done remotely with ``ipmitool`` and Serial Over LAN. For example
.. code-block:: console
ipmitool -I lanplus -H 192.168.1.30 -U admin -P root sol activate
-
References
==========
-Docs: http://docs.openstack.org/developer/bifrost/
+Bifrost documentation: http://docs.openstack.org/developer/bifrost/
-Troubleshooting: http://docs.openstack.org/developer/bifrost/troubleshooting.html
+Bifrost troubleshooting guide: http://docs.openstack.org/developer/bifrost/troubleshooting.html
-Code: https://github.com/openstack/bifrost
+Bifrost code repository: https://github.com/openstack/bifrost