From 289feb7f7fdd6287d806cc4f9e5c40e2186725e7 Mon Sep 17 00:00:00 2001
From: erihban <rihab.banday@ericsson.com>
Date: Fri, 12 Jul 2019 15:08:08 +0200
Subject: [PATCH] Adding deployment guide for Openstack Helm with OVS-DPDK
This patch adds a deployment guide for installing Openstack Helm
with OVS-DPDK
Co-Authored-By: Georg Kunz <georg.kunz@ericsson.com>
Change-Id: Ic8078537a7317c4132e4b11494e0d827365109d9
---
.../install/developer/deploy-ovs-dpdk.rst | 189 ++++++++++++++++++
doc/source/install/developer/index.rst | 1 +
2 files changed, 190 insertions(+)
create mode 100644 doc/source/install/developer/deploy-ovs-dpdk.rst
diff --git a/doc/source/install/developer/deploy-ovs-dpdk.rst b/doc/source/install/developer/deploy-ovs-dpdk.rst
new file mode 100644
index 0000000000..82f6199dcd
--- /dev/null
+++ b/doc/source/install/developer/deploy-ovs-dpdk.rst
@@ -0,0 +1,189 @@
+===============
+Deploy OVS-DPDK
+===============
+
+Requirements
+============
+
+A correct DPDK configuration depends heavily on the specific hardware resources
+and its configuration. Before deploying Openvswitch with DPDK, check the amount
+and type of available hugepages on the host OS.
+
+.. code-block:: shell
+
+ cat /proc/meminfo | grep Huge
+ AnonHugePages: 0 kB
+ ShmemHugePages: 0 kB
+ HugePages_Total: 8
+ HugePages_Free: 6
+ HugePages_Rsvd: 0
+ HugePages_Surp: 0
+ Hugepagesize: 1048576 kB
+
+In this example, 8 hugepages of 1G size have been allocated. 2 of those are
+being used and 6 are still available.
+
+More information on how to allocate and configure hugepages on the host OS can
+be found in the `Openvswitch documentation
+<http://docs.openvswitch.org/en/latest/intro/install/dpdk/>`_.
+
+In order to allow OVS inside a pod to make use of hugepages, the corresponding
+type and amount of hugepages must be specified in the resource section of the
+OVS chart's values.yaml:
+
+.. code-block:: yaml
+
+ resources:
+ enabled: true
+ ovs:
+ db:
+ requests:
+ memory: "128Mi"
+ cpu: "100m"
+ limits:
+ memory: "1024Mi"
+ cpu: "2000m"
+ vswitchd:
+ requests:
+ memory: "128Mi"
+ cpu: "100m"
+ limits:
+ memory: "1024Mi"
+ cpu: "2000m"
+ # set resources to enabled and specify one of the following when using dpdk
+ hugepages-1Gi: "1Gi"
+ # hugepages-2Mi: "512Mi"
+
+Additionally, the default configuration of the neutron chart must be adapted according
+to the underlying hardware. The corresponding configuration parameter is labeled with
+"CHANGE-ME" in the script "values_overrides/dpdk.yaml". Specifically, the "ovs_dpdk"
+configuration section should list all NICs which should be bound to DPDK with
+their corresponding PCI-IDs. Moreover, the name of each NIC needs to be unique,
+e.g., dpdk0, dpdk1, etc.
+
+.. code-block:: yaml
+
+ network:
+ interface:
+ tunnel: br-phy
+ conf:
+ ovs_dpdk:
+ enabled: true
+ driver: uio_pci_generic
+ nics:
+ - name: dpdk0
+ # CHANGE-ME: modify pci_id according to hardware
+ pci_id: '0000:05:00.0'
+ bridge: br-phy
+ migrate_ip: true
+ bridges:
+ - name: br-phy
+ bonds: []
+
+In the example above, bonding isn't used and hence an empty list is passed in the "bonds"
+section.
+
+Deployment
+==========
+
+Once the above requirements are met, start deploying Openstack Helm using the deployment
+scripts under the dpdk directory in an increasing order
+
+.. code-block:: shell
+
+ ./tools/deployment/developer/dpdk/<script-name>
+
+One can also specify the name of Openstack release and container OS distribution as
+overrides before running the deployment scripts, for instance,
+
+.. code-block:: shell
+
+ export OPENSTACK_RELEASE=rocky
+ export CONTAINER_DISTRO_NAME=ubuntu
+ export CONTAINER_DISTRO_VERSION=bionic
+
+Note that OVS-DPDK deployment has been tested with Openstack Rocky release and Ubuntu
+Bionic container distributions. If the above variables aren't set, the defaults (currently
+Openstack Ocata and Ubuntu Xenial) will be used.
+
+Troubleshooting
+===============
+
+OVS startup failure
+-------------------
+
+If OVS fails to start up because of no hugepages are available, check the
+configuration of the OVS daemonset. Older versions of helm-toolkit were not
+able to render hugepage configuration into the Kubernetes manifest and just
+removed the hugepage attributes. If no hugepage configuration is defined for
+the OVS daemonset, consider using a newer version of helm-toolkit.
+
+.. code-block:: shell
+
+ kubectl get daemonset openvswitch-vswitchd -n openstack -o yaml
+ [...]
+ resources:
+ limits:
+ cpu: "2"
+ hugepages-1Gi: 1Gi
+ memory: 1Gi
+ requests:
+ cpu: 100m
+ memory: 128Mi
+ [...]
+
+Adding a DPDK port to Openvswitch fails
+---------------------------------------
+
+When adding a DPDK port (a NIC bound to DPDK) to OVS fails, one source of error
+is related to an incorrect configuration with regards to the NUMA topology of
+the underlying hardware. Every NIC is connected to one specific NUMA socket. In
+order to use a NIC as DPDK port in OVS, the OVS configuration regarding
+hugepages and PMD threas needs to match the NUMA topology.
+
+The NUMA socket a given NIC is connected to can be found in the ovs-vswitchd log:
+
+.. code-block::
+
+ kubectl logs -n openstack openvswitch-vswitchd-6h928
+ [...]
+ 2019-07-02T13:42:06Z|00016|dpdk|INFO|EAL: PCI device 0000:00:04.0 on NUMA socket 1
+ 2019-07-02T13:42:06Z|00018|dpdk|INFO|EAL: probe driver: 1af4:1000 net_virtio
+ [...]
+
+In this example, the NIC with PCI-ID 0000:00:04.0 is connected to NUMA socket
+1. As a result, this NIC can only be used by OVS if
+
+1. hugepages have been allocated on NUMA socket 1 by OVS, and
+2. PMD threads have been assigned to NUMA socket 1.
+
+To allocate hugepages to NUMA sockets in OVS, ensure that the
+``socket_memory`` attribute in values.yaml specifies a value for the
+corresponding NUMA socket. In the following example, OVS will use one 1G
+hugepage for NUMA socket 0 and socket 1.
+
+.. code-block::
+
+ socket_memory: 1024,1024
+
+
+To allocate PMD threads to NUMA sockets in OVS, ensure that the ``pmd_cpu_mask``
+attribute in values.yaml includes CPU sockets on the corresponding NUMA socket.
+In the example below, the mask of 0xf covers the first 4 CPU cores which are
+dsitributed across NUMA sockets 0 and 1.
+
+.. code-block::
+
+ pmd_cpu_mask: 0xf
+
+The mapping of CPU cores to NUMA sockets can be determined by means of ``lspci``, for instance:
+
+.. code-block:: shell
+
+ lspci | grep NUMA
+ NUMA node(s): 2
+ NUMA node0 CPU(s): 0,2,4,6,8,10,12,14
+ NUMA node1 CPU(s): 1,3,5,7,9,11,13,15
+
+More information can be found in the `Openvswitch documentation
+<http://docs.openvswitch.org/en/latest/intro/install/dpdk/>`_.
diff --git a/doc/source/install/developer/index.rst b/doc/source/install/developer/index.rst
index 554bf050c0..ed4498a35e 100644
--- a/doc/source/install/developer/index.rst
+++ b/doc/source/install/developer/index.rst
@@ -10,5 +10,6 @@ Contents:
kubernetes-and-common-setup
deploy-with-nfs
deploy-with-ceph
+ deploy-ovs-dpdk.rst
exercise-the-cloud
cleaning-deployment