1036 Commits

Author SHA1 Message Date
James Denton
e6f12e2069 Add support for OVN in MNAIO deploy
This patch adds support for deploying the ML2/Open Virtual Network (OVN)
plugin for Neutron to an MNAIO deployment. A new var,
osa_enable_networking_ovn, can be set to lay down the appropriate bits.

Change-Id: Ib5bd4e0c20be62ddbf0bff13c91d1918907bf230
2019-01-28 02:14:39 +00:00
Kevin Carter
a1d6ebe4d3 remove dynamic ns.enable generators
The ns.enabled generators will fail when running packetbeat with a limit.
These generators were dynamically enabling/disabling packetbeat features
based on things discovered in the environment however they we're
attempting to be a little to fancy, especially when running packetbeat
in a non-osa cloud. The values for the services have been reset to the
provider defaults and should teh deployer want to configure these option
they can use config_template.

Change-Id: I36d7298ca5142e8b5f926ab5d59ab8283704b5af
Signed-off-by: Kevin Carter <kevin@cloudnull.com>
2019-01-10 16:00:37 -06:00
Kevin Carter
7491b6df8e Update the embedded-ansible-setup process to be configurable
This change allows the embedded ansible process to be configurable by
the end user.
  * Python requirements and ansible roles will all now be user
    configurable.
  * Setup is now a local only playbook. This playbook replaces the bash
    commands we were rerunning when the `bootstrap-embedded-ansible.sh`
    script was executed.
  * Embedded ansible version is now 2.7.5 as default.
  * Deprecation warnings have been resolved.
  * Tests impacted by this change have been updated.

Change-Id: I4303c44e249cda31457a4f05a681e298d225a8b7
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2019-01-04 11:46:19 -06:00
Kevin Carter
b1232aead5
Add pretasks to exit quick when needed
The journalbeat playbook uses conditionals to know when to deploy the
journalbeat collector. This change makes it so the playbook simply exits
when the journal is not found or the environment being deployed is not
using systemd. This change will result in faster deployments in mixed
environments as the role will no longer need to iterate over its
conditional.

Change-Id: I581b61902723f54237623036566a83c9be79210e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2019-01-03 09:05:32 -06:00
Kevin Carter
5586d8a80f Convert template setup to a role
This change reduces code throughout the playbooks thereby speeding up
the task execution.
  * A new role named `elastic_beat_setup` was created to
    facilitate template setup as needed.
  * Beats retention policies are now defined on the elastic-logstash
    nodes instead of on all target hosts. This method will speed-up
    deployments on massive installations while streamlining all deployments.
  * Kibana variable assumptions have been fixed. This will allow for
    deployments without Kibana to be accomplished.

Change-Id: I36343264042e81dfcb68bad0f6c3a503e525eceb
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2019-01-02 20:38:47 -06:00
Kevin Carter
9a896aa81a Fence options before casting to json
These options could be "undefined" which is an object and not json
serializable. This change ensures if an option is undefined it defaults
to an empty set which will allow the option to be json serialized.

Change-Id: I1a81bafa441aa6400bfbec50d57e56df4d09bda3
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-12-20 17:38:18 -06:00
Kevin Carter
eb3bcb8daa Add local facts for template creations
The template creates can take a lot of time when dealing with large data
sets. This change makes it so template create will only happen on a
greenfield installation or when upgrading. A simple rerun of the
playbooks will not trigger template creations which will allow deployers
the ability to better change or modify deploymens without having to
worry about extended runtimes due to template interactions.

Change-Id: Ia9b77277553fbdbe0444737f39ec3de75f07cc0f
2018-12-19 17:59:21 -06:00
Kevin Carter
ad91d5773e Extend auto change detection
This change makes it possible for a deployer to modify the set of
indexes and weights assosiated with them. If modified, the local
facts will be automatically updated.

Change-Id: Iaea1f22d8aad2abdd02801dd9acad5f969b78d0e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-12-19 16:15:28 -06:00
Zuul
f7552334ba Merge "Add missing prometheus port for ceph auto discover" 2018-12-19 16:25:22 +00:00
Zuul
8ce606a64d Merge "1.modify the http link to https links 2.modify the misspelling world" 2018-12-19 16:16:30 +00:00
Zuul
238d99c36c Merge "Change openstack-dev to openstack-discuss" 2018-12-19 16:04:08 +00:00
Zuul
f025ab9b37 Merge "Update link address" 2018-12-19 16:04:07 +00:00
Zuul
950f4ac390 Merge "Better handling for resuming leapfrog upgrade" 2018-12-19 16:04:06 +00:00
Zuul
6a46e8f6ac Merge "Updated the connection plugins and add env options" 2018-12-19 16:01:13 +00:00
Michael Vollman
0ea548f979 Add missing prometheus port for ceph auto discover
When ceph prometheus metrics are auto discovered the metricbeat config
should point to the ceph mgr prometheus port. Adding missing
brackets around metricset so the default is treated as an array.
Dropping ceph dir detection for prometheus auto discover and relying on
is the port availability and inventory group only.

Change-Id: Iaba0fdece00414e17bc172f39e624374a9d273e8
2018-12-19 09:56:42 -05:00
cloudnull
c2514c00cd Updated the connection plugins and add env options
These changes allow users to leverage newer versions of ansible with
sane defaults providing a consistent user experience. Things like
retries, host key checking, and pipelinning are not generally set
by default and when left unset these options can create an
inconsistent experience with intermittent failures.

The plugins repo checkout was updated to resolve issues with newer
versions of ansible requiring SFTP options to be set.

** The cluster jobs have been commented out for the time being.
   we're seing lots of timeouts with them and it doesn't make
   sense to keep consuming resources for jobs that are known
   fail due to timeout issues.

Change-Id: Ia103b2b31b7dcd540db977ae7c2d486c80bbdffe
Signed-off-by: cloudnull <kevin.carter@rackspace.com>
2018-12-19 08:52:49 -06:00
Kevin Carter
e08c58dd15 modify fact gathering to use local facts
This change makes the retention gathering operation faster by storing
the retention values as "local facts". The local facts are then
referenced in templates loading from the local fact file instead of
running repetitive queries which are slow making very large deployments
cumbersome. To make the retention policy fact gathering process smarter
it will now automatically refresh if undefined or should the
elasticsearch cluster size change. This will ensure we're improving
speed of execution while also catering to the needs of deployers to
grow, or shrink, elasticsearch cluster sizes.

Documentation has been added regarding the new option and why it may be
of use to depoyers.

Change-Id: I3936ee94461ac39fb8bc78dc2c873e6067552461
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-12-19 01:37:06 -06:00
Zuul
2923df2295 Merge "Update mailinglist from dev to discuss" 2018-12-18 10:34:08 +00:00
Antony Messerli
6c101e313c Better handling for resuming leapfrog upgrade
Copies the openstack-release file to another location
so that the original version being upgraded can be referenced
in the event of a failure where openstack-setup-hosts has
replaced the openstack-release file.  That way the list
of versions to upgrade is correct upon re-running the
leapfrog upgrade script.

Change-Id: I63de16bf27813dd623e996a5ac5de7bb8f8668b6
2018-12-12 17:14:12 -06:00
Matthew Thode
d07df2c513
Chage cpu model to allow for nested virt to work
vmx flag was not passed in to the guest, so test VMs failed to boot on
compute2.  With host-passthrough the cpu flags are unfiltered.

Change-Id: I1a4607ba5bc96c92512a2f4aa43df51f7eef6b27
2018-12-12 12:22:36 -06:00
Jesse Pretorius
bd82c6a3e5 MNAIO: Recovery the galera cluster with no gvwstate.dat file
On occasion, a set of saved images may not have a gvwstate.dat file.
Given that the MNAIO tooling is for test purposes only, we can take
the risk of just making one up in order to ensure that the cluster
starts on boot. DO NOT DO THIS IN PRODUCTION!

Change-Id: I1b055ef0d02d77afe03d1a90baa2e6890e986c32
2018-12-12 12:51:45 +00:00
Zuul
3bb90710fe Merge "MNAIO: If there is no secondary data disk, use the root disk" 2018-12-11 15:30:11 +00:00
Zuul
aa3da6ba00 Merge "MNAIO: Fix warning for guestfish workaround task" 2018-12-11 15:28:00 +00:00
ZhongShengping
7524891371 Update mailinglist from dev to discuss
openstack-dev was decomissioned this night in https://review.openstack.org/621258
Update openstack-dev to openstack-discuss

Change-Id: I8927e9a27ad9b249639955c6bd44ba4f86a3c0fc
2018-12-05 09:16:43 +08:00
ShangXiao
e436a483da Add bugs url link to README
Change-Id: I0f8b4f5073aa96340dd2a779d9ae06af388918c8
2018-12-04 10:02:13 +00:00
98k
e009329d90 Change openstack-dev to openstack-discuss
Mailinglists have been updated. Openstack-discuss replaces openstack-dev.

Change-Id: Ia36318adb868d5939e6deee5c16a016fa0d14a92
2018-12-04 04:02:28 +00:00
Zuul
553c5a7454 Merge "Remove non-existant 'functional' test" 2018-11-30 10:17:48 +00:00
Zuul
c67f4460f4 Merge "add short name for distros" 2018-11-30 04:51:09 +00:00
Zuul
2d4548f273 Merge "Update elk_6x for 6.5.x" 2018-11-29 23:06:35 +00:00
cloudnull
d4fd3c6746 add short name for distros
Change-Id: Ia5babcafa1a7a58bdf8a4f5961341a840ca1c105
Signed-off-by: cloudnull <kevin.carter@rackspace.com>
2018-11-29 11:27:41 -06:00
Jesse Pretorius
f8e94f75c8 Remove non-existant 'functional' test
There is no 'functional' tox env, so the 'functional'
test run by zuul re-runs lint again. Rather than waste
resources we remove the common template and just add
the linters job.

Change-Id: Ibb5152b4d7f731aebbeab46ac4ccd38658cd30a1
2018-11-28 16:44:28 +00:00
Antony Messerli
2ec3466ad8 Allow the option to specify OSA git repo URL
Given the leapfrog branches are all EOL and frozen, an operator
may need to make changes to branches on a local repo in order to combat
bit rot over time.

Change-Id: Ia8214a273b38418808b063b0192bbf81bf455daf
2018-11-28 10:35:56 -06:00
Jesse Pretorius
97f20c5e9d MNAIO: If there is no secondary data disk, use the root disk
In order to allow testing within OpenStack-CI, we need to cater
for the ability to run the tooling without a secondary data disk.

This patch changes the tooling to try the detection, and if there
is no secondary disk it will just use the root disk for /data/images.

Change-Id: I56d76e68b6914b3e12948b022e0e702557684c50
2018-11-28 13:41:02 +00:00
Jesse Pretorius
daf758775c MNAIO: Fix warning for guestfish workaround task
This patch resolves the warning about using jinja braces
the conditional for the guestfish workaround task. While
doing so, the task is made idempotent and we also ensure
that the task that follows exits immediately on any errors.

Change-Id: I7ec254a96312e07229b0a5c799d4a98b5d9c8957
2018-11-28 13:34:35 +00:00
Zuul
820aeb6a60 Merge "MNAIO: Add ability to deploy Ceph" 2018-11-28 10:26:00 +00:00
Kevin Carter
3a69a1c43d Update elk_6x for 6.5.x
This change updates the roles / playbooks to begin using Elasticsearch
relesae 6.5.x. Core to this change is the conversion of the journalbeat
role from custom compiled go, to simple package install which was made
possible by the folks at elastic within this release. Because of the
conversion the "beats-community" playbook has been removed given its now
empty.

A change to the bootstrap script was made allowing it to parse an OS id
with a "-" in it, like "opensuse-tumbleweed".

Change-Id: Ic9b80234d6a6ce876bff885f3223874602d55dd6
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-11-27 18:56:22 +00:00
Jacob Wagner
6b36fd8825 MNAIO: Add ability to deploy Ceph
This commit enables overriding the default storage options of
cinder/swift to a Ceph backed storage for the multinode all in one
deployments.

We also correct the README and build.sh script to show/use the
current defaults correctly for VM_DISK_SIZE and INFRA_VM_SERVER_RAM.

Change-Id: I9e1f1b09d1bcf224f4afa765c585baf28e6cafa8
2018-11-27 15:51:46 +00:00
MaoyangLiu
a9fd3214aa 1.modify the http link to https links
2.modify the misspelling world

Change-Id: Icd92e680cd5f313c8294acd59199948872369503
2018-11-25 04:17:18 +00:00
zhangdebo
15788c42ba Update link address
Change-Id: I31cd17ef8874f0e11522f4ddd3e6750898b6c232
2018-11-24 14:01:46 +08:00
Kevin Carter
9af219910e
reduce container count on cluster test
Change-Id: Id70967834f43009e6c91fb71877606118242a429
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-11-21 23:19:28 -06:00
Kevin Carter
6797da6b1e
Correct clustered gating job
The cluster gatting job was failing due to memory locking, which is
impossible due to limited resources in the CI environment. This change
disables memory locking in the clustered gate test.

Change-Id: I0a146c41a1b82425539e014b1baee2011d464e05
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-11-20 10:25:19 -06:00
Michael Vollman
3f70fc76d9 Add auto discover for ceph prometheus metrics
Add to elk_metrics_6x the capability to auto discover the ceph mgr
prometheus plugin metric port.  Pull ceph metrics from prometheus when
the plugin is enabled.

Change-Id: I530a99f42e396ba7b2cd2c1b3d587f528ef84242
2018-11-09 12:42:28 -05:00
Michael Vollman
bc180319ba Metricbeat role check for a ceph api to monitor
Before enabling the metricbeat ceph module, first check that the
restapi port is up and responding.

Change-Id: Ic795df02b93ca22c19fe67d6d2319889dc0f06a0
2018-11-08 15:50:50 +00:00
Michael Vollman
b5ed2d024f Fix ansible path for elk_metrics_6x tests
Fix the failing elk_metrics_6x tests by changing the ansible venv path
to ~/ansible_venv.  Update tests to ensure when the bootstrap ansible
script changes both osquery and elk_metrics_6x tests are run.

Change-Id: Ic79c52ae21294373ca67be264fd287fdf32b4c9a
2018-11-08 09:45:30 -06:00
Dave Wilde
58c7890bfc MNAIO: flat networking fixes
With the switch to systemd-networkd the veth pairs for flat networking
and lbaas aren't created.  This creates veth pairs if the server has the
appropriate server_networks defined.

* Create the veth network files

With this simple Match directive networkd will bring the interface up.

Change-Id: I0ae4375be128594a861c18aa33a98378859dd9d3
2018-11-01 07:56:21 -05:00
tonytan4ever
27aeb36389 Parameterize ansible version in embedded runtime
Change-Id: Ie880e10d4915c06d86084b60cbb8d994d5d9965d
2018-10-24 14:19:19 -04:00
Jesse Pretorius
05e235859d MNAIO: Implement branch var for pinning Ansible & Plugins
Due to [1] the task 'Create data cinder-volumes group' is currently
failing when run on a restored image due to the idempotence failure.

In this patch we enable pinning the requirements file and plugins
repo to a different OSA branch, and set it to stable/rocky which is
currently pinned to Ansible 2.5.10 and does not exhibit this issue.

[1] https://github.com/ansible/ansible/issues/47301

Change-Id: Id0f98f3ea0f53173dae800d45f1a4554c8a0c38a
2018-10-24 10:25:05 +01:00
Jesse Pretorius
1f54688f4c MNAIO: Add legacy os-infra_hosts group back
In https://review.openstack.org/611582 we removed
the legacy group as it has been deprecated since
Newton, however it appears to still be used by some
downstream tests, so we add it back, but make it
only get implemented if the associated services
are enabled.

Change-Id: I477a46d606d75d44a1ecd5bcfcb29c8308c65245
2018-10-23 16:27:52 +00:00
Zuul
19edaabf87 Merge "Update the test scripts" 2018-10-23 14:50:41 +00:00
Zuul
7b87e3b195 Merge "Fixes related to installation on Trusty" 2018-10-23 14:46:08 +00:00