933 Commits

Author SHA1 Message Date
Jesse Pretorius
acd9e30bde MNAIO: Add instructions for using file-backed images
This patch adds some instructions for downloading and
using the file-backed images after they're saved off
box.

Change-Id: Ia61ea5c36fd7a5d4b37d8f7063e8f87fd82c730e
2018-10-09 20:25:28 +01:00
Kevin Carter
cad0efe3f7 Move the embedded ansible tools to its own subdir
The embedded ansible tooling has uses outside of the elk_metrics_6x
effort. This change moves the tools out so that they can be
updates/modified in an isolated work stream.

Change-Id: I4d9d067d464ac8fd0d1c8b3fdaa7f42480157fc4
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-10-08 16:50:45 +00:00
Zuul
a638bcf721 Merge "Make OSA repo clone optional" 2018-10-06 22:07:05 +00:00
Wayne Warren
ebfe08b9ef Make OSA repo clone optional
I am working on an experimental approach to running the MNAIO playbooks
in a local docker container with osa and osa-ops already checked out and
volume-mounted into the docker container. This change supports that
approach by making the git clone step optional in the deploy-osa.yml
playbook.

Also, don't force update the OSA repo.

Change-Id: I5312e88d4f41033a223d145e72f0767be45adda1
2018-10-04 14:44:51 -05:00
Zuul
9064f8a8fa Merge "multi-noda-aio loadbalancer routing issues fix" 2018-10-03 21:48:20 +00:00
Shannon Mitchell
10502c9bad multi-noda-aio loadbalancer routing issues fix
The loadbalancer vm is configured to utilize br-dhcp for the external
vip and br-mgmt for the internal vip. Users end up accessing these
from various networks depending on the dev/testing going on. For
example if you utilize the flat network for vms that need to hit the
external api, the calls fail. This is due to traffic coming in on
br-dhcp then going back out on br-flat. This change is to remove the
unused interfaces to cut down on routing issues in the load balancer.

Change-Id: Ic3c3bd18e63943ff288692da20d449852a316b5e
Closes-Bug: #1795710
2018-10-02 12:42:53 -05:00
OpenStack Proposal Bot
a4b431d427 Updated from OpenStack Ansible Tests
Change-Id: I1b40882e696d0179f898eab4170117904289e6ca
2018-10-02 14:51:42 +00:00
Victor Palma
3aa5fcd8fc multiple updates to osquery
* move playbooks to roles
   * update documentation
   * update haproxy
     - set 6443 as default port for kolide fleet
   * add galera support

Change-Id: I2fdefcb6bec98486c16b54cf33e2b7940b88d50b
2018-10-01 14:20:08 -05:00
Zuul
b5e99adc01 Merge "fix tox python3 overrides" 2018-09-29 17:56:26 +00:00
OpenStack Proposal Bot
9ec69af8f3 Updated from OpenStack Ansible Tests
Change-Id: I8a15f3840bbb7f6de9b5d41449a7f793476cf9e8
2018-09-29 04:53:18 +00:00
Zuul
5943aa62b5 Merge "Fix auditd logs using Filebeat" 2018-09-28 06:36:26 +00:00
Zuul
fd64af0c76 Merge "Fix blank microversion" 2018-09-28 06:35:26 +00:00
Zuul
6b6c0d0dbe Merge "Add Instance ID to logs" 2018-09-28 06:33:17 +00:00
Zuul
10c4c0a2d0 Merge "[Trivial Fix] Replace Chinese punctuation with English punctuation" 2018-09-28 05:30:11 +00:00
Zuul
c05b21dd26 Merge "Refactor Filebeat configuration file" 2018-09-28 03:04:36 +00:00
Zuul
578b0adc48 Merge "Add changes to the sysconfig defaults file" 2018-09-28 03:02:05 +00:00
inspurericzhang
3af2caebb8 [Trivial Fix] Replace Chinese punctuation with English punctuation
Curly quotes(Chinese punctuation) usually input from Chinese input method.
When read from english context, it makes some confusion.

Change-Id: I1b34eef0913dc0cda1c58d27e8f53ffdcfc3aa22
2018-09-28 09:41:38 +08:00
Mohammed Naser
e5b3b6e688 Fix auditd logs using Filebeat
Filebeat auditd module only works by using Ingest which means that
the dashboards are broken when using Logstash.  This patch adds
some basic information in order for the dashboard to become
functional.

Change-Id: I18283354ea49b26e1716dcab1f6452948e52845f
2018-09-27 18:04:33 -04:00
Mohammed Naser
3b8961beca Fix blank microversion
It is possible that OpenStack has no microversion logged which
would fail to parse.  This fixes that.

Change-Id: Ic0d49541b5863d1bdff3138c36d9de6331a7cafa
2018-09-27 15:48:32 -04:00
Mohammed Naser
4f03c51118 Add Instance ID to logs
This will parse the logs and grab the instance ID out of it.

Change-Id: I9ad0c0e8d6101cca1fc3c4a7cb5cabc3504e6e28
2018-09-27 14:56:52 -04:00
Mohammed Naser
aa647953e0 Refactor Filebeat configuration file
- Avoid checking item by item, we always enable modules and
  prospectors, with an option to disable with opt-in
- Updated MySQL and Apache modules to point to right path
- Improved and clean-up tagging
- All the prospectors are managed using a variable

Change-Id: I2a091669d6a77fd2c89a073cf9071292793e2f6b
2018-09-27 14:54:51 -04:00
Jesse Pretorius
9b0f2b5dbd MNAIO: Ensure VM's are shut down before doing image save
When the virt module returns success for the shutdown state, it
has only sucessfully sent the signal to shut down. It may still
take a few more minutes for the VM's to actually complete their
shut down. If we try to change the image while the machine is
still busy shutting down, the image conversion/compression fails
and the resulting state is incomplete.

In this patch we do the following:

1. Find and shut down the running VM's without needing to look
   at the inventory. This reduces complexity in the play.
2. Makes sure that the VM's are all in the 'shut off' state,
   before continuing on to saving the images.

Change-Id: Icf337447f7a9b4033af261910f77216a170937ed
2018-09-27 18:05:47 +01:00
Kevin Carter
4c86cb9be2
Add changes to the sysconfig defaults file
These changes mirror systemd tunables for elasticsearch and are needed
to ensure any OS without systemd (like Ubuntu 14.04) has the same
capabilities and OS's with systemd. This also adds a specific sysctl
file to use when making sysctl changes. This will ensure we're not
subjecting our deployment to other changes from other sources, like an
OSA playbook run.

Change-Id: Ic0e0bc0f93a12298c1e2f634cf5a1b4c6be2995e
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-09-27 09:44:09 -05:00
Zuul
9237bc3abe Merge "Add additional API tests to confirm functionality" 2018-09-27 04:38:10 +00:00
Doug Hellmann
4fe837b26c fix tox python3 overrides
We want to default to running all tox environments under python 3, so
set the basepython value in each environment.

We do not want to specify a minor version number, because we do not
want to have to update the file every time we upgrade python.

We do not want to set the override once in testenv, because that
breaks the more specific versions used in default environments like
py35 and py36.

Change-Id: Id8547917da6ffae8b4967ff90d9989f79734ce6f
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
2018-09-26 18:42:40 -04:00
Jesse Pretorius
404b9a77db MNAIO: Only adjust user_variables.yml if pre_config_osa is true
If pre_config_osa is not true, then the folder won't exist and
the task will fail.

Change-Id: Idb20e298d4f0f7f4ab175a2810492b9842f30f8b
2018-09-26 17:41:31 +01:00
Kevin Carter
f1423bdfee
Add additional API tests to confirm functionality
The changes here add additional tests to the system to ensure all of
the required APIs are up, available, and are returning data as expected.
To ensure the tests run the `q_mem` override value has been set a little
higher, which will allow logstash to start (before it would result in a
heap size of 115M which is not enough to start logstash).

Change-Id: Ie640f609ef345e9b4908eb17a03be5390b72b186
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-09-26 11:15:24 -05:00
Jesse Pretorius
ddcef522d4 MNAIO: Fix file-backed VM deployment
Recent patches broke the file-backed VM deployment.
This patch fixes it again. It also ensures that the
cleanup of VM's removes any VM's running, rather than
the active inventory.

Change-Id: Ic1cc887bf61162a8c8d96dd53f6db9583c4c089b
2018-09-26 14:51:00 +01:00
Jesse Pretorius
14567e0c3c MNAIO: Remove mnaio_hosts group and use vm_hosts only
The mnaio_hosts group is redundant. We switch to just using the
vm_hosts group instead.

Change-Id: I854befd283b62c98e956a5e677ca9d7dbd33416b
2018-09-26 14:00:57 +01:00
Zuul
49340c566d Merge "MNAIO: Disable metering services by default" 2018-09-25 16:37:17 +00:00
Zuul
4b289c5dc9 Merge "MNAIO: Ensure consistent defaults" 2018-09-25 16:37:16 +00:00
Zuul
605cfb5b06 Merge "MNAIO: Correct README regarding DATA_DISK_DEVICE" 2018-09-25 16:32:56 +00:00
Zuul
10e0f2442f Merge "Make root partition size configurable" 2018-09-25 16:10:01 +00:00
Zuul
439b269525 Merge "Enable setting mnaio disk size by pxe server group" 2018-09-25 16:10:00 +00:00
Zuul
907a5ff7c7 Merge "Use group vars to reduce redundancy in host vars" 2018-09-25 16:10:00 +00:00
Zuul
3195d3bf06 Merge "Change VM definition in deploy-vm mnaio playbook" 2018-09-25 16:09:59 +00:00
Zuul
b2c9cf4221 Merge "Add host metadata to core beats output" 2018-09-25 15:59:17 +00:00
Zuul
27ef77f601 Merge "Add tags for beats setup tasks" 2018-09-25 15:24:03 +00:00
Zuul
a9482e571b Merge "Convert refresh fact to strings" 2018-09-25 15:21:06 +00:00
Jonathan Rosser
1a48236ced Add host metadata to core beats output
This change adds fields such as host OS, version and platform to the
core beats output, giving extra query/filter capabilities.

Change-Id: Iff61bb4402eaa45b8f1c134a6a39cebe6613cbf3
2018-09-25 13:34:18 +01:00
Jonathan Rosser
ac46b2be6a Fix journalbeat installation for mixed environments
The previous code would terminate the play immediatley if any hosts
in the environment did not have a journal directory. This change runs
the journalbeat install role selectively on hosts that have the journal
directory, and skips hosts that do not.

In addition a legacy task to stop the play after uninstallation is removed,
this functionality is currently broken.

Change-Id: I412e3594c4b2292caafafb580bb4ede9ccfd3944
2018-09-25 12:34:21 +01:00
Jonathan Rosser
8cf20bfea2 Add tags for beats setup tasks
Previously the beat setup tasks were tagged with 'setup' but the include
statements were not, so the tasks were always skipped when using '--tags
setup'. This change adds tags to the includes so that the tasks are executed
as expected

Change-Id: If16069cd273d84a22b229b8140e5a8d56eed86d1
2018-09-25 12:30:22 +01:00
Zuul
af4e551c09 Merge "Fix Kibana dashboard uploading for mirrors and proxies" 2018-09-25 03:10:53 +00:00
Zuul
a7c581e8b5 Merge "Add the pip package hvac to support hashicorp vault" 2018-09-25 03:10:27 +00:00
Kevin Carter
dfc919bb0e
Convert refresh fact to strings
While the use of "to_json" resulted in a string it also created a JSON
escaped a string full of slashes which then corrupts kibana index patterns.

Change-Id: I2c26ab9dd4930226f3e554c2f9bed5e382cdafa5
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
2018-09-24 21:53:48 -05:00
Zuul
40fd4f2433 Merge "Remove PrivateDevices" 2018-09-24 22:39:59 +00:00
Zuul
0e4dc162b4 Merge "Increase beat input inactivity timeout" 2018-09-24 20:30:43 +00:00
Zuul
3b086c860a Merge "Clean-up filtering for API requests" 2018-09-24 20:30:43 +00:00
Zuul
144ec7628d Merge "Create filter for contextual logs" 2018-09-24 20:30:42 +00:00
Jonathan Rosser
bc374b8688 Fix Kibana dashboard uploading for mirrors and proxies
To upload the Kibana dashboard behind http proxies we need to

* Point to a local mirror for nodejs if one is defined
* Install nodejs directly with apt rather than the magic script
* Be explicit about no_proxy when uploading the dashboard

This change also uploads the dashboard only once rather than on
each elastic-logstash node.

Change-Id: I4695d6fe6f85d9120f83abc9a92c54ac3ad68c95
2018-09-24 19:55:21 +00:00