17fb37f075
Most of the changes in this PR are for style and to adapt the playbooks so that the system can operate on a multi-node cloud. Functional change includes the removal of mainline Java 8 in favor of OpenJDK 8. A site playbook was add to allow an operator to just run everything. Old tools that no longer function within the stack have been removed. Packetbeat was added to the install list Auditbeat was added to the install list All of the config files have been updated for the recent ElasticStack 6.x changes. Change-Id: I01200ad4772ff200b9c5c93f8f121145dfb88170 Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
install Elk stack with topbeat to gather metrics
- tags
-
openstack, ansible
Changelog
2018-03-06 Per Abildgaard Toft (per@minfejl.dk): Updated to version Elasticsearch,Logstash and Kibana 6.x. Changed Topebeat (deprecated) to metricbeat. Included haproxy endpoint configuration.
About this repository
This set of playbooks will deploy elk cluster (Elasticsearch, Logstash, Kibana) with topbeat to gather metrics from hosts metrics to the ELK cluster.
Process
Clone the elk-osa repo
cd /opt
git clone https://github.com/openstack/openstack-ansible-opsCopy the env.d file into place
cd openstack-ansible-ops/elk_metrics_6x
cp env.d/elk.yml /etc/openstack_deploy/env.d/Copy the conf.d file into place
cp conf.d/elk.yml /etc/openstack_deploy/conf.d/In elk.yml, list your logging hosts under elastic-logstash_hosts to create the elasticsearch cluster in multiple containers and one logging host under kibana_hosts to create the kibana container
vi /etc/openstack_deploy/conf.d/elk.ymlCreate the containers
cd /opt/openstack-ansible-playbooks
openstack-ansible lxc-containers-create.yml -e 'container_group=elastic-logstash:kibana'install master/data elasticsearch nodes on the elastic-logstash containers
cd /opt/openstack-ansible-ops/elk_metrics_6x
openstack-ansible installElastic.ymlInstall Logstash on all the elastic containers
cd /opt/openstack-ansible-ops/elk_metrics_6x
openstack-ansible installLogstash.ymlInstall Kibana, nginx reverse proxy and metricbeat on the kibana container
cd /opt/openstack-ansible-ops/elk_metrics_6x
openstack-ansible installKibana.ymlinstall Metricbeat everywhere to start shipping metrics to our logstash instances
cd /opt/openstack-ansible-ops/elk_metrics_6x
openstack-ansible installMetricbeat.ymlOptional | conigure haproxy endpoints
Edit the /etc/openstack_deploy/user_variables.yml file and add fiel following lines
haproxy_extra_services:
- service:
haproxy_service_name: kibana
haproxy_ssl: False
haproxy_backend_nodes: "{{ groups['kibana'] | default([]) }}"
haproxy_port: 81 # This is set using the "kibana_nginx_port" variable
haproxy_balance_type: tcp
- service:
haproxy_service_name: elastic-logstash
haproxy_ssl: False
haproxy_backend_nodes: "{{ groups['elastic-logstash'] | default([]) }}"
haproxy_port: 5044 # This is set using the "logstash_beat_input_port" variable
haproxy_balance_type: tcp
- service:
haproxy_service_name: elastic-logstash
haproxy_ssl: False
haproxy_backend_nodes: "{{ groups['elastic-logstash'] | default([]) }}"
haproxy_port: 9201 # This is set using the "elastic_hap_port" variable
haproxy_check_port: 9200 # This is set using the "elastic_port" variable
haproxy_backend_port: 9200 # This is set using the "elastic_port" variable
haproxy_balance_type: tcpOptional | run the haproxy-install playbook
cd /opt/openstack-ansible/playbooks/
openstack-ansible haproxy-install.yml --tags=haproxy-service-configTrouble shooting
If everything goes bad, you can clean up with the following command
openstack-ansible lxc-containers-destroy.yml --limit=kibana:elastic-logstash_all