Currently elastic-recheck is hammering the ElasticSearch cluster every
15 minutes, basically bringing everything else to a halt while these
queries are processed. Spread elastic-recheck periodic queries out, run
each once an hour and 20 minutes after the previous set of queries.
Also, remove the auto triggering when files change, instead we wait for
the hour to roll over and give us new data.
Change-Id: Ibf5dad1071f65f8d676e8b9a0ec4331bada9943f
We want to display the hits for all queues and for just the gate queue.
A parameter was added to elastic-recheck-graph to support this, and a
second patch updates the web page to give the gate queue only option.
This patch updates the usage of elastic-recheck-gate.
Related elastic-recheck patchs:
* I9217a2ceedf86ffe04851084df78238384fccd51
* I23ed8d945c9517221622e1ca215c80539bb07bbd
Change-Id: Iac51aeaf152f50f8b9a064cd249b0920bef824df
puppet exec doesn't run under a real shell, so running cd doesn't
work because it's not a command. Instead use cwd properly to
actually run these commands.
Also actually call the correct command to generate the uncategorized
jobs page.
Change-Id: I84ed19bbdc16d185316e3967efbd798e06a521d1
this adds the new elastic-recheck-unclassified report, which
generates an html page of all the unclassified gate failures.
it also runs this command, and the elastic-recheck-graph command
not only on cron, but also on updates to the repository. In order
to do this safely, we create a new er_safe_run.sh in /usr/local
that provides a flock front end to ensure we won't race on our
html writes.
Change-Id: I624f2629e2ac19a1394866f73f1aa9ba18e968ce
The installation of elastic-recheck is done on multiple machines but the
irc bot is only run on one machine. Therefore don't notify the bot
service from the install exec instead subscribe to the exec from the
service.
Change-Id: I11f3d32e40408a02d948d0533f5a4c45c983adae
this provides the changes needed to handle the new elastic-recheck
web console installed from the elastic-recheck package in the
share tree.
factor out bot code to elastic_recheck::bot
this change moves elastic_recheck bot starting code out into
a separate module, which lets us include all the elastic_recheck
base module into the static site config.
Change-Id: If53523754494a746c152c5d1384db5426b096fc1
this provides the changes needed to handle the new elastic-recheck
web console installed from the elastic-recheck package in the
share tree.
Change-Id: I8c8b76cb53b2f2eea026eb0f321e48903df71636
in I86317fcf6f1886ab5b6c0ee154b29e71865c52b7 we want to move to
a directory of queries instead of a single file. In order to
support this we need to support linking in that directory as well.
This needs to land prior to that change.
Change-Id: I3af42cff1050754c6844db994004fd203063bb3d
The pip vs. easy_install interactions continually get the systems
into a weird state. pip install $path_to_repo, on the other hand,
works like a charm.
Left out two graphite installs because they are doing path manipulation
and I want to do further testing before touching that.
Change-Id: I373b29aca03f1ebd20e533ebaaf5de9ef2db017e
this ensures we notify the er daemon to restart after we've updated
the code, making it so we deploy instantly into the environment.
Change-Id: Ifa266c7c0248ce9d0431f44e76de87958503f401
* manifests/site.pp: Expect the SSH key in /home rather than in
/var/run.
* modules/elastic_recheck/manifests/init.pp: Change the recheck
shell account for elastic-recheck to use a homedir in /home.
Change-Id: I43f4a8cdcb803328388b7e92732e5cb9d6acb2a8
* modules/elastic_recheck/files/elastic-recheck.init: Minor
whitespace cleanup for space-before-tab indent on a comment line.
* modules/elastic_recheck/files/logging.config: New logging
configuration file for elastic-recheck IRC bot.
* modules/elastic_recheck/manifests/init.pp: Install the new logging
configuration file.
* modules/elastic_recheck/templates/elastic-recheck.conf.erb: Tell
the IRC bot to use the new logging configuration file.
Change-Id: I47cabb01df209c9f020846bcb272b9a2f796b0e5
* modules/elastic_recheck/files/elastic-recheck.init: Update the
daemon invocation now that it has an executable entrypoints wrapper.
* modules/elastic_recheck/manifests/init.pp: Correct the Puppet
dependency tree sufficiently to be able to apply the manifest
successfully.
Change-Id: Ib018564e66c29d73b30aaf50c9306dc6d67ba2b2
This elastic recheck thing that classifies failures based on logstash
queries is awesome. We should run it on the OpenStack developer
infrastructure. Deploy it on logstash.o.o to keep related services
together.
Change-Id: I16ae7a32c2c48474454df17931779d0f60c3ed85
