It appears quay is now returning their own domain in their blob
redirects. We currently don't cache it so in order for it to work we
need to add cdn01.quay.io and cdn02.quay.io to the proxy config
Change-Id: I2b603d6a5d057e388d473f71bfbaf822d65dd4e1
These volumes were removed with
I050f737521fa6837f3b6b52b8028a839a29f7bd2 but I forgot to remove them
from this list.
Change-Id: I6b7f4a3aef55627d523eca2183379dff15554046
We don't need to keep using the old 2.2 Order, Allow, Deny, Satisfy acl
primitives because we are now running Apache 2.4 everywhere. Stick to
these as they simplify understanding of acls by being consistent.
Change-Id: I9ed4edf15f206006fd79bdef298f8ed101a7a381
Docker has long planned to turn this off and it appears that they have
done so. Planning details can be found at:
https://www.docker.com/blog/registry-v1-api-deprecation/
Removing this simplifies our configs as well as testing. Do this as part
of good hygiene.
Change-Id: I11281167a87ba30b4ebaa88792032aec1af046c1
LXC3 is usable with CentOS 8, while lxc2 is not available for it anymore
So it's worth adding it to reduce network related issues in CI
Change-Id: I562a7d8000ecda8790da88f08128c35b1ec4a2c9
We need to tell apache to listen on the ports used by the Quay Registry
Mirror. Without this we aren't actually able to provide connections to
this vhost.
Add testing to ensure this is working in a simple manner.
Change-Id: I28bdb7aeb9c3252c6319658acaa530a7d7c25a72
Previously we had enabled SSL on our main vhost for the mirrors. Do
similar for all of the proxy cache vhosts for docker and other external
resources.
As part of this change we improve the testing to ensure that the new
vhosts are working as expected. One testing specific change to note is
the testinfra node names did not match our existing system-config-run
job nodenames. This has been corrected.
Additionally RHRegistryMirror and QuayMirror may not be working and
fixing those is left as a followup.
Change-Id: I9dbbd4080c3a2cce4acc39d63244f7a645503553
We've noticed that our mirrors will semi regularly have problems due to
old stale works. For example using old ssl certs or having connection
problems to round robin backend services. In all cases restarting the
service (killing old workers) seems to fix things. Try to force this to
automatically happen by setting a reasonable connection limit per worker
before we recycle them.
Change-Id: Ic377f48d1a5a3eecbcb183327c9255134c4364ab
This is a follow-up on nocanon from [1].
This ensures Apache does not filter any requests originally
containing encoded slashes.
[1] I5a3a6551536e2d1e87aa074e0de7619a367b1971
Change-Id: I94fcb67a914da6ab4d6e1bfd0a2e02121d22559c
Kolla Monasca Grafana images are currently not buildable in CI
due to:
404 Not Found - GET http://mirror.bhs1.ovh.openstack.org:8080/registry.npmjs/@types%2fcolor-name
The url-escaped slash gets mangled by Apache on its way to
registry which causes 404.
This patch fixes that.
Change-Id: I5a3a6551536e2d1e87aa074e0de7619a367b1971
As described in the dependent change, which removes the environment
var to set this up, this is no longer required.
Story: #2006598
Task: #39014
Change-Id: I93455dd1512aeb9111feaf516abfb60695976663
Depends-On: https://review.opendev.org/714543
Ceph Nautilus is released and the official mirror
is available. This adds the Ceph Nautilus mirror
so we can sync it for Stretch and Bionic.
Based on the same change that was done when Mimic
was released [1]
[1] https://review.opendev.org/#/c/571989/
Change-Id: I9424d1f4df58acde8ea70dc16283d4de89189bae
This change adds a proxy config for quay which should assist
us when gating using images provided by the publically
available registry.
Change-Id: I971705e59724e70bd9d42a6920cf4f883556f673
Signed-off-by: Kevin Carter <kecarter@redhat.com>
Default apache mimetypes don't include .log as text/plain; add it.
Log export was added with I67870f6d439af2d2a63a5048ef52cecff3e75275 so
match the .log.1 file that logrotate creates for our rsync mirror logs
too.
Change-Id: Iaf3f19d26f3a6fda7ef3571573af219a31f1dced
This used to be mirrored, however there were issues when upstream
dropped the PC1 repositories a few months back. The puppet openstack
jobs are still trying to leverage this mirror but it does not exist in
some regions because it was disabled on the afs content. This change
fixes the reprepo configuration to still pull down puppet5/6 for xenial
and strech and add the symlink back to the mirrors.
Change-Id: I71ad5afe086a503d75a365543ad8869e35ef873b
This change adds a proxy config for registry.access.redhat which should
assist us when gating using images provided by the publically available
registry.
Change-Id: Ica7477d63659610de852d305a63f3e78d0dd8c4f
Signed-off-by: Kevin Carter <kecarter@redhat.com>
This adds a periodic job to copy logs to a mirror volume, and export
it via the usual mirror http.
I have precreated the log volume; just as a R/W volume because this is
expected to be very low volume access.
Change-Id: I67870f6d439af2d2a63a5048ef52cecff3e75275
This reverts commit b3ce1c52dc7ca455ffd94ea07d8a4fb1b6905fa8.
It removed the AFS mirror at the same time it added the proxy,
but jobs don't know to look for the proxy since it's on a
totally different TCP port.
Change-Id: I87cc03eb3322bd7b093dd6fe798aadb48f319805
This is an intermediate step to having both kafs and openafs testing
in the gate; this just makes it clear which host is which.
Change-Id: I8cd006227ed47ad5f2c5eec664083477dd7ba397
Unfortunately it's not uncommon to be correlating mirror access logs
to remote requests to see what is going on with access failures. It
is *much* easier when everything is in ISO8601 format and not using
strings for month names, etc.
Switch the log format used everywhere to "compact ISO 8601 format,
including micro-seconds". The errorlog format is based on
http://httpd.apache.org/docs/2.4/mod/core.html#errorlogformat
Change-Id: I0aafb831d5c5ca3abf031771d5f59c986611e197
Having proxy_[80|443]_access.log is wrong beacuse they're not really
proxies (I think I just copied this incorrectly). Change it to
mirror_, and update the macro that is only used on the mirror portions
too.
Change-Id: I8eca941fee9606d25dd25bc54bc552ccc7094e0f
The /var/www/mirror/ubuntu -> /afs/openstack.org/mirror/ubuntu symlink
was missing so we weren't serving ubuntu mirror content from the opendev
mirror. Add this to the list of afs content symlinks we create.
Change-Id: I10b985afbaa737033cd5c1d4dd72eb8e77f8eb32
Fix the reported stat name for the mirror playbook.
Run the mirror job in gate.
Set follow=false so that we're telling Ansible to set the perms
on the link rather than the target (which is the default).
Change-Id: Id594cf3f7ab1dacae423cd2b7e158a701d086af6
Tumbleweed is only rarely used in the openStack CI, so mirroring it
fully is not worth the time/space overhead. a caching proxy
should be good enough. Add it to the directories to clean up
and remove the older entries because they will no longer be
matching.
Change-Id: I987da098cf4a7330cdec8da9ae3cfbff2f330bf8
This impelements mirrors to live in the opendev.org namespace. The
implementation is Ansible native for deployment on a Bionic node.
The hostname prefix remains the same (mirrorXX.region.provider.) but
the groups.yaml splits the opendev.org mirrors into a separate group.
The matches in the puppet group are also updated so to not run puppet
on the hosts.
The kerberos and openafs client parts do not need any updating and
works on the Bionic host.
The hosts are setup to provision certificates for themselves from
letsencrypt. Note we've added a new handler for mirror nodes to use
that restarts apache on certificate issue/renewal.
The new "mirror" role is a port of the existing puppet mirror.pp. It
installs apache, sets up some modules, makes some symlinks, sets up a
cleanup cron job and installs the apache vhost configuration.
The vhost configuration is also ported from the extant puppet. It is
simplified somewhat; but the biggest change is that we have extracted
the main port 80 configuration into a macro which is applied to both
port 80 and 443; i.e. the host will have SSL support. The other ports
are left alone for now, but can be updated in due course.
Thus we should be able to CNAME the existing mirrors to new nodes, and
any existing http access can continue. We can update our mirror setup
scripts to point to https resources as appropriate.
Change-Id: Iec576d631dd5b02f6b9fb445ee600be060f9cf1e
