After the big OpenDev rename, these repos got renamed again. Update the
redirects for git.airshipit.org and git.starlingx.io to point to the
current location.
Update test_static.py for this, change the test repo since
airship-in-a-bottle was first renamed to in-a-bottle and later to
airship-in-a-bottle.
Change-Id: I71b786cd528aac9ae68464618db02e22cd4c0b5b
zuul and nodepool now life in opendev, avoid double redirects and
redirect directly to final location.
Change-Id: Ia55d76b24f07ec64cb55055955c4549f3706a95b
This adds the Open Edge (formerly Fortnebula) CI mirror.
Change-Id: I1ccf2a602f8a41e00bc64a9516a326cc07d9b254
Depends-On: https://review.opendev.org/711787
Sister change for Ia5caff34d3fafaffc459e7572a4eef6bd94422ea and
removing earlier references to the mirror server in preparation for
building and adding the new one.
Change-Id: I7d506be85326835d5e77a0c9c461f2d457b1dfd3
This reverts commit c25e91f49632d8e187f35807f250567446bd5102.
This script parses the Apache logs and writes out a local count of the
404 data to files.openstack.org, and then exports it via
files.openstack.org.
As part of the spec [1] we're trying to remove publishing from local
volumes, in general.
Since this is not widely used, there is only one link to it, it's not
discoverable from the landing page of files.openstack.org (which just
shows the afs directory listing), it has a very long latency making it
not that useful for debugging and grepping the logs there have been no
accesses in the past 2 weeks (as far back as logs go) I propose we
remove it.
If we want to retain this, we should publish the output alongside the
docs AFS volume. That could certainly be done by distributing the
docs keytab to the host and having it write out in a similar cron job.
Another option could be to setup a keypair for remote login and keep
that as a secret in Zuul, and do the same from a periodic job
(complicated by apache logs being root only, so needs some sudo magic
or similar). Or, we could figure out an altogether better, privacy
respecting client analytics solution.
[1] https://docs.opendev.org/opendev/infra-specs/latest/specs/retire-static.html
Depends-On: https://review.opendev.org/709036
Change-Id: Iccf24a72cf82592bae8c699f9f857aa54fc74f10
A bunch of the sites we've started managing LE certs for are not
getting their expirations checked, so fix that. In particular, sites
recently moved off the multi-domain SAN cert for the old
static.openstack.org server (omitted the logs site as it's
deprecated), and many of the rebuilt CI mirrors (with the exception
of mirror01.gra1.ovh.opendev.org which is presently in a SHUTOFF
state for unknown reasons). Also add graphite which was previously
missed, and review-dev because we can now that it's no longer
sporting snakeoil.
When this merges, we're also going to start getting alerts for an
expired cert on mirror.gra1.ovh.opendev.org, unless someone gets a
chance to look into it first.
Change-Id: I98a98e0d2ff081c51c33d980274f3ee8c0266802
The insecure-ci-registry.opendev.org service uses an X.509 cert on
5000/tcp, so we should track this to catch when it's going to
expire.
Change-Id: I5d18599e5b5b258ce158f964cb1ff95df6dc6d92
The ssldomains file we use for our cert check is getting longish,
and sorting it will make entries easier to find.
Change-Id: Iad182ecee45274d6c8f336a97d20a3130e4b8abe
Now that opendev.org backends requests certs unique to each backend we
should check these backends directly and not only through the frontend.
This was if a specific backend doesn't end up updating with LE properly
we will catch it.
Change-Id: Icabb1bcb725937da45ae9aaef2c9da412a30a319
This runs gerrit in a container on review-dev01 using podman.
Remove an unused web_server.py file that we found from copying it
from puppet to ansible.
Change-Id: I399d3cf8471bc8063022b0db0ff81718b2ee2941
Ceph Nautilus is released and the official mirror
is available. This adds the Ceph Nautilus mirror
so we can sync it for Stretch and Bionic.
Based on the same change that was done when Mimic
was released [1]
[1] https://review.opendev.org/#/c/571989/
Change-Id: I9424d1f4df58acde8ea70dc16283d4de89189bae
Sharing an updates file between the Debian and Ubuntu reprepro runs
causes some warnings, and is generally just unclean. They use
different release naming and repositories, so should just have
separate updates files to track them (they're already separate on
the server, they were just being copied from the same source file in
the module).
While here, remove the label and suite parameters from the Debian
reprepro distribution templates, as they're unnecessary and
potentially confusing (job nodes should never be relying on the
suite names as they change at the next release).
Also allow signatures from subkeys of the listed keys to be
sufficient to verify the debian-security mirror's release files,
like we do for the debian mirror.
Change-Id: Id0ff476864f936bbd7c4637f3dc9e2c219c6e465
There have been several Web sites added to files.o.o which missed
getting x509 SSL/TLS certificate checking added through our
certcheck cron job. Add those now so we know in advance whether
they're at risk of expiration.
Change-Id: I3eda77f165348e510d43344b172cf5b56ce2b003
Docker hosts report back mounts in container directories via snmp
storage queries
# php -q /usr/share/cacti/cli/add_graphs.php --host-id=585 --snmp-field=hrStorageDescr --list-snmp-values
Known values for hrStorageDescr for host 585: (name)
...
/var/lib/docker/containers/05ed2dc...
/var/lib/docker/containers/7cebed4...
/var/lib/docker/containers/f452861...
Because these can keep changing, hosts just end up getting more and
more invalid graphs in their results (see gitea0X hosts in cacti at
the moment).
Filter out docker directories from the query
Change-Id: Ia1db628975e7a67ad531438ef85735abae1ce652
Because of a limitation in GnuPG we need to have the Jessie archive
signing key in the list of VerifyRelease key IDs for the Debian
reprepro mirror. Also some suites (currently buster-backports) are
signed by a subkey of an archive signing key, so add the "+" suffix
to all these key IDs indicating subkey signatures are also allowed.
As always, Debian signing keys are published and available here:
https://ftp-master.debian.org/keys.html
Change-Id: Iedce38318718a18ace7b2c638755a7d7d4dcd69b
The buster-backports suite on Debian mirrors is not signed by the
old jessie signing key we have set to verify in reprepro, but also
we're not mirroring Debian 8/jessie any longer anyway. Replace that
list with the 9/stretch and 10/buster signing keys and switch to
longer key IDs which match the names used for them in the Puppet
manifest. Also add Puppetry and keyfile for the buster keys so that
they will be installed accordingly. The official list of keys can be
found here: https://ftp-master.debian.org/keys.html
Change-Id: Ia193f040b2b707329948955eb091a186eabf8096
Apply the exclusion for trusted CI comments to the hide function's
conditional case as well as the toggle function's.
Change-Id: Ia4e5ec22a097a8b8cb564c237fd0aa48ab6f8724
When filtering CI system comments, don't hide those from Zuul, our
gating CI system. It is important to see these comments as not all
results may match the patterns used to expose them as rows in the CI
table. Rename the "Toggle CI" button to "Toggle Extra CI" so that
the name remains accurate without being too verbose.
Change-Id: Id0cd8429ee5ce914aebbbc4a24bef9ebf675e21c
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).
Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.
Most magic is included in the scripts, so there is not much more to do
than copy them. The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).
Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously. This will also require a manual
clean to remove the cron jobs as a once-off when merging.
The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories. They are left as future work for now.
Testing is added to ensure dependencies and scripts are all in place.
Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
This reverts commit b3ce1c52dc7ca455ffd94ea07d8a4fb1b6905fa8.
It removed the AFS mirror at the same time it added the proxy,
but jobs don't know to look for the proxy since it's on a
totally different TCP port.
Change-Id: I87cc03eb3322bd7b093dd6fe798aadb48f319805
This removes groups.openstack.org as this service was shut down. Add new
opendev services behind ssl.
Change-Id: I14c667c8fbde07c3a52778bc2c5e93abf8f053a4
As a follow-on to I0e110ef51c8ed301fd8280ae7fc039e3b01db92c; this
dropped the /centos/ from the base mirror, add it back.
Also switch the mirror to the only one on the altarch-mirrors page
that is in US/TX, which from the name is in Dallas, which must be
pretty close to rax.dfw where the update server lives.
Change-Id: If4d71865f4328e73a26c7b38300767ed6b790579
CentOS keeps non-x86 architectures in /altarch/ directory (contrary to
/centos/ one for x86-64). We have aarch64 (arm64) machines in infra and
they fail due to lack of CentOS altarch mirror.
List of wanted alternative architectures is controlled by ALTARCHS
variable (aarch64 and ppc64le enabled). As CentOS has several other
architectures too they are listed in ALTARCHS_IGNORED so we do not fetch
them.
Current CentOS mirror lands in same /mirror/centos/7/ directory. Altarch
mirrors goes to /mirror/centos/altarch/7/ one.
Change-Id: I0e110ef51c8ed301fd8280ae7fc039e3b01db92c
The yum-puppetlabs mirror exceeded its 100GB quota as of April 26.
Rather than increase the quota, start excluding packages for old
platforms we don't provide like RHEL5-6 and Fedora F20-27. We could
probably get even more aggressive with it, but this get the
utilization back under 50% which is plenty of headroom for now.
Change-Id: I9665b3a2a89f991f9433fe7f45bc1bb0e0c7632b
It seems the openSUSE build process can leave artifacts behind,
in the form of .~tmp~ files in the mirror. I assume these are
wrongfully present.
This is a problem, as those ~tmp~ files prevent syncing the
repositories.
While it's most likely that openSUSE files will be cleaned in the
source repos, should this problem arise in the future, it's also
more robust to skip the syncing of those files.
This has the extra benefit of temporarily unblock mirroring of
openSUSE Leap 15.1 in infra, as of today.
Change-Id: I0124b992483cfda9f97960b43bddf94efa008030
Build a container image with the haproxy-statsd script, and run that
along with the haproxy container.
Change-Id: I18be70d339df613bf9a72e115e80a6da876111e0
It doesn't seem like this is used anymore. Let's remove it before
we update the rest of this, so that we don't have to, you know,
update abandoned things.
Change-Id: I1c3708021046a428da82eaa843961091915ba4af
openSUSE Leap 15.1 was released May 23rd, 2019 and we want to switch
the nodepool jobs against this asap in order to be able to remove
openSUSE Leap 42.3 (End of Life in June 2019) and eventually
openSUSE Leap 15.0 as well once all users have been converted over.
Change-Id: Ia2f8b9f4073a247875c97eafd80204e291affb8e
Tumbleweed is only rarely used in the openStack CI, so mirroring it
fully is not worth the time/space overhead. a caching proxy
should be good enough. Add it to the directories to clean up
and remove the older entries because they will no longer be
matching.
Change-Id: I987da098cf4a7330cdec8da9ae3cfbff2f330bf8
