This adds mirroring of CentOS 8. It is somewhat simpler because the
architecture we're interested in are in the base repos, no need for
altarch.
The current mirror doesn't have a 8/ directory; possibly they require
their own mirroring filter updates? Use an up-to-date mirror for 8
(we can switch 7 too, but leaving alone for now).
Additionally, the altarch mirror we are using appears to have gone
offline for at least a few days. Switch to another one that is in
Texas, which should be close-ish to the DFW servers.
Change-Id: I33d95fa6b2df23fbfdb6745a3079761e228f677b
Looks like leaseweb, kernel.org and others are not properly
syncing the Suse mirrors as they are out of date for Leap15.0 and
missing files (deltainfo.xml.gz) which causes job breakage for
Leap15.0 based jobs.
Revert this to the original mirror from a year ago which is updated
and not broken
Change-Id: Id7184ee973bbabfec3f601fc9200ffac17322558
Kolla uses this to build hacluster images.
Direct usage is causing timeouts recently.
I changed the formatting to make it more readable
and slapped a comment note for maintainers.
Change-Id: I68d7155718c0ae0744198ca96aca1a207bab7ed6
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
As described inline, this should make our mirror pulses more robust
against timeouts.
This is probably ripe for turning into more of a library situation for
all the other "vos release" calls too. But one thing at a time ... I
think we test with this for a while to see if stability returns.
Change-Id: I041a290053e4e8ceba80785598a5945e5adcf6f1
We are debugging why the fedora mirror has started to take 8+ hours to
release. There is suspicion that rsync is updating the metadata of a
lot of files, causing the incremental update to be more of a full
release.
If you read the man page carefully:
At the single -v level of verbosity, this does not mention when a
file gets its attributes changed. If you ask for an itemized list of
changed attributes (either --itemize-changes or adding "%i" to the
--out-format setting), the output (on the client) increases to
mention all items that are changed in any way
So we need "-i" to see these updates
Change-Id: Ia9cff1e126404bc5a3346e758a15314834aef409
We're having trouble with this volume; this makes the logging more
helpful for humans trying to establish the order of things happening.
It turns down the tracing, puts timestamps on the logs messages, makes
the messages more descriptive, and makes sure to timestamp the output
of the vos commands so we can see how long each step took.
Change-Id: Ia832b7ef86350efcbbe7d3b957e820a8a39df3e1
We don't need things like efi configs or pxeboot configs or ISOs.
Exclude these items to further reduce the size of the fedora mirror.
Change-Id: I93003b2f48d79dae627026e2c7af748ea9a9e34d
The fedora mirror is our largest mirror (850GB about twice as big as the
next mirror). Much of this size is due to the fedora atomic images we
mirror.
On further investigation I notice that we are mirroring ppc images (for
which we do not have cpus to run them on), image for fedora 25 and 36
which are quite EOL'd, and our exclusion of the raw.xz and vagrant
images is failing.
Update the rsync excludes to ensure we don't mirror any of these images
we don't need.
Change-Id: I86856cb4e51b0e687aac45a1f014f87c5141318f
pubmirror1.math.uh.edu is currently offline and listed as an altonly.
pubmirror2 seems to work fine so switch to it.
Change-Id: I2562f8686146d17d4fad3997b9be22361fa05fca
This adds a periodic job to copy logs to a mirror volume, and export
it via the usual mirror http.
I have precreated the log volume; just as a R/W volume because this is
expected to be very low volume access.
Change-Id: I67870f6d439af2d2a63a5048ef52cecff3e75275
We have not put /usr/local/bin into the cron path, so it does not find
the update scripts. Since the scripts were working with this path on
the old host, restore it as a cron variable to maintain the status
quo.
Change-Id: Id9b7533720d3ccd9251055dec5b452cf5963dc85
Currently we start all jobs at the same time, which was not the
intent. Switch this to seed on the unique name of the job, which
should space jobs out randomly.
Change-Id: Ib41d8ca10aefe4a29bdd02935de8a588ab881958
Keytabs are slightly longer than what is being tested; upto 100 bytes
or so. This means the encoded data breaks over lines, which means you
need to be more careful about quoting.
Update the testing to a longer keytab (100 bytes of random data) and
fix up the quoting. Also enable no_logging to avoid putting key
material into the logs.
Change-Id: I73c391a2ebd2c962dc9a422f9d44265160210852
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).
Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.
Most magic is included in the scripts, so there is not much more to do
than copy them. The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).
Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously. This will also require a manual
clean to remove the cron jobs as a once-off when merging.
The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories. They are left as future work for now.
Testing is added to ensure dependencies and scripts are all in place.
Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
