In our onging quest to find a stable upstream source, switch this to
the facebook mirror which seems to have been working well enough for
centos.
While we're here, reduce the frequency a bit as there's no need to
sync every two hours; it's a bit unfriendly to the remote end.
Change-Id: I50f38b58b3f0c0557fb1e2e0667a17a4f1d1dec6
We build our own images from scratch using yum/dnf and don't rely on the
isolinux/ or images/ content. Clean this up from our mirrors to reduce
their size and time to sync.
Change-Id: Ifa47c8ae1f3a080e69e4c00b0c2ef38f4094e1a2
Our current mirror is out of sync, it has happened twice now. It's
been several years since we switched away from kernel.org mirrors
(Ie6050fbf20259df39eab11f8e326d58351d98ea5) so it might be worth
trying again (they are currently in sync). Also we are using the
".us" version here, to keep things more local to the update server.
Change-Id: I29cf3895e7fa8e401a1d686119e56b4fddb0cc47
We have removed Fedora 34 from our CI system. Fedora has a short
lifetime for each release, and in this particular case Fedora 34 did not
boot reliably in all our clouds. Now that the images have been removed
we can remove the release from our mirrors too.
Change-Id: I07dfca0ef88dc9531e4cb2c67ebbca5e0503594e
This reverts commit aa5623982fbecb4958ee90ab7ddc54d6d0d49747.
The MIT mirror seems to now be missing Fedora 34 indices, but the
one we were using before at UH looks just fine now.
Change-Id: I59649ea93cc1ce13715096dcd0b8f828ce6b6724
This converts the reprepro mirror script to use the common functions
for timestamps and vos release. This function ssh's to the AFS server
and runs vos release directly there, avoiding many issues with
kerberos timeouts. This has been working successfully for the rsync
mirrors. This will also send stats back so we can keep an eye on the
timing.
Change-Id: I1be29f2d9ecaad03b22c87819e5ae8d16c4f177e
The upstream mirror may have private contents in dirs like .~tmp~/ or
snapshot/. We exclude these to avoid syncing problems when we don't have
permissions to read them.
Change-Id: I8d366f0e95667bfbe65f259877b13bd0d93cd877
There is a single Fedora Atomic image used by Magnum at this point that
we mirror. Lets mirror just that one image and then we can manually rm
the others.
Change-Id: I669247beb64bae41afddd0edce02c0b58e45aa6c
We've removed the images from nodepool in the depends-on and now we can
stop mirroring the distro.
Depends-On: https://review.opendev.org/754471
Change-Id: Ifd4b1fbc92514a76ffa86b7cb42a81f97c245604
The pubmirror[12].math.uh.edu mirrors of Fedora 31 updates for
x86_64 have been sitting stale for several days with a corrupt
index, causing jobs which access our copy of this from our mirror
network to fail. Instead mirror Fedora releases/updates from
mirrors.mit.edu which seems to be updating just fine currently. We
can switch this back if/when the situation with the uh.edu mirrors
is resolved.
We're continuing to mirror EPEL and Fedora Atomic from
pubmirror[12].math.uh.edu for now, as we've had no reports of EPEL
problems on our mirrors (yet anyway), and it's hard to find any
other rsync mirrors of Atomic.
Change-Id: Iefd02602e2f2b39c4b72dc4d95ac62993ca65cdd
Fedora 33 is not released yet and the TripleO team would
like to perform some tests on that image.
Change-Id: I39f6bedadc12277739292cf31cc601bc3b6e30ec
We found that new data (since we removed -t) was no correctly being
skipped for re-download. We have found that this doesn't happen with
-t on later rsyncs, which have included fixes for -t to not touch the
timestamps if things are not updated. We have updated mirror-update
to Focal that has this rsync, so restore the flag.
Change-Id: I3fa16dbf6487a442549c540796807ef4916d4e6e
After investigating why every release seems to take hours, we noticed
that even in zero-delta updates where nothing has changed upstream,
we'd get a utimensat() call that updated the modified time nanoseconds
portion; e.g.
lstat("Everything/x86_64/Packages/a/arch-install-scripts-23-1.fc31.noarch.rpm", {st_mode=S_IFREG|0644, st_size=28004, ...}) = 0
utimensat(AT_FDCWD, "Everything/x86_64/Packages/a/arch-install-scripts-23-1.fc31.noarch.rpm",
[UTIME_NOW, {tv_sec=1585676005, tv_nsec=6041000} /* 2020-03-31T17:33:25.006041000+0000 */], AT_SYMLINK_NOFOLLOW) = 0
This does not apply; in fact openafs uses the ns field as some sort of
generation counter [1]. This update is enough to convince openafs the
file has changed and it needs to be resynced, meaning that basically
every rsync run results in a full release.
This unnecessary update been fixed with [2] but is only in rsync
3.1.3+; our bionic host is currently 3.1.2. Dropping "-t" from the
rsync commands avoids transferring modification times and should avoid
this problem.
While looking, "-D" turns on "--devices" and "--specials" to transfer
block devices and named sockets/fifos. Turn this off.
Also remove "-p" if it was present. We already did this for centos
with Ib5db052cdd23e39aecbeead15cf08d4bd7fcab38 and Fedora with
Id24196791f80cd99fe8a330fb2c7c6d893fc9995, where odd upstream
permissions such as setgid on directories can't be synced to afs.
Consistently remove it.
Also switch back the fedora updates to just "-v"; we had it at "-i"
for debugging.
[1] http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-06-15.log.html#t2020-06-15T02:58:08-2
[2] https://git.samba.org/?p=rsync.git;a=patch;h=0f8e9e2d8638e47d646a6baba694b303ac84e695
Change-Id: I78f3d4990b168c71185eb1c4900ceeaca4b6a16f
It's difficult to know if a release process is running too long when
we don't have a history of how long it should run for.
This is mostly the stats function from run_all.sh that has been
sending stats about runtimes there. Wrap it in a vos_release function
with some minor refactoring, and update the scripts.
As noted inline, there's already release timer stats going to
afs.release.<volume> for the periodic release of docs/tarballs etc.
Change-Id: I3d79d1a0997af8977050b7f6e7cf3b7578cc8491
Remove Fedora 29 from mirroring and purge the content.
The jobs were removed in preparation for
https://review.opendev.org/#/c/712032/ which removed the nodes from
nodepool. So, no job can use Fedora 29 anymore.
Fedora 28 is removed, no need to purge it again.
Change-Id: I59015a880f39b43b57a889b4254353f8c3a557ea
Patch set [1] does not provide updates directory for
Fedora 31 which is required to run CI jobs.
[1] https://review.opendev.org/#/c/712680/
Change-Id: I01b9b50660f3c7a3b4eb05cc8a8c0847f47dc7ee
Some jobs are using Fedora 29 release, which is deprecated.
Create new mirror with release 31 and wait for changing job
rules.
Change-Id: I418f4c815d337c2b1edd0df1aa065536d992cb62
We switched Fedora to do vos release via ssh with localauth in
I56ecdb2511597197deeeadf51f50da7e02f56954 and it has been working.
Switch the rest of the update scripts. There is an increasing amount
of common code, start a common functions.sh script where we can put
this.
Change-Id: I4ba6d64a84bb66e8686901b16010352de942f303
Use the new vos_release user on the remote host to release the volume
via localauth, to avoid any timeouts.
Change-Id: I56ecdb2511597197deeeadf51f50da7e02f56954
Depends-On: https://review.opendev.org/#/c/695554/
Add a purge phase to remove old releases; remove Fedora 28 and stop
mirroring. Update the atomic list while we're there.
Fedora 28 was removed with Ic0b4b065a217dcfaa8c230cda53114793e93b803
Change-Id: If713844ac90ea37e8c4db30108c45d7a59832776
As described inline, this should make our mirror pulses more robust
against timeouts.
This is probably ripe for turning into more of a library situation for
all the other "vos release" calls too. But one thing at a time ... I
think we test with this for a while to see if stability returns.
Change-Id: I041a290053e4e8ceba80785598a5945e5adcf6f1
We are debugging why the fedora mirror has started to take 8+ hours to
release. There is suspicion that rsync is updating the metadata of a
lot of files, causing the incremental update to be more of a full
release.
If you read the man page carefully:
At the single -v level of verbosity, this does not mention when a
file gets its attributes changed. If you ask for an itemized list of
changed attributes (either --itemize-changes or adding "%i" to the
--out-format setting), the output (on the client) increases to
mention all items that are changed in any way
So we need "-i" to see these updates
Change-Id: Ia9cff1e126404bc5a3346e758a15314834aef409
We're having trouble with this volume; this makes the logging more
helpful for humans trying to establish the order of things happening.
It turns down the tracing, puts timestamps on the logs messages, makes
the messages more descriptive, and makes sure to timestamp the output
of the vos commands so we can see how long each step took.
Change-Id: Ia832b7ef86350efcbbe7d3b957e820a8a39df3e1
We don't need things like efi configs or pxeboot configs or ISOs.
Exclude these items to further reduce the size of the fedora mirror.
Change-Id: I93003b2f48d79dae627026e2c7af748ea9a9e34d
The fedora mirror is our largest mirror (850GB about twice as big as the
next mirror). Much of this size is due to the fedora atomic images we
mirror.
On further investigation I notice that we are mirroring ppc images (for
which we do not have cpus to run them on), image for fedora 25 and 36
which are quite EOL'd, and our exclusion of the raw.xz and vagrant
images is failing.
Update the rsync excludes to ensure we don't mirror any of these images
we don't need.
Change-Id: I86856cb4e51b0e687aac45a1f014f87c5141318f
pubmirror1.math.uh.edu is currently offline and listed as an altonly.
pubmirror2 seems to work fine so switch to it.
Change-Id: I2562f8686146d17d4fad3997b9be22361fa05fca
This move was prompted by wishing to expose the mirror update logs for
the rsync updates so that debugging problems does not require a root
user (note: not actually done in this change; will be a follow-on).
Rather than start hacking at puppet, the rsync mirror scripts make a
nice delination point for starting an Ansible-first/Bionic update.
Most magic is included in the scripts, so there is not much more to do
than copy them. The host uses the existing kerberos and openafs roles
and copies the key material into place (to be added before merge).
Note the scripts are removed from the extant puppet so we don't have
two updates happening simultaneously. This will also require a manual
clean to remove the cron jobs as a once-off when merging.
The other part of mirror-update is the reprepro based scripts for the
various debuntu repositories. They are left as future work for now.
Testing is added to ensure dependencies and scripts are all in place.
Change-Id: I525ac18b55f0e11b0a541b51fa97ee5d6512bf70
