The rsync mirror we were relying on ended up incomplete on a recent
sync, causing all OpenSUSE 15 jobs to fail updating the package
lists. Switch to an alternative that seems to have all the same
things for which we used the previous one.
Change-Id: I661bdbfcbc766966793cd64d7f21201879d3dbaa
This trims out old Fedora and SLES content from our yum-puppetlabs repo.
I think that Fedora might not be used at all but we can clean that up in
a followon as we get a better grasp of how this is used.
Change-Id: I53a9b6cb529f0d3e3822864aca7725a222c78113
Add a dry-run flag and abstract the rsync command to make for easier
testing if modifying the copy flags.
Change-Id: Ie658b60257b94436b1eda0cddf6deb639a87d659
The upstream mirror seems to have some dotfiles that give permissions
errors when trying to rsync. Ignore all dotfiles.
Change-Id: Ic165b27ba190667f2b6b9e17cfa3ebe2ea6b9fb3
The current source for our sync is outdated [1], and does not include
the latest CentOS 7.9 content. This is breaking some jobs relying on
CentOS, since they may have more recent packages in the image than
in the repositories [2].
[1] - https://mirror-status.centos.org/
[2] - https://bugs.launchpad.net/tripleo/+bug/1904214
Change-Id: Ie404396b0f108121241bdc26dd8d22c85106b0c4
This converts the reprepro mirror script to use the common functions
for timestamps and vos release. This function ssh's to the AFS server
and runs vos release directly there, avoiding many issues with
kerberos timeouts. This has been working successfully for the rsync
mirrors. This will also send stats back so we can keep an eye on the
timing.
Change-Id: I1be29f2d9ecaad03b22c87819e5ae8d16c4f177e
This converts the reprepro configuration from our existing puppet to
Ansible.
This takes a more direct approach; the templating done by the puppet
version started simple but over the years grew several different
options to handle various use-cases. This means you not only had to
understand the rather obscure reprepro configuration, but then *also*
figure out how to translate that from our puppet template layers.
Here the configuration files are kept directly (they were copied from
the existing mirror-update.openstack.org) and deployed with some light
wrapper tasks in reprepro/tasks/utils which avoids most duplication.
Note the initial cron jobs are left disabled so we can run some manual
testing before letting it go automatically.
Change-Id: I96a9ff1efbf51c4164621028b7a3a1e2e1077d5c
The upstream mirror may have private contents in dirs like .~tmp~/ or
snapshot/. We exclude these to avoid syncing problems when we don't have
permissions to read them.
Change-Id: I8d366f0e95667bfbe65f259877b13bd0d93cd877
There is a single Fedora Atomic image used by Magnum at this point that
we mirror. Lets mirror just that one image and then we can manually rm
the others.
Change-Id: I669247beb64bae41afddd0edce02c0b58e45aa6c
We've converted our opensuse leap 15 image to 15.2. If things look good
we should be clear to stop mirroring 15.1.
Change-Id: Id31a3b57f48a5be671c76a76d5c48b4ef5000c3e
We've removed the images from nodepool in the depends-on and now we can
stop mirroring the distro.
Depends-On: https://review.opendev.org/754471
Change-Id: Ifd4b1fbc92514a76ffa86b7cb42a81f97c245604
The pubmirror[12].math.uh.edu mirrors of Fedora 31 updates for
x86_64 have been sitting stale for several days with a corrupt
index, causing jobs which access our copy of this from our mirror
network to fail. Instead mirror Fedora releases/updates from
mirrors.mit.edu which seems to be updating just fine currently. We
can switch this back if/when the situation with the uh.edu mirrors
is resolved.
We're continuing to mirror EPEL and Fedora Atomic from
pubmirror[12].math.uh.edu for now, as we've had no reports of EPEL
problems on our mirrors (yet anyway), and it's hard to find any
other rsync mirrors of Atomic.
Change-Id: Iefd02602e2f2b39c4b72dc4d95ac62993ca65cdd
Leap 15.2 has released. We should mirror it, then bump our opensuse-15
image to 15.2. Once that is done we can clean up the 15.1 mirror
contents.
Change-Id: Ib8905602b6a65639d59d2da1fe0c1b9a0837013b
Fedora 33 is not released yet and the TripleO team would
like to perform some tests on that image.
Change-Id: I39f6bedadc12277739292cf31cc601bc3b6e30ec
We found that new data (since we removed -t) was no correctly being
skipped for re-download. We have found that this doesn't happen with
-t on later rsyncs, which have included fixes for -t to not touch the
timestamps if things are not updated. We have updated mirror-update
to Focal that has this rsync, so restore the flag.
Change-Id: I3fa16dbf6487a442549c540796807ef4916d4e6e
After investigating why every release seems to take hours, we noticed
that even in zero-delta updates where nothing has changed upstream,
we'd get a utimensat() call that updated the modified time nanoseconds
portion; e.g.
lstat("Everything/x86_64/Packages/a/arch-install-scripts-23-1.fc31.noarch.rpm", {st_mode=S_IFREG|0644, st_size=28004, ...}) = 0
utimensat(AT_FDCWD, "Everything/x86_64/Packages/a/arch-install-scripts-23-1.fc31.noarch.rpm",
[UTIME_NOW, {tv_sec=1585676005, tv_nsec=6041000} /* 2020-03-31T17:33:25.006041000+0000 */], AT_SYMLINK_NOFOLLOW) = 0
This does not apply; in fact openafs uses the ns field as some sort of
generation counter [1]. This update is enough to convince openafs the
file has changed and it needs to be resynced, meaning that basically
every rsync run results in a full release.
This unnecessary update been fixed with [2] but is only in rsync
3.1.3+; our bionic host is currently 3.1.2. Dropping "-t" from the
rsync commands avoids transferring modification times and should avoid
this problem.
While looking, "-D" turns on "--devices" and "--specials" to transfer
block devices and named sockets/fifos. Turn this off.
Also remove "-p" if it was present. We already did this for centos
with Ib5db052cdd23e39aecbeead15cf08d4bd7fcab38 and Fedora with
Id24196791f80cd99fe8a330fb2c7c6d893fc9995, where odd upstream
permissions such as setgid on directories can't be synced to afs.
Consistently remove it.
Also switch back the fedora updates to just "-v"; we had it at "-i"
for debugging.
[1] http://eavesdrop.openstack.org/irclogs/%23opendev/%23opendev.2020-06-15.log.html#t2020-06-15T02:58:08-2
[2] https://git.samba.org/?p=rsync.git;a=patch;h=0f8e9e2d8638e47d646a6baba694b303ac84e695
Change-Id: I78f3d4990b168c71185eb1c4900ceeaca4b6a16f
It's difficult to know if a release process is running too long when
we don't have a history of how long it should run for.
This is mostly the stats function from run_all.sh that has been
sending stats about runtimes there. Wrap it in a vos_release function
with some minor refactoring, and update the scripts.
As noted inline, there's already release timer stats going to
afs.release.<volume> for the periodic release of docs/tarballs etc.
Change-Id: I3d79d1a0997af8977050b7f6e7cf3b7578cc8491
Currently used mirror is not up to date and is not on the list of official
CentOS mirrors.
Changing to a mirror also located somewhere in US,TX - which should be
close-ish to DFW servers.
Change-Id: Id8267bc7e6c8a96ccdfd98c16390d36842f26de3
Remove Fedora 29 from mirroring and purge the content.
The jobs were removed in preparation for
https://review.opendev.org/#/c/712032/ which removed the nodes from
nodepool. So, no job can use Fedora 29 anymore.
Fedora 28 is removed, no need to purge it again.
Change-Id: I59015a880f39b43b57a889b4254353f8c3a557ea
Patch set [1] does not provide updates directory for
Fedora 31 which is required to run CI jobs.
[1] https://review.opendev.org/#/c/712680/
Change-Id: I01b9b50660f3c7a3b4eb05cc8a8c0847f47dc7ee
Some jobs are using Fedora 29 release, which is deprecated.
Create new mirror with release 31 and wait for changing job
rules.
Change-Id: I418f4c815d337c2b1edd0df1aa065536d992cb62
This migrates the afsmon script from puppet deploying on
mirror-update.openstack.org to ansible deploying on
mirror-update.opendev.org.
There is nothing particularly special and this just a straight install
with some minor dependencies. Since we have log publishing running on
the opendev.org server, we publish the update logs alongside the
others.
Change-Id: Ifa3b4d59f8d0fc23a4492e50348bab30766d5779
This is a migration of the current periodic "vos release" script to
mirror-update.opendev.org.
The current script is deployed by puppet and run by a cron job on
afsdb01.dfw.openstack.org.
My initial motivation for this was wanting to better track our release
of these various volumes. With tarballs and releases moving to AFS
publishing, we are going to want to track the release process more
carefully.
Initially, I wanted to send timing statistics to graphite so we could
build a dashboard and track the release times of all volumes. Because
this requires an additional libraries and since we are deprecating
puppet, further development there is unappealing and it would better
live in ansible.
Since I6c96f89c6f113362e6085febca70d58176f678e7 we have the ability to
call "vos release" with "-localauth" permissions via ssh on
mirror-update; this avoids various timeout issues (see the changelog
comment there for more details). So we do not need to run this script
directly on the afsdb server.
We are alreadying publishing mirror update logs from mirror-update,
and it would be good to also publish these release logs so anyone can
see if there are problems.
All this points to mirror-update.opendev.org being a good future home
for this script.
The script has been refactored some to
- have a no-op mode
- send timing stats for each volume release
- call "vos release" via the ssh mecahnism we created
- use an advisory lock to avoid running over itself
It runs from a virtualenv and it's logs are published via the same
mechanism as the mirror logs (slightly misnamed now).
Note this script is currently a no-op to test the deployment, running
and log publishing. A follow-up will disable the old job and make
this active.
Change-Id: I62ae941e70c7d58e00bc663a50d52e79dfa5a684
This appears to be the same thing we saw in Fedora with
Id24196791f80cd99fe8a330fb2c7c6d893fc9995, somehow upstream
directories have started acquiring a setgid bit, which breaks AFS
mirroring
rsync: failed to set permissions on "/afs/.openstack.org/mirror/centos/8/AppStream/aarch64/os": Permission denied (13)
and when we look
chmod("AppStream/aarch64/os/Packages", 02755) = -1 EACCES (Permission denied)
Drop the "-p" so we don't try and replicate these permissions.
Change-Id: Ib5db052cdd23e39aecbeead15cf08d4bd7fcab38
We switched Fedora to do vos release via ssh with localauth in
I56ecdb2511597197deeeadf51f50da7e02f56954 and it has been working.
Switch the rest of the update scripts. There is an increasing amount
of common code, start a common functions.sh script where we can put
this.
Change-Id: I4ba6d64a84bb66e8686901b16010352de942f303
Use the new vos_release user on the remote host to release the volume
via localauth, to avoid any timeouts.
Change-Id: I56ecdb2511597197deeeadf51f50da7e02f56954
Depends-On: https://review.opendev.org/#/c/695554/
Add a purge phase to remove old releases; remove Fedora 28 and stop
mirroring. Update the atomic list while we're there.
Fedora 28 was removed with Ic0b4b065a217dcfaa8c230cda53114793e93b803
Change-Id: If713844ac90ea37e8c4db30108c45d7a59832776
The extant "logrotate_daily" varaible doesn't really do anything and
isn't used. Modify this to be able to set a range of rotation periods
or a size.
Update mirror rsync mirrors to rotate weekly, as often releases run
overnight and it's a pain to reconstruct.
Change-Id: I121dc5f4fe7f226b66d18b9ec39e7e3839be4d40
This adds mirroring of CentOS 8. It is somewhat simpler because the
architecture we're interested in are in the base repos, no need for
altarch.
The current mirror doesn't have a 8/ directory; possibly they require
their own mirroring filter updates? Use an up-to-date mirror for 8
(we can switch 7 too, but leaving alone for now).
Additionally, the altarch mirror we are using appears to have gone
offline for at least a few days. Switch to another one that is in
Texas, which should be close-ish to the DFW servers.
Change-Id: I33d95fa6b2df23fbfdb6745a3079761e228f677b
Looks like leaseweb, kernel.org and others are not properly
syncing the Suse mirrors as they are out of date for Leap15.0 and
missing files (deltainfo.xml.gz) which causes job breakage for
Leap15.0 based jobs.
Revert this to the original mirror from a year ago which is updated
and not broken
Change-Id: Id7184ee973bbabfec3f601fc9200ffac17322558
Kolla uses this to build hacluster images.
Direct usage is causing timeouts recently.
I changed the formatting to make it more readable
and slapped a comment note for maintainers.
Change-Id: I68d7155718c0ae0744198ca96aca1a207bab7ed6
Signed-off-by: Radosław Piliszek <radoslaw.piliszek@gmail.com>
As described inline, this should make our mirror pulses more robust
against timeouts.
This is probably ripe for turning into more of a library situation for
all the other "vos release" calls too. But one thing at a time ... I
think we test with this for a while to see if stability returns.
Change-Id: I041a290053e4e8ceba80785598a5945e5adcf6f1
