There is some correlation that running the manage-projects playbook
gives our gitea fits. The bulk of the work done here is in trying to
update the descriptions of all projects. There isn't a good way to see
if the description is already set first so we just try and ignore
errors. This creates potentially thousands of operations all at once and
could be why things are sad.
We move these operations under the always update flag which is not set
on normal runs. If we really need to converge to a good updated state we
can manually run the playbook/role with always update set.
We also don't set a limit on the number of ThreadPoolExecutor workers
which will default to 5 * NumProcs. Could be that tuning this down would
make gitea happier.
One other thought is that we may not be using request sessions properly
for connection reuse. In particular requests notes that you need to set
stream to False or read request content to return a connection back to
the pool for reuse. We might look into this for further improvements.
Change-Id: I6e6fb1eb08303e9da7e38cf493d1871364340000
There appears to be a gitea bug that causes PATCH updates to projects to
fail when the cache is in a bad state for that project. We use PATCH
updates to a project to set the project descriptions. Since project
descriptions are not critical to gitea functionality (we weren't
updating them until last week) we can treat this as best effort and
ignore these failures.
We'll log these cases to aid in further debugging but continue on. The
next pass can retry.
Change-Id: I625bdc0856caaccb6b55931b0cdc6cf11a0bf3e1
When we decide we don't need to create a project we set the project
description. The reason for this is that humans like to see their
project descriptions update when they change them.
Rather than get, compare, and set the descrition we just set it under
the assumption this will be fewer requests and thus quicker. The impact
on the db likely plays into this too but our gitea dbs are mostly idle
so should be fine.
Change-Id: I04bdd747f8934d0b35bf76aec5d70be01b921285
Note this shouldn't be used until we can configure Gerrit to do similar
with jeepyb. Otherwise we'll end up with mismatched branches between our
canonical source (Gerrit) and our mirrors (Gitea).
Change-Id: I8d353cbc90c2d354e7cdebfc4e247f3f73d97d86
According to gitea swagger definitions all of these GET requests for
lists of items are paginated with a max limit of 50 items per request.
Update our ansible machinery to properly page these items to avoid
problems in the future.
Note we should try and confirm that this is how it works for production
gitea.
Change-Id: I5df13288b497fb4fb716b4223b3dd61c698a7739
We list gitea repos to determine if we need to create a repo. If the
repo isn't listed by gitea we create it. New gitea paginates these
listings so we were only getting 30 repos listed when we had far more.
This resulted in us trying to create repos which already exist which is
a gitea http 409 error.
Fix this by paging through the listings until we've seen all the
repos. This should give us a complete listing.
To test this we run our manage-projects playbook twice in the
system-config-run-gitea job. The first pass creates all the new
projects. Then the second pass should noop cleanly.
Change-Id: I73b77b9ddaa0106d4dc0a49c4d4b7751a39a16f9
Co-Authored-By: Jeremy Stanley <fungi@yuggoth.org>
We use project-config for gerrit, gitea and nodepool config. That's
cool, because can clone that from zuul too and make sure that each
prod run we're doing runs with the contents of the patch in question.
Introduce a flag file that can be touched in /home/zuulcd that will
block zuul from running prod playbooks. By default, if the file is
there, zuul will wait for an hour before giving up.
Rename zuulcd to zuul
To better align prod and test, name the zuul user zuul.
Change-Id: I83c38c9c430218059579f3763e02d6b9f40c7b89
We're making these requests to localhost over an ssh connection.
The password warning, on the other hand, is a real thing. Let's not
log the gitea password when we run this in prod.
Change-Id: I2157e4027dce5ab9ebceb3f78dbeff22a83d9fad
This runs repo creation across two orgs at the same time. It doesn't
help to parallelize more than 2 since openstack runs the entire time
in one thread (so the other thread handles all the other orgs).
Parallelizing by org avoids database contention for updating the user
table, since each org is a different user. However, there's a weird
locking thing going on with the first update to the settings table,
so this does some extra work to serialize actions until we perform
that first update, then switches to parallel.
This is the maximum we can parallelize repo creation at the moment,
and it also maximizes settings updates (the settings updates take less
time than repo creation, so no further optimization helps).
Change-Id: I7f83dcdb4531a547ae5281434d7cda825dd50059
This keeps repo creation serialized (because of a bug in gitea),
but it parallelizes updating the settings. This should reduce
our time by about half.
It also uses a requests session, though I'm not sure if that
really gets us anything.
It eliminates a couple of extraneous GET calls following 302
redirect responses from the POSTs on setting updates.
This will automatically paralellize to nproc * 5 threads.
Change-Id: I5549562d667c0939d0af1151d44b9190774196f9
When determining whether a project exists, we need to compare to
just the name, not the full data structure about the project.
Also, if the project exists, don't try to create it again; that
will return a 409 conflict error.
Change-Id: I0b8affac96b17fa73253082b1b87d4c00bf23463
Add the full remote_puppet_git playbook that we actually use in
production so that we can test the whole kit and caboodle. For
now don't add a review.o.o server to the mix, because we aren't
testing anything about it.
Change-Id: If1112a363e96148c06f8edf1e3adeaa45fc7271c
Sadly, as readable as the use of the uri module to do the interactions
with gitea is, more reent ansible changed how subprocesses are forked
and this makes iterating over all the projects in projects.yaml take
an incredibly long amount of time.
Instead of doing it in yaml, make a python module that takes the list
one time and does looping and requests calls. This should make it be
possible to run the actual gitea creation playbook in integration tests.
Change-Id: Ifff3291c1092e6df09ae339c9e7dddb5ee692685
I ran our global gitea project sync playbook across all eight gitea
hosts and one failed with a 404 against a specific project. Rerunning
the playbook against that one gitea server worked fine.
Until we sort out why this might happen lets retry our HTTP POSTs up to
3 times until they succeed.
Some numbers: We have ~2k repos and 8 servers and make two http requests
per repo for a total of 32k requests. If one fails out of that the
success rate is very high so retrying a few times should be fine.
Change-Id: I937a4f852f6713a419c03a17c3b4984a97eae0d8
Some projects use storyboard and some use launchpad as their issue
tracker. Handle this dynamically based on the projects.yaml content when
we create projects and set the appropriate urls.
Change-Id: Ie1d79bf1a662c00078689f2c752231e610610eda
Story: 2004627
Task: 29849
The web page from which this call is taken has a readme selector
and a default value of "Default" that we're not sending in this
request. Send it to avoid gitea not being able to find the empty
readme.
Also, add gitea-git-repos to the files section of system-config-run-gitea
so that we actually test it.
Change-Id: Ieec94aadb63fa097f10a3f325dd105b30e610dd9
This reverts commit b076aefa0864dc12269a4454a921f4830b2a2d9a.
This may still not work. The last time it got 404s when
running as part of project creation. So possibly it was
failing because there was no git content yet and thus
master was an invalid value. If that's the case, we may
need to add auto_init to true just to get a default repo
and metadata and count on gerrit force-pushing during first
replication.
Use auto_init to create a repo with a master branch
It's not possible to set the default_branch to master if there is
no repo, and without that first replication sets the branch
incorrectly.
Set auto_init to true to get an empty repo which will be replaced
by the initial gerrit force-push anyway.
Change-Id: I66de43a4d00583c06cf92bb7083ce28d4b8d4df9
Story: 2004627
Task: 29847
Ansible task execution can be a bit slow - so the noop case where
we don't end up doing anything can still be costly. Instead, put
the when on the loop call, which will apply to each iteration of
the loop, not running the loop itself. This way we should only
include_tasks if we need to.
In order for the utility playbook to keep working, we also run all
of the iterations of the loop if gitea_always_update is true. This
will make a sync run take a long time but be comprehensive.
Change-Id: Ib60c736d46d8253e603de097eb80bc84b3366310
We normally only do project settings when we create the project.
Add a playbook we can use to do a manual sync from project-config.
Change-Id: I5260f2de697420a01d796acc3128be00705a53ee
This is 404ing. It's not strictly needed for the moment, let's
come back to it.
This reverts commit 266b2dd3fc4bcbe2416f77f34540705a6ba6cdde.
Change-Id: Id9ab3c233753025d7a01eb4664ecee6d102bbf19
If, for some reason, a repo gets created via gerrit replication
instead of via the api, the default branch setting can be incorrectly
inferred. While we're setting things, just set master to be the default
branch everywhere.
Change-Id: I388afe670221bd8ec28fbba71041ab460e66411b
This uses the form post approach also used in repo renaming. There
is no official REST API for this yet, so the form post values were
taken from what the web ui does.
We should submit a patch upstream to get an actual api call for this.
Run it idempotently, submitting the settings for every repo every time.
Change-Id: I9265837039df962e85f11d16419e043fb9a56ff8
We are currently attempting to create repos which already exist. This
fails.
The reason for this is we set the gitea_org_repos fact for each org
overriding the last org. This means only the last org processed has any
projects in this list. We then check against this list when creating
projects so that we only create projects if they aren't in the list.
Meaning any project for repos not in the last org attempts to get
recreated.
We can address this by keeping a global list of repos regardless of org
then checking against that.
An alternative solution would be to process projects for each org
separately. Or to have gitea give us the global list so we don't have to
build it ourselves.
Change-Id: Id9a480634918dad2160a4e040a41ce6226ae67d8
Switch to a heredoc for the gitea sql_statement so that we don't have to
figure out multiple levels of "" quoting.
Change-Id: I734640936a9d15d03026fc3b05b5fbc221957b94
Go ahead and actually clone project-config on bridge, because
we're going to read the projects.yaml file with a lookup. Because
it's a local action, not a remote action.
Change-Id: I77454bcb10b797ce5b48018caef7fecb31947b97
We need to make sure repos are created in gitea before gerrit so
that we can safely create to gerrit and have it replicate.
Change-Id: If3efc7ed0d7995f1ef6f52cd2eefb9260193e020
Story: 2004627
Task: 29703
