By default gitea caches everything in memory using a Go hashmap. There
is suspicion that the now many persistent AI web crawlers cause this
hashmap to grow in ways that eventually cause the Go GC system to pause
the world in noticeable ways when loading pages. Restarting the gitea
services seems to temporarily reset things (as it should with an in
memory map) until we cross some threshold and things become slow again.
The good news is that gitea supports several backends (called adapters)
for the cache. We elect to use memcached because it is relatively
simple and has a FOSS license (unlike redis). The other alternative we
could consider is twoqueue which also caches within memory in the Go
runtime but allows for setting a size limit. I've gone with memcached
because it doesn't rely on Golang GC, but twoqueue is likely simpler if
we want to start there.
Note we also bump the job timeout to 5400 seconds (90 minutes) from 4800
seconds (80 minutes) because a run on ovh-gra1 timed out while running
testinfra test cases (the very end of the job). It is possible that
using memcache is slightly slower than using in process memory caching,
but the goal here isn't to make things faster it is to make things more
consistent over time. As long as memcached performance is within the
same ballpark and doesn't degrade over time this is acceptable.
Change-Id: Ie9ca246a8321fe84d9a1582e35cd4c5459b48bee
In preparation for a switch from docker-compose to docker compose (with
a shim so that docker-compose is docker compose) we switch away from
direct docker exec to using docker-compose exec because the two
different compose implementations use different container names. This
way we shouldn't need to lookup container names at runtime or have
special casing depending on which system we want to use. Instead the
compose runtime will find the correct container to exec in based on the
human readable name in our config file.
Change-Id: I5ffb0007997dfdf2da67df3ada5db4534dc383a7
By default our mariadb database for gitea nodes limits itself to a
maximum of 100 connections. We've seen errors like this:
...eb/routing/logger.go:102:func1() [I] router: completed POST /openstack/requirements/git-upload-pack for 127.0.0.1:50562, 500 Internal Server Error in 2.6ms @ context/user.go:17(web.gitHTTPRouters.UserAssignmentWeb)
...ules/context/repo.go:467:RepoAssignment() [E] GetUserByName: Error 1040: Too many connections
And after reading gitea's source code this appears to be related to user
lookups to determine if the user making a request against a repo owns
the repo. To do this gitea does a db request to lookup the user from the
request and when this hits the connection limit it bubbles up the mysql
error 1040: Too may connections error.
This problem seems infrequent so we double the limit to 200 which is
both much larger but still a reasonable number.
We also modify the test that checks for gitea server errors without an
http 500 return code to avoid it matching this change improperly. This
was happening because the commit message ends up in the rendered pages
for system-config in the test gitea.
Change-Id: If8c72ab277e88ae09a44a64a1571f94e43df23f8
When manually testing the gitea 1.21.3 upgrade tonyb discovered "500"
errors on the code search page. The http side reported all 2XX response
codes but the page rendered a giant 500. Turns out the problem was in
template rendering which produces the giant 500 in the page but doesn't
necessarily send a 500 http error code.
Test for this automatically on a number of pages by inspecting the page
content for 500 status page content.
Note this is somewhat fragile because they could change the template
content at any time, but it seems better to do this than do nothing at
all.
Change-Id: I1964be7be87ef5a6e75c6639a4d75d9090a14db8
Upgrade Gitea to 1.21.3. The changelogs for this release can be found
here:
https://github.com/go-gitea/gitea/blob/v1.21.3/CHANGELOG.md
I have attempted to collect the interesting bits in this commit message
as well as information on why we do or don't make changes to address
these items.
1.21.0
* BREAKING
* Restrict certificate type for builtin SSH server (https://github.com/go-gitea/gitea/pull/26789)
* We don't use the builtin SSH server and don't use certificates
for auth. Nothing to do here.
* Refactor to use urfave/cli/v2 (https://github.com/go-gitea/gitea/pull/25959)
* The major change here updated `gitea` to stop accepting
`gitea web`'s command options. Our dockerfile is set up to use
`CMD ["/usr/local/bin/gitea", "web"]` so we are not affected.
* Move public asset files to the proper directory (https://github.com/go-gitea/gitea/pull/25907)
* We update the testinfra test for robots.txt to more robustly
check file contents. Previously it checked a very generic
prefix which may indicate a generic file being served.
* We move custom/public/img into custom/public/assets/img.
Screenshots should be used to confirm this works as expected.
* Remove commit status running and warning to align GitHub (https://github.com/go-gitea/gitea/pull/25839)
(partially reverted: Restore warning commit status (https://github.com/go-gitea/gitea/pull/27504) (https://github.com/go-gitea/gitea/pull/27529))
* We don't rely on commit statuses as this is a read only replica
of Gerrit.
* Remove "CHARSET" config option for MySQL, always use "utf8mb4" (https://github.com/go-gitea/gitea/pull/25413)
* We don't set [database].CHARSET. Doesn't affect us.
* Set SSH_AUTHORIZED_KEYS_BACKUP to false (https://github.com/go-gitea/gitea/pull/25412)
* We don't set this value explicitly so the default will flip from
true to false for us. I don't think this is an issue because we
keep track of our pubkeys in git.
* SECURITY
* Dont leak private users via extensions (https://github.com/go-gitea/gitea/pull/28023) (https://github.com/go-gitea/gitea/pull/28029)
* We don't use private users.
* Expanded minimum RSA Keylength to 3072 (https://github.com/go-gitea/gitea/pull/26604)
* We have rotated keys used to replicate from gerrit to gitea to
work around this. Now are keys are long enough to make gitea
happy.
* BUILD
* Dockerfile small refactor (https://github.com/go-gitea/gitea/pull/27757) (https://github.com/go-gitea/gitea/pull/27826)
* I've updated our Dockerfile to mimic these changes. Comment
whitespace as well as how things are copied and chmoded in the
build image have been updated.
* TODO the file copies aren't working for us. I think due to how we
ultimately clone the git repo. We use RUN but upstream is using
COPY against the local build dir. I've aligned as best as I can,
but we should see if we can do a similar COPY on our end.
* Fix build errors on BSD (in BSDMakefile) (#27594) (#27608)
* We don't run on BSD.
* Fully replace drone with actions (#27556) (#27575)
* This is how upstream builds their images. Doesn't affect our
builds.
* Enable markdownlint no-duplicate-header (#27500) (#27506)
* Build time linters are somethign we don't care too much about on
our end.
* Enable production source maps for index.js, fix CSS sourcemaps (https://github.com/go-gitea/gitea/pull/27291) (https://github.com/go-gitea/gitea/pull/27295)
* This emits a source map for index.js which can be used for in
browser debugging. Don't think this is anything we need to take
action on.
* Update snap package (#27021)
* We don't use a snap package.
* Bump go to 1.21 (https://github.com/go-gitea/gitea/pull/26608)
* Our go version is updated in the Dockerfile.
* Bump xgo to go-1.21.x and node to 20 in release-version (https://github.com/go-gitea/gitea/pull/26589)
* Our node version is updated in the Dockerfile.
* Add template linting via djlint (#25212)
* Build time linters are somethign we don't care too much about on
our end.
1.21.1
* SECURITY
* Fix comment permissions (https://github.com/go-gitea/gitea/pull/28213) (https://github.com/go-gitea/gitea/pull/28216)
* This affects disclosure of private repo content. We don't have
private repos so shouldn't be affected.
1.21.2
* SECURITY
* Rebuild with recently released golang version
* We'll automatically rebuild with newer golang too.
* Fix missing check (https://github.com/go-gitea/gitea/pull/28406) (https://github.com/go-gitea/gitea/pull/28411)
* There is minimal info here but it appears to be related to
issues. We don't use issues so shouldn't affect us.
* Do some missing checks (https://github.com/go-gitea/gitea/pull/28423) (https://github.com/go-gitea/gitea/pull/28432)
* There is minimal info here but it appears to be related to
checks around private repos. We don't use private repos so this
shouldn't affect us.
1.21.3
* SECURITY
* Update golang.org/x/crypto (https://github.com/go-gitea/gitea/pull/28519)
* This addresses recent concerns found in ssh for gitea's built in
ssh implementation. We use openssh as provided by debian so will
rely on our distro to provide fixes.
Finally 1.21.x broke rendering of code search templates. The issue is
here: https://github.com/go-gitea/gitea/issues/28607. To address this
I've vendored the two fixed template files
(https://github.com/go-gitea/gitea/pull/28576/files)into our custom
template dirs. Once upstream makes a release with these fixes we can
drop the custom files entirely as we don't override anything special in
them.
Change-Id: Id714826a9bc7682403afcf90f2761db8c84eacbf
Please carefully review the changelog:
https://github.com/go-gitea/gitea/blob/v1.17.1/CHANGELOG.md
and ensure that we've properly addressed the items listed there.
I have listed the breaking changes list here and any actions we've taken
or justification for why they don't affect us:
* Require go1.18 for Gitea 1.17 (#19918)
We were already using go 1.18.
* Make AppDataPath absolute against the AppWorkPath if it is not (#19815)
Path is already absolute:
playbooks/roles/gitea/templates/app.ini.j2:APP_DATA_PATH = /data/gitea
* Nuke the incorrect permission report on /api/v1/notifications (#19761)
This has to do with how that api endpoint returns permissions. We
don't use this anywhere as far as I can tell.
* Refactor git module, make Gitea use internal git config (#19732)
In the gitea container /data/git/.gitconfig is present but we don't
appear to manage this in system-config. I think that means this
change is a noop for us as gitea will move its managed .gitconfig
from /data/git/.gitconfig to /data/git/repositories/.gitconfig.
I expect the contents to be the same since gitea must be managing
the file old content today.
* Remove RequireHighlightJS field, update plantuml example. (#19615)
This was a flag that toggled syntax highlighting on and off as best
as I can tell. The default is to just have it turned on and we don't
check the flag in any of our templates.
* Increase minimal required git version to 2.0 (#19577)
Debian Bullseye ships with 2.30.2-1.
* Add a directory prefix gitea-src-VERSION to release-tar-file (#19396)
They were tarbombing people and their tarballs extracted into the
current dir. They now no longer do that. We build from git so this
doesn't affect us.
* Use "main" as default branch name (#19354)
We explicitly set the default branch name to master for both gitea and
gerrit. This should be a noop for us. Testing has been added to check
this.
https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gitea-git-repos/library/gitea_create_repos.py#L129-L132https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/manage_projects.py#L488
* Make cron task no notice on success (#19221)
I'm not aware of us relying on any cron tasks or any cron task
notifications.
* Add pam account authorization check (#19040)
We don't integrate with pam so the change in behavior to check
authorization does not affect us.
* Show messages for users if the ROOT_URL is wrong, show JavaScript errors (#18971)
This message shows up in CI because ROOT_URL is https://opendev.org
but we access gitea in testing via localhost. I don't think this
is worth fixing. Its a good reminder that the instance is a test
instance.
* Refactor mirror code & fix StartToMirror (#18904)
We don't mirror repos with gitea. Should be a noop for us.
* Remove deprecated SSH ciphers from default (#18697)
hmac-sha1-96, diffie-hellman-group1-sha1, and arcfour{128,256} are
removed. The only ssh user is gerrit's replication. MINA should
be able to support more modern ciphers and be fine.
* Add the possibility to allow the user to have a favicon which differs from the main logo (#18542)
Previously, logo.svg was used as the favicon.svg and gitea only fell
back to favicon.png if the browser couldn't so the .svg. But now they
want to support users having different logo.svg and favicon.svg. This
necessitates explicitly adding a favicon.svg. Something we already do.
Details at https://github.com/go-gitea/gitea/pull/18542
* Update reserved usernames list (#18438)
This shouldn't be a problem for us as we don't have regular users and
gerrit is not a reserved name.
* Support custom ACME provider (#18340)
We run ACME with LE out of band. This doesn't affect us.
* Change initial TrustModel to committer (#18335)
This changes the signed commits trust model from collaborator
to committer. THis won't affect us as we aren't maintaining trusted
keys. But basically this now shows if the signed commit by the
committer matches the committer's key.
* Update HTTP status codes (#18063)
This changed redirect HTTP codes from 302 to 307. Shouldn't
affect us.
* Upgrade Alpine from 3.13 to 3.15 (#18050)
We build on Debian and not alpine. The alpine nodejs version did
change from 14 to 16 in this change and we've updated to match.
* Restrict email address validation (#17688)
If we had real users this may pose a problem as they are limiting
the set of emails gitea would accept to a smaller set than they
accepted before. Also fewer than actually allowed by email. But
we don't have real users so this should be fine.
* Refactor Router Logger (#17308)
This streamlines and improves the log format of some of the gitea
logs. We aren't automatically processing these logs today so this
shouldn't have a major impact on us.
Additionally this release adds a new git.HOME_PATH setting to set the
location for writing out git configs and potential gnupg configs. We
should be fine to let gitea write this content out to the default path,
but there is potential for this to impact our ssh daemon.
Changes made include:
* Minimal updates to web templates to match 1.17
* Updating nodejs to v16 as v14 failed to build gitea
* Disabling the new enabled by default "packages" feature
* New test to check repos have a master branch by default instead of
Gitea's new default of main.
Change-Id: I88105eccd118e3daca72f0b86a6b351c35e37413
This moves the gitea partial clone test from our setup playbook into
testinfra/test_gitea.py. We should avoid asserting too much state and
behavior in the ansible as it makes the split between testinfra and
ansible more confusing. To address this we move this behavior check into
testinfra where it belongs.
Change-Id: I6a649bc380f850425c51e9b4632c798a23ab0e0e
Over a few upgrades, we've managed to break some of the default avatar
logos you see when browsing code on opendev.org.
After investigating ways to fix this up, we established that there
isn't an exposed API for setting these, but we can do a simple query
to point to logo files on disk. This implements that.
One caveat is that the logos should be PNG files; particiularly we
note that SVG files don't work reliably because they don't get served
with the image/svg+xml mime-type.
Change-Id: Ie6799de2fb27e09f936c488258dc1bd1c638c370
Instead of having testinfra check that gitead serves robots.txt,
let's have it check that apache serves or proxies it (since that's
what the load balancer will see).
Change-Id: I809fe9c5d5b43e73a216b61d72eea95546b9619c
This tests that we can rename both the project and the org the project
lives in. Should just add a bit more robustness to our testing.
Change-Id: I0914e864c787b1dba175e0fabf6ab2648a554d16
It would be useful to test our rename playbook against gitea and gerrit
when we make changes to these related playbooks, roles, and docker
images. To do this we need to converge our test and production setups
for gerrit a bit more. We create an openstack-project-creator account in
the test gerrit to match prod and we have rename_repos.yaml talk to
localhost for gerrit ssh commands.
With that done we can run the rename_repos.yaml playbook from
test-gitea.yaml and test-gerrit.yaml to help ensure the playbook
functions as expected against these services.
Co-Authored-By: Ian Wienand <iwienand@redhat.com>
Change-Id: I49ffaf86828e87705da303f40ad4a86be030c709
This started with me wondering why gerritbot was putting all its
output into /var/log/syslog -- it turns out Xenial docker is
configured to use journalctl (which forwards to syslog) and Bionic
onwards uses json-file.
Both are sub-optimial; but particularly the json-file because we lose
the logs when the container dies. This proposes moving to a more
standard model of having the containers log to syslog and redirecting
that to files on disk.
Install a rsyslog configuration to capture "docker-*" program names
and put them into logfiles in /var/log/containers. Also install
rotation for these files.
In an initial group of docker-compose files, setup logging to syslog
which should then be captured into these files. Add some basic
testing.
If this works OK, I think we can standardise our docker-compose files
like this to caputure the logs the same everywhere.
Change-Id: I940a5b05057e832e2efad79d9a2ed5325020ed0c
As described inline, this crawler is causing us problems as it hits
the backends indiscriminately. Block it via the known UA strings,
which luckily are old so should not cause real client issues.
Change-Id: I0d78a8b625b69f600e00e8b3ea64576e0fdb84d9
This adds an option to have an Apache based reverse proxy on port 3081
forwarding to 3000. The idea is that we can use some of the Apache
filtering rules to reject certain traffic if/when required.
It is off by default, but tested in the gate.
Change-Id: Ie34772878d9fb239a5f69f2d7b993cc1f2142930
We want to use stop_grace_period to manage gerrit service stops. This
feature was added in docker-compose 1.10 but the distro provides 1.5.
Work around this by installing docker-compose from pypi.
This seems like a useful feature and we want to manage docker-compose
the same way globally so move docker-compose installation into the
install-docker role.
New docker-compose has slightly different output that we must check for
in the gitea start/stop machinery. We also need to check for different
container name formatting in our test cases. We should pause here and
consider if this has any upgrade implications for our existing services.
Change-Id: Ia8249a2b84a2ef167ee4ffd66d7a7e7cff8e21fb
Currently we don't have any logs from our gitea sshd processes because
sshd logs to syslog by default and /dev/log isn't in our containers. You
can ask sshd nicely to log to stderr instead with the -e flag which
docker will pick up and store for us.
Update the sshd command to include -e then use testinfra to check we
collect logs and they are accssible from docker.
Change-Id: Ib7d6d405554c3c30be410bc08c6fee7d4363b096
Add new IP addresses to inventory for the rebuild, but don't
reactivate it in the haproxy pools yet.
Note this switches the gitea testing to use a host called gitea99 so
that it doesn't conflict with our changes of the production hosts.
Change-Id: I9779e16cca423bcf514dd3a8d9f14e91d43f1ca3
Git can segfault and cause a gitea error due to the size of the
openstack/openstack repo. Give it more stack space.
The hard limit is a workaround for
https://github.com/moby/moby/issues/39125
Change-Id: Ibce79d8ab27af3070bf9c5f584d0d78f2b266388
