By default gitea caches everything in memory using a Go hashmap. There
is suspicion that the now many persistent AI web crawlers cause this
hashmap to grow in ways that eventually cause the Go GC system to pause
the world in noticeable ways when loading pages. Restarting the gitea
services seems to temporarily reset things (as it should with an in
memory map) until we cross some threshold and things become slow again.
The good news is that gitea supports several backends (called adapters)
for the cache. We elect to use memcached because it is relatively
simple and has a FOSS license (unlike redis). The other alternative we
could consider is twoqueue which also caches within memory in the Go
runtime but allows for setting a size limit. I've gone with memcached
because it doesn't rely on Golang GC, but twoqueue is likely simpler if
we want to start there.
Note we also bump the job timeout to 5400 seconds (90 minutes) from 4800
seconds (80 minutes) because a run on ovh-gra1 timed out while running
testinfra test cases (the very end of the job). It is possible that
using memcache is slightly slower than using in process memory caching,
but the goal here isn't to make things faster it is to make things more
consistent over time. As long as memcached performance is within the
same ballpark and doesn't degrade over time this is acceptable.
Change-Id: Ie9ca246a8321fe84d9a1582e35cd4c5459b48bee
67662bb735