Problem with cache sharing between concurrent runners
I've repeatedly been seeing this issue with some of the CI builds: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1151
A while back I changed the gitlab-runner settings to concurrent = 2
which means it can run up to two jobs simultaneously. The problem, which seems absurd to me (and completely contradictory to the point of caching) is that each cache is local to a specific runner. Which means if I have one build stage that populates the cache (e.g. "setup" in our pipeline) and another stage (e.g. "test") which needs to restore the cache, restoring the cache from the previous stage will fail if it the second job is run on a separate runner from the job that populated the cache.
This is a several year old issue on GitLab, and still doesn't seem to be resolved. You can used a shared external cache (e.g. on S3 or GCS) but that seems an absurd workaround when it would seem just as easy to have a local cache that's shared between runners. I'll look into it.