Failing docker multiarch builds in CI
We have from time to time the issue that our docker multiarch builds are failing like this: https://dev.funkwhale.audio/funkwhale/funkwhale/-/jobs/184619
#78 [api linux/arm/v7 builder 6/7] RUN --mount=type=cache,target=~/.cache/pip; set -eux; pip3 install --upgrade pip; pip3 install setuptools wheel; grep -Ev 'cryptography|lxml|pillow|psycopg2|watchfiles' /requirements.txt | pip3 install -r /dev/stdin cryptography==38.0.3 lxml==4.9.3 pillow==9.3.0 psycopg2==2.9.5 watchfiles==0.18.1
#78 599.7 Preparing metadata (pyproject.toml): still running...
#78 599.8 Preparing metadata (pyproject.toml): finished with status 'error'
#78 599.9 error: subprocess-exited-with-error
#78 599.9
#78 599.9 × Preparing metadata (pyproject.toml) did not run successfully.
#78 599.9 │ exit code: 1
#78 599.9 ╰─> [16 lines of output]
#78 599.9 Checking for Rust toolchain....
#78 599.9 Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-q9l4ifpq --interpreter /venv/bin/python3`
#78 599.9 Updating crates.io index
#78 599.9 warning: spurious network error (2 tries remaining): failed to mmap. Could not write data: Out of memory; class=Os (2)
#78 599.9 warning: spurious network error (1 tries remaining): failed to mmap. Could not write data: Out of memory; class=Os (2)
#78 599.9 error: Unable to update registry `crates-io`
#78 599.9
#78 599.9 Caused by:
#78 599.9 failed to fetch `https://github.com/rust-lang/crates.io-index`
#78 599.9
#78 599.9 Caused by:
#78 599.9 failed to mmap. Could not write data: Out of memory; class=Os (2)
#78 599.9 💥 maturin failed
#78 599.9 Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?
#78 599.9 Caused by: `cargo metadata` exited with an error:
#78 599.9 Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', '/tmp/pip-modern-metadata-q9l4ifpq', '--interpreter', '/venv/bin/python3']' returned non-zero exit status 1.
#78 599.9 [end of output]
#78 599.9
#78 599.9 note: This error originates from a subprocess, and is likely not a problem with pip.
#78 599.9 error: metadata-generation-failed
#78 599.9
#78 599.9 × Encountered error while generating package metadata.
#78 599.9 ╰─> See above for output.
#78 599.9
#78 599.9 note: This is an issue with the package mentioned above, not pip.
#78 599.9 hint: See above for details.
#78 ERROR: process "/bin/sh -c set -eux; pip3 install --upgrade pip; pip3 install setuptools wheel; grep -Ev 'cryptography|lxml|pillow|psycopg2|watchfiles' /requirements.txt | pip3 install -r /dev/stdin cryptography==38.0.3 lxml==4.9.3 pillow==9.3.0 psycopg2==2.9.5 watchfiles==0.18.1" did not complete successfully: exit code: 1
------
> [api linux/arm/v7 builder 6/7] RUN --mount=type=cache,target=~/.cache/pip; set -eux; pip3 install --upgrade pip; pip3 install setuptools wheel; grep -Ev 'cryptography|lxml|pillow|psycopg2|watchfiles' /requirements.txt | pip3 install -r /dev/stdin cryptography==38.0.3 lxml==4.9.3 pillow==9.3.0 psycopg2==2.9.5 watchfiles==0.18.1:
#78 599.9 [end of output]
#78 599.9
#78 599.9 note: This error originates from a subprocess, and is likely not a problem with pip.
#78 599.9 error: metadata-generation-failed
#78 599.9
#78 599.9 × Encountered error while generating package metadata.
#78 599.9 ╰─> See above for output.
#78 599.9
#78 599.9 note: This is an issue with the package mentioned above, not pip.
#78 599.9 hint: See above for details.
The issue seems to be a weird combination of docker volumes, qemu and cargo.
More details can be found on Github, there is also a proposed workaround.