In-place imports abort after 300 seconds
Steps to reproduce
- Navigate to a library. This will be at a URL like https://funkwhale.example.com/library/${uuid}/
- Navigate to the "import" tab.
- Start an in-place import that takes 300+ seconds. Anecdotally, importing ~10k files from spinning rust does the trick.
What happens?
The import fails part way through execution. A message like the following appears in the logs:
funkwhale-celeryworker_1 | [2021-05-07 20:01:34,223: WARNING/ForkPoolWorker-4] Handling batch 10 (1000 items) - running for 284s, previous batch took 25s
funkwhale-celeryworker_1 | [2021-05-07 20:01:34,277: WARNING/ForkPoolWorker-4] Import summary:
funkwhale-celeryworker_1 | [2021-05-07 20:01:34,277: WARNING/ForkPoolWorker-4] - 1000 files found matching this pattern: ['/srv/funkwhale/data/music/Music']
funkwhale-celeryworker_1 | [2021-05-07 20:01:34,278: WARNING/ForkPoolWorker-4] - 0 files already found in database
funkwhale-celeryworker_1 | [2021-05-07 20:01:34,278: WARNING/ForkPoolWorker-4] - 1000 new files
funkwhale-celeryworker_1 | 2021-05-07 20:01:47,634 funkwhale_api.music.tasks INFO [Album 2466] scanning covers from ...
funkwhale-celeryworker_1 | 2021-05-07 20:01:47,658 funkwhale_api.music.tasks INFO [Album 2466] scanning covers from ...
funkwhale-celeryworker_1 | [2021-05-07 20:01:49,475: ERROR/MainProcess] Task handler raised error: TimeLimitExceeded(300)
funkwhale-celeryworker_1 | Traceback (most recent call last):
funkwhale-celeryworker_1 | File "/venv/lib/python3.7/site-packages/billiard/pool.py", line 684, in on_hard_timeout
funkwhale-celeryworker_1 | raise TimeLimitExceeded(job._timeout)
funkwhale-celeryworker_1 | billiard.exceptions.TimeLimitExceeded: TimeLimitExceeded(300,)
funkwhale-celeryworker_1 | [2021-05-07 20:01:49,475: ERROR/MainProcess] Hard time limit (300s) exceeded for music.fs_import[ca6a57c3-ac7d-4a28-af13-a2c9fc56cde7]
funkwhale-celeryworker_1 | [2021-05-07 20:01:49,596: ERROR/MainProcess] Process 'ForkPoolWorker-4' pid:13 exited with 'signal 9 (SIGKILL)'
What is expected?
The import completes.
Context
Funkwhale version(s) affected: 1.1.1
Deployed on Docker, though I doubt it matters.
The only documented setting with a default value of 300 is DB_CONN_MAX_AGE. I tried bumping it, but that had no effect.
CELERY_TASK_TIME_LIMIT
also has a default value of 300. I expect that bumping that value would fix the issue.
However, the value isn't read from the environment, so I can't bump it with funkwhale.env
.
That said, bumping that value seems a bit awkward. Not all of my libraries are this large. Some are small, with just a couple hundred files. I don't want a borked import of that size to hang for 30 minutes before dying. And if workers are used for other tasks like transcoding, I don't want them to live for 30 minutes. Could imports be chunked, so that each import of 1,000 files is a different task?