Random UniqueViolation duplicate key error during track processing by celery workers
Steps to reproduce
- Upload album
- Get an error randomly on some tracks
What happens?
Track upload sometime triggers an error during upload processing. Relaunching the processing, the track gets processed just fine the second time. The rate of errors is low, once on 57 tracks last upload.
What is expected?
The album is fully imported without any error
Context
Funkwhale version(s) affected: 0.20, custom setup based on the funkwhale/funkwhale
docker image, in a Kubernetes cluster.
As this setup is using four concurrent celery workers, there is likely a race condition involved.
Worker stack trace:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "music_album_mbid_b09e59fe_uniq"
DETAIL: Key (mbid)=(020cf4a5-a8d2-4b3a-986e-ce2594349639) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/celery/app/trace.py", line 385, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/celery/app/trace.py", line 648, in __protected_call__
return self.run(*args, **kwargs)
File "/app/funkwhale_api/taskapp/celery.py", line 51, in inner
return function(*args, **kwargs)
File "/app/funkwhale_api/music/tasks.py", line 202, in process_upload
final_metadata, attributed_to=upload.library.actor
File "/usr/lib/python3.6/contextlib.py", line 52, in inner
return func(*args, **kwds)
File "/app/funkwhale_api/music/tasks.py", line 399, in get_track_from_import_metadata
track = _get_track(data, attributed_to=attributed_to)
File "/app/funkwhale_api/music/tasks.py", line 536, in _get_track
models.Album, query, defaults=defaults, sort_fields=["mbid", "fid"]
File "/app/funkwhale_api/music/tasks.py", line 362, in get_best_candidate_or_create
return model.objects.create(**defaults), True
File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 422, in create
obj.save(force_insert=True, using=self.db)
File "/app/funkwhale_api/music/models.py", line 123, in save
return super().save(**kwargs)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 741, in save
force_update=force_update, update_fields=update_fields)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 779, in save_base
force_update, using, update_fields,
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 870, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 908, in _do_insert
using=using, raw=raw)
File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1335, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python3.6/site-packages/raven/contrib/django/client.py", line 127, in execute
return real_execute(self, sql, params)
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "music_album_mbid_b09e59fe_uniq"
DETAIL: Key (mbid)=(020cf4a5-a8d2-4b3a-986e-ce2594349639) already exists.