value too long for type character varying(255) during import_files
Steps to reproduce
- Arrange for there to be a file with an extremely long title in your music directory. In my case, the file is:
/music/Music/Godspeed You! Black Emperor/2000 - Lift Your Skinny Fists Like Antennas to Heaven!/2-02 Antennas to Heaven (Moya Sings _Baby-O_... _ Edgyswingsetacid _ [Glockenspiel Duet Recorded on a Campsite in Rhinebeck, N.Y.] _ _Attention... Mon Ami...mp3
and the ID3 title tag isAntennas to Heaven (Moya Sings "Baby-O"... / Edgyswingsetacid / [Glockenspiel Duet Recorded on a Campsite in Rhinebeck, N.Y.] / "Attention... Mon Ami... Fa-Lala-Lala-La-La..." [55-St.Laurent] / She Dreamt She Was a Bulldozer, She Dreamt She Was Alone in an Empty Field / Deathkamp Drone / [Antennas to Heaven...])
. - Do an in-place import from the directory containing Funkwhale's
docker-compose.yml
file by running:docker-compose run --rm api python manage.py import_files --recursive --replace --in-place --prune --no-input --verbosity=3 a48fe69b-2993-4ab7-b894-93f7b097ba29 /music/Music
(Obviously if you try to replicate this you'll have to substitute your own library ID and directory path.)
What happens?
When the scanner encounters the file with an extremely long title tag, an exception of type psycopg2.errors.StringDataRightTruncation
is raised:
File #35058 at file:///music/Music/Godspeed You! Black Emperor/2000 - Lift Your Skinny Fists Like Antennas to Heaven!/2-02 Antennas to Heaven (Moya Sings _Baby-O_... _ Edgyswingsetacid _ [Glockenspiel Duet Recorded on a Campsite in Rhinebeck, N.Y.] _ _Attention... Mon Ami...mp3 was modified, updating metadata…
Traceback (most recent call last):
File "/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(255)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 27, in <module>
execute_from_command_line(sys.argv)
File "/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/app/funkwhale_api/music/management/commands/import_files.py", line 244, in handle
return self._handle(*args, **kwargs)
File "/app/funkwhale_api/music/management/commands/import_files.py", line 331, in _handle
checked_paths = check_updates(
File "/app/funkwhale_api/music/management/commands/import_files.py", line 799, in check_updates
check_upload(stdout, upload)
File "/app/funkwhale_api/music/management/commands/import_files.py", line 832, in check_upload
tasks.update_track_metadata(upload.get_metadata(), track)
File "/usr/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/app/funkwhale_api/music/tasks.py", line 938, in update_track_metadata
obj.save(update_fields=obj_updated_fields)
File "/app/funkwhale_api/music/models.py", line 580, in save
super().save(**kwargs)
File "/app/funkwhale_api/music/models.py", line 137, in save
return super().save(**kwargs)
File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 739, in save
self.save_base(using=using, force_insert=force_insert,
File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
updated = self._save_table(
File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 858, in _save_table
updated = self._do_update(base_qs, using, pk_val, values, update_fields,
File "/venv/lib/python3.8/site-packages/django/db/models/base.py", line 912, in _do_update
return filtered._update(values) > 0
File "/venv/lib/python3.8/site-packages/django/db/models/query.py", line 802, in _update
return query.get_compiler(self.db).execute_sql(CURSOR)
File "/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
cursor = super().execute_sql(result_type)
File "/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
cursor.execute(sql, params)
File "/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
File "/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
django.db.utils.DataError: value too long for type character varying(255)
ERROR: 1
Full output here: import_files_output.txt
The postgres_1
container also logs this error:
ERROR: value too long for type character varying(255)
STATEMENT: UPDATE "music_track" SET "title" = 'Antennas to Heaven (Moya Sings "Baby-O"... / Edgyswingsetacid / [Glockenspiel Duet Recorded on a Campsite in Rhinebeck, N.Y.] / "Attention... Mon Ami... Fa-Lala-Lala-La-La..." [55-St.Laurent] / She Dreamt She Was a Bulldozer, She Dreamt She Was Alone in an Empty Field / Deathkamp Drone / [Antennas to Heaven...])' WHERE "music_track"."id" = 23986
What is expected?
Files with extremely long titles are added/updated successfully.
Perhaps the titles would be truncated if they're longer than 255 characters.
Context
Funkwhale version(s) affected: 1.2.4, 1.2.5, probably others.
Docker version 20.10.15, build fd82621
docker-compose version 1.29.2, build 5becea4c
Ubuntu 22.04 LTS
Linux keep 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux