Verified Commit 7d6e6c6a authored by Eliot Berriot's avatar Eliot Berriot
Browse files

Merge branch 'release/0.14'

parents 7b18e46f 73bde2fc
......@@ -11,3 +11,4 @@ WEBPACK_DEVSERVER_PORT=8080
......@@ -10,6 +10,210 @@ This changelog is viewable on the web at
.. towncrier
0.14 (2018-06-02)
Upgrade instructions are available at
- Admins can now configure default permissions that will be granted to all
registered users (#236)
- Files management interface for users with "library" permission (#223)
- New action table component for quick and efficient batch actions (#228) This
is implemented on the federated tracks pages, but will be included in other
pages as well depending on the feedback.
- Added a new "upload" permission that allows user to launch import and view
their own imports (#230)
- Added Support for OggTheora in import.
- Autoremove media files on model instance deletion (#241)
- Can now import a whole remote library at once thanks to new Action Table
component (#164)
- Can now use album covers from flac/mp3 metadata and separate file in track
directory (#219)
- Implemented getCovertArt in Subsonic API to serve album covers (#258)
- Implemented scrobble endpoint of subsonic API, listenings are now tracked
correctly from third party apps that use this endpoint (#260)
- Retructured music API to increase performance and remove useless endpoints
- Consistent constraints/checks for URL size (#207)
- Display proper total number of tracks on radio detail (#225)
- Do not crash on flac import if musicbrainz tags are missing (#214)
- Empty save button in radio builder (#226)
- Ensure anonymous users can use the app if the instance is configured
accordingly (#229)
- Ensure inactive users cannot get auth tokens (#218) This was already the case
bug we missed some checks
- File-upload import now supports Flac files (#213)
- File-upload importer should now work properly, assuming files are tagged
- Fixed a few broken translations strings (#227)
- Fixed broken ordering in front-end lists (#179)
- Fixed ignored page_size paremeter on artist and favorites list (#240)
- Read ID3Tag Tracknumber from TRCK (#220)
- We now fetch album covers regardless of the import methods (#231)
- Added missing subsonic configuration block in deployment vhost files (#249)
- Moved upgrade doc under install doc in TOC (#251)
- Removed acoustid support, as the integration was buggy and error-prone (#106)
Files management interface
This is the first bit of an ongoing work that will span several releases, to
bring more powerful library management features to Funkwhale. This iteration
includes a basic file management interface where users with the "library"
permission can list and search available files, order them using
various criterias (size, bitrate, duration...) and delete them.
New "upload" permission
This new permission is helpful if you want to give upload/import rights
to some users, but don't want them to be able to manage the library as a whole:
although there are no controls yet for managing library in the fron-end,
subsequent release will introduce management interfaces for artists, files,
Because of that, users with the "library" permission will have much more power,
and will also be able to remove content from the platform. On the other hand,
users with the "upload" permission will only have the ability to add new
Also, this release also includes a new feature called "default permissions":
those are permissions that are granted to every users on the platform.
On public/open instances, this will play well with the "upload" permission
since everyone will be able to contribute to the instance library without
an admin giving the permission to every single user.
Smarter album cover importer
In earlier versions, covers where only imported when launching a YouTube import.
Starting from this release, covers will be imported regardless of the import mode
(file upload, youtube-dl, CLI, in-place...). Funkwhale will look for covers
in the following order:
1. In the imported file itself (FLAC/MP3 only)
2. In a cover.jpg or cover.png in the file directory
3. By fetching cover art from Musibrainz, assuming the file is tagged correctly
This will only work for newly imported tracks and albums though. In the future,
we may offer an option to refetch album covers from the interface, but in the
meantime, you can use the following snippet:
.. code-block:: python
# Store this in /tmp/
from import Album, TrackFile
from import update_album_cover
albums_without_covers = Album.objects.filter(cover='')
total = albums_without_covers.count()
print('Found {} albums without cover'.format(total))
for i, album in enumerate(albums_without_covers.iterator()):
print('[{}/{}] Fetching cover for {}...'.format(i+1, total, album.title))
f = TrackFile.objects.filter(track__album=album).filter(source__startswith='file://').first()
update_album_cover(album, track_file=f)
Then launch it::
# docker setups
cat /tmp/ | docker-compose run --rm api python shell -i python
# non-docker setups
source /srv/funkwhale/load_env
source /srv/funkwhale/virtualenv/bin/activate
cat /tmp/ | python shell -i python
# cleanup
rm /tmp/
.. note::
Depending on your number of albums, the previous snippet may take some time
to execute. You can interrupt it at any time using ctrl-c and relaunch it later,
as it's idempotent.
Music API changes
This release includes an API break. Even though the API is advertised
as unstable, and not documented, here is a brief explanation of the change in
case you are using the API in a client or in a script. Summary of the changes:
- ``/api/v1/artists`` does not includes a list of tracks anymore. It was to heavy
to return all of this data all the time. You can get all tracks for an
artist using ``/api/v1/tracks?artist=artist_id``
- Additionally, ``/api/v1/tracks`` now support an ``album`` filter to filter
tracks matching an album
- ``/api/v1/artists/search``, ``/api/v1/albums/search`` and ``/api/v1/tracks/search``
endpoints are removed. Use ``/api/v1/{artists|albums|tracks}/?q=yourquery``
instead. It's also more powerful, since you can combine search with other
filters and ordering options.
- ``/api/v1/requests/import-requests/search`` endpoint is removed as well.
Use ``/api/v1/requests/import-requests/?q=yourquery``
instead. It's also more powerful, since you can combine search with other
filters and ordering options.
Of course, the front-end was updated to work with the new API, so this should
not impact end-users in any way, apart from slight performance gains.
.. note::
The API is still not stable and may evolve again in the future. API freeze
will come at a later point.
Flac files imports via upload
You have nothing to do to benefit from this, however, since Flac files
tend to be a lot bigger than other files, you may want to increase the
``client_max_body_size`` value in your Nginx configuration if you plan
to upload flac files.
Missing subsonic configuration bloc in vhost files
Because of a missing bloc in the sample Nginx and Apache configurations,
instances that were deployed after the 0.13 release are likely to be unable
to answer to Subsonic clients (the missing bits were properly documented
in the changelog).
Ensure you have the following snippets in your Nginx or Apache configuration
if you plan to use the Subsonic API.
location /rest/ {
include /etc/nginx/funkwhale_proxy.conf;
proxy_pass http://funkwhale-api/api/subsonic/rest/;
<Location "/rest">
ProxyPass ${funkwhale-api}/api/subsonic/rest
ProxyPassReverse ${funkwhale-api}/api/subsonic/rest
0.13 (2018-05-19)
......@@ -38,6 +38,10 @@ v1_patterns += [
('funkwhale_api.instance.urls', 'instance'),
('funkwhale_api.manage.urls', 'manage'),
('funkwhale_api.federation.api_urls', 'federation'),
......@@ -97,6 +97,7 @@ THIRD_PARTY_APPS = (
......@@ -302,6 +303,9 @@ ROOT_URLCONF = 'config.urls'
WSGI_APPLICATION = 'config.wsgi.application'
ASGI_APPLICATION = "config.routing.application"
# This ensures that Django will be able to detect a secure connection
# ------------------------------------------------------------------------------
......@@ -433,12 +437,6 @@ USE_X_FORWARDED_PORT = True
assert REVERSE_PROXY_TYPE in ['apache2', 'nginx'], 'Unsupported REVERSE_PROXY_TYPE'
# Wether we should check user permission before serving audio files (meaning
# return an obfuscated url)
# This require a special configuration on the reverse proxy side
# See for example
# Which path will be used to process the internal redirection
# **DO NOT** put a slash at the end
PROTECT_FILES_PATH = env('PROTECT_FILES_PATH', default='/_protected')
......@@ -76,3 +76,4 @@ LOGGING = {
......@@ -22,10 +22,6 @@ from .common import * # noqa
# Raises ImproperlyConfigured exception if DJANGO_SECRET_KEY not in os.environ
# This ensures that Django will be able to detect a secure connection
# properly on Heroku.
# django-secure
# ------------------------------------------------------------------------------
# INSTALLED_APPS += ("djangosecure", )
# -*- coding: utf-8 -*-
__version__ = '0.13'
__version__ = '0.14'
__version_info__ = tuple([int(num) if num.isdigit() else num for num in __version__.replace('-', '.', 1).split('.')])
......@@ -3,4 +3,4 @@ from rest_framework.pagination import PageNumberPagination
class FunkwhalePagination(PageNumberPagination):
page_size_query_param = 'page_size'
max_page_size = 25
max_page_size = 50
from django.conf import settings
from django import forms
from dynamic_preferences import serializers
from dynamic_preferences import types
from dynamic_preferences.registries import global_preferences_registry
......@@ -10,3 +14,38 @@ class DefaultFromSettingMixin(object):
def get(pref):
manager = global_preferences_registry.manager()
return manager[pref]
class StringListSerializer(serializers.BaseSerializer):
separator = ','
sort = True
def to_db(cls, value, **kwargs):
if not value:
if type(value) not in [list, tuple]:
raise cls.exception(
"Cannot serialize, value {} is not a list or a tuple".format(
if cls.sort:
value = sorted(value)
return cls.separator.join(value)
def to_python(cls, value, **kwargs):
if not value:
return []
return value.split(',')
class StringListPreference(types.BasePreferenceType):
serializer = StringListSerializer
field_class = forms.MultipleChoiceField
def get_api_additional_data(self):
d = super(StringListPreference, self).get_api_additional_data()
d['choices'] = self.get('choices')
return d
from rest_framework import serializers
class ActionSerializer(serializers.Serializer):
A special serializer that can operate on a list of objects
and apply actions on it.
action = serializers.CharField(required=True)
objects = serializers.JSONField(required=True)
filters = serializers.DictField(required=False)
actions = None
filterset_class = None
# those are actions identifier where we don't want to allow the "all"
# selector because it's to dangerous. Like object deletion.
dangerous_actions = []
def __init__(self, *args, **kwargs):
self.queryset = kwargs.pop('queryset')
if self.actions is None:
raise ValueError(
'You must declare a list of actions on '
'the serializer class')
for action in self.actions:
handler_name = 'handle_{}'.format(action)
assert hasattr(self, handler_name), (
'{} miss a {} method'.format(
self.__class__.__name__, handler_name)
super().__init__(self, *args, **kwargs)
def validate_action(self, value):
if value not in self.actions:
raise serializers.ValidationError(
'{} is not a valid action. Pick one of {}.'.format(
value, ', '.join(self.actions)
return value
def validate_objects(self, value):
qs = None
if value == 'all':
return self.queryset.all().order_by('id')
if type(value) in [list, tuple]:
return self.queryset.filter(pk__in=value).order_by('id')
raise serializers.ValidationError(
'{} is not a valid value for objects. You must provide either a '
'list of identifiers or the string "all".'.format(value))
def validate(self, data):
dangerous = data['action'] in self.dangerous_actions
if dangerous and self.initial_data['objects'] == 'all':
raise serializers.ValidationError(
'This action is to dangerous to be applied to all objects')
if self.filterset_class and 'filters' in data:
qs_filterset = self.filterset_class(
data['filters'], queryset=data['objects'])
assert qs_filterset.form.is_valid()
except (AssertionError, TypeError):
raise serializers.ValidationError('Invalid filters')
data['objects'] = qs_filterset.qs
data['count'] = data['objects'].count()
if data['count'] < 1:
raise serializers.ValidationError(
'No object matching your request')
return data
def save(self):
handler_name = 'handle_{}'.format(self.validated_data['action'])
handler = getattr(self, handler_name)
result = handler(self.validated_data['objects'])
payload = {
'updated': self.validated_data['count'],
'action': self.validated_data['action'],
'result': result,
return payload
......@@ -3,7 +3,6 @@ from django.conf import settings
from rest_framework import serializers
from funkwhale_api.activity import serializers as activity_serializers
from import TrackSerializerNested
from import TrackActivitySerializer
from funkwhale_api.users.serializers import UserActivitySerializer
......@@ -35,7 +34,6 @@ class TrackFavoriteActivitySerializer(activity_serializers.ModelSerializer):
class UserTrackFavoriteSerializer(serializers.ModelSerializer):
# track = TrackSerializerNested(read_only=True)
class Meta:
model = models.TrackFavorite
fields = ('id', 'track', 'creation_date')
......@@ -12,12 +12,6 @@ from . import models
from . import serializers
class CustomLimitPagination(pagination.PageNumberPagination):
page_size = 100
page_size_query_param = 'page_size'
max_page_size = 100
class TrackFavoriteViewSet(mixins.CreateModelMixin,
......@@ -26,7 +20,6 @@ class TrackFavoriteViewSet(mixins.CreateModelMixin,
serializer_class = serializers.UserTrackFavoriteSerializer
queryset = (models.TrackFavorite.objects.all())
permission_classes = [ConditionalAuthentication]
pagination_class = CustomLimitPagination
def create(self, request, *args, **kwargs):
serializer = self.get_serializer(
......@@ -24,7 +24,7 @@ class LibraryFilter(django_filters.FilterSet):
class LibraryTrackFilter(django_filters.FilterSet):
library = django_filters.CharFilter('library__uuid')
imported = django_filters.CharFilter(method='filter_imported')
status = django_filters.CharFilter(method='filter_status')
q = fields.SearchFilter(search_fields=[
......@@ -32,11 +32,15 @@ class LibraryTrackFilter(django_filters.FilterSet):
def filter_imported(self, queryset, field_name, value):
if value.lower() in ['true', '1', 'yes']:
queryset = queryset.filter(local_track_file__isnull=False)
elif value.lower() in ['false', '0', 'no']:
queryset = queryset.filter(local_track_file__isnull=True)
def filter_status(self, queryset, field_name, value):
if value == 'imported':
return queryset.filter(local_track_file__isnull=False)
elif value == 'not_imported':
return queryset.filter(
elif value == 'import_pending':
return queryset.filter(import_jobs__status='pending')
return queryset
class Meta:
# Generated by Django 2.0.4 on 2018-05-21 17:02
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('federation', '0005_auto_20180413_1723'),
operations = [
field=models.URLField(max_length=500, unique=True),
......@@ -139,7 +139,7 @@ class Library(models.Model):
uuid = models.UUIDField(default=uuid.uuid4)
url = models.URLField()
url = models.URLField(max_length=500)
# use this flag to disable federation with a library
federation_enabled = models.BooleanField()
......@@ -166,8 +166,8 @@ def get_file_path(instance, filename):
class LibraryTrack(models.Model):
url = models.URLField(unique=True)
audio_url = models.URLField()
url = models.URLField(unique=True, max_length=500)
audio_url = models.URLField(max_length=500)
audio_mimetype = models.CharField(max_length=200)
audio_file = models.FileField(
......@@ -10,8 +10,11 @@ from rest_framework import serializers
from dynamic_preferences.registries import global_preferences_registry
from funkwhale_api.common import utils as funkwhale_utils
from funkwhale_api.common import serializers as common_serializers
from import models as music_models
from import tasks as music_tasks
from . import activity
from . import filters
from . import models
from . import utils
......@@ -26,16 +29,16 @@ logger = logging.getLogger(__name__)
class ActorSerializer(serializers.Serializer):
id = serializers.URLField()
outbox = serializers.URLField()
inbox = serializers.URLField()
id = serializers.URLField(max_length=500)
outbox = serializers.URLField(max_length=500)
inbox = serializers.URLField(max_length=500)
type = serializers.ChoiceField(choices=models.TYPE_CHOICES)
preferredUsername = serializers.CharField()
manuallyApprovesFollowers = serializers.NullBooleanField(required=False)
name = serializers.CharField(required=False, max_length=200)
summary = serializers.CharField(max_length=None, required=False)
followers = serializers.URLField(required=False, allow_null=True)
following = serializers.URLField(required=False, allow_null=True)
followers = serializers.URLField(max_length=500, required=False, allow_null=True)
following = serializers.URLField(max_length=500, required=False, allow_null=True)
publicKey = serializers.JSONField(required=False)
def to_representation(self, instance):
......@@ -224,7 +227,7 @@ class APILibraryFollowUpdateSerializer(serializers.Serializer):
class APILibraryCreateSerializer(serializers.ModelSerializer):
actor = serializers.URLField()
actor = serializers.URLField(max_length=500)
federation_enabled = serializers.BooleanField()
uuid = serializers.UUIDField(read_only=True)
......@@ -293,6 +296,7 @@ class APILibraryCreateSerializer(serializers.ModelSerializer):
class APILibraryTrackSerializer(serializers.ModelSerializer):
library = APILibrarySerializer()
status = serializers.SerializerMethodField()
class Meta:
model = models.LibraryTrack
......@@ -311,13 +315,25 @@ class APILibraryTrackSerializer(serializers.ModelSerializer):
def get_status(self, o):
if o.local_track_file is not None:
return 'imported'
except music_models.TrackFile.DoesNotExist:
for job in o.import_jobs.all():
if job.status == 'pending':
return 'import_pending'
return 'not_imported'
class FollowSerializer(serializers.Serializer):
id = serializers.URLField()
object = serializers.URLField()
actor = serializers.URLField()
id = serializers.URLField(max_length=500)
object = serializers.URLField(max_length=500)
actor = serializers.URLField(max_length=500)