Verified Commit 7d6e6c6a authored by Agate's avatar Agate 💬

Merge branch 'release/0.14'

parents 7b18e46f 73bde2fc
......@@ -11,3 +11,4 @@ WEBPACK_DEVSERVER_PORT=8080
MUSIC_DIRECTORY_PATH=/music
BROWSABLE_API_ENABLED=True
CACHEOPS_ENABLED=False
FORWARDED_PROTO=http
......@@ -10,6 +10,210 @@ This changelog is viewable on the web at https://docs.funkwhale.audio/changelog.
.. towncrier
0.14 (2018-06-02)
-----------------
Upgrade instructions are available at
https://docs.funkwhale.audio/upgrading.html
Features:
- Admins can now configure default permissions that will be granted to all
registered users (#236)
- Files management interface for users with "library" permission (#223)
- New action table component for quick and efficient batch actions (#228) This
is implemented on the federated tracks pages, but will be included in other
pages as well depending on the feedback.
Enhancements:
- Added a new "upload" permission that allows user to launch import and view
their own imports (#230)
- Added Support for OggTheora in import.
- Autoremove media files on model instance deletion (#241)
- Can now import a whole remote library at once thanks to new Action Table
component (#164)
- Can now use album covers from flac/mp3 metadata and separate file in track
directory (#219)
- Implemented getCovertArt in Subsonic API to serve album covers (#258)
- Implemented scrobble endpoint of subsonic API, listenings are now tracked
correctly from third party apps that use this endpoint (#260)
- Retructured music API to increase performance and remove useless endpoints
(#224)
Bugfixes:
- Consistent constraints/checks for URL size (#207)
- Display proper total number of tracks on radio detail (#225)
- Do not crash on flac import if musicbrainz tags are missing (#214)
- Empty save button in radio builder (#226)
- Ensure anonymous users can use the app if the instance is configured
accordingly (#229)
- Ensure inactive users cannot get auth tokens (#218) This was already the case
bug we missed some checks
- File-upload import now supports Flac files (#213)
- File-upload importer should now work properly, assuming files are tagged
(#106)
- Fixed a few broken translations strings (#227)
- Fixed broken ordering in front-end lists (#179)
- Fixed ignored page_size paremeter on artist and favorites list (#240)
- Read ID3Tag Tracknumber from TRCK (#220)
- We now fetch album covers regardless of the import methods (#231)
Documentation:
- Added missing subsonic configuration block in deployment vhost files (#249)
- Moved upgrade doc under install doc in TOC (#251)
Other:
- Removed acoustid support, as the integration was buggy and error-prone (#106)
Files management interface
^^^^^^^^^^^^^^^^^^^^^^^^^^
This is the first bit of an ongoing work that will span several releases, to
bring more powerful library management features to Funkwhale. This iteration
includes a basic file management interface where users with the "library"
permission can list and search available files, order them using
various criterias (size, bitrate, duration...) and delete them.
New "upload" permission
^^^^^^^^^^^^^^^^^^^^^^^
This new permission is helpful if you want to give upload/import rights
to some users, but don't want them to be able to manage the library as a whole:
although there are no controls yet for managing library in the fron-end,
subsequent release will introduce management interfaces for artists, files,
etc.
Because of that, users with the "library" permission will have much more power,
and will also be able to remove content from the platform. On the other hand,
users with the "upload" permission will only have the ability to add new
content.
Also, this release also includes a new feature called "default permissions":
those are permissions that are granted to every users on the platform.
On public/open instances, this will play well with the "upload" permission
since everyone will be able to contribute to the instance library without
an admin giving the permission to every single user.
Smarter album cover importer
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In earlier versions, covers where only imported when launching a YouTube import.
Starting from this release, covers will be imported regardless of the import mode
(file upload, youtube-dl, CLI, in-place...). Funkwhale will look for covers
in the following order:
1. In the imported file itself (FLAC/MP3 only)
2. In a cover.jpg or cover.png in the file directory
3. By fetching cover art from Musibrainz, assuming the file is tagged correctly
This will only work for newly imported tracks and albums though. In the future,
we may offer an option to refetch album covers from the interface, but in the
meantime, you can use the following snippet:
.. code-block:: python
# Store this in /tmp/update_albums.py
from funkwhale_api.music.models import Album, TrackFile
from funkwhale_api.music.tasks import update_album_cover
albums_without_covers = Album.objects.filter(cover='')
total = albums_without_covers.count()
print('Found {} albums without cover'.format(total))
for i, album in enumerate(albums_without_covers.iterator()):
print('[{}/{}] Fetching cover for {}...'.format(i+1, total, album.title))
f = TrackFile.objects.filter(track__album=album).filter(source__startswith='file://').first()
update_album_cover(album, track_file=f)
Then launch it::
# docker setups
cat /tmp/update_albums.py | docker-compose run --rm api python manage.py shell -i python
# non-docker setups
source /srv/funkwhale/load_env
source /srv/funkwhale/virtualenv/bin/activate
cat /tmp/update_albums.py | python manage.py shell -i python
# cleanup
rm /tmp/update_albums.py
.. note::
Depending on your number of albums, the previous snippet may take some time
to execute. You can interrupt it at any time using ctrl-c and relaunch it later,
as it's idempotent.
Music API changes
^^^^^^^^^^^^^^^^^
This release includes an API break. Even though the API is advertised
as unstable, and not documented, here is a brief explanation of the change in
case you are using the API in a client or in a script. Summary of the changes:
- ``/api/v1/artists`` does not includes a list of tracks anymore. It was to heavy
to return all of this data all the time. You can get all tracks for an
artist using ``/api/v1/tracks?artist=artist_id``
- Additionally, ``/api/v1/tracks`` now support an ``album`` filter to filter
tracks matching an album
- ``/api/v1/artists/search``, ``/api/v1/albums/search`` and ``/api/v1/tracks/search``
endpoints are removed. Use ``/api/v1/{artists|albums|tracks}/?q=yourquery``
instead. It's also more powerful, since you can combine search with other
filters and ordering options.
- ``/api/v1/requests/import-requests/search`` endpoint is removed as well.
Use ``/api/v1/requests/import-requests/?q=yourquery``
instead. It's also more powerful, since you can combine search with other
filters and ordering options.
Of course, the front-end was updated to work with the new API, so this should
not impact end-users in any way, apart from slight performance gains.
.. note::
The API is still not stable and may evolve again in the future. API freeze
will come at a later point.
Flac files imports via upload
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
You have nothing to do to benefit from this, however, since Flac files
tend to be a lot bigger than other files, you may want to increase the
``client_max_body_size`` value in your Nginx configuration if you plan
to upload flac files.
Missing subsonic configuration bloc in vhost files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Because of a missing bloc in the sample Nginx and Apache configurations,
instances that were deployed after the 0.13 release are likely to be unable
to answer to Subsonic clients (the missing bits were properly documented
in the changelog).
Ensure you have the following snippets in your Nginx or Apache configuration
if you plan to use the Subsonic API.
Nginx::
location /rest/ {
include /etc/nginx/funkwhale_proxy.conf;
proxy_pass http://funkwhale-api/api/subsonic/rest/;
}
Apache2::
<Location "/rest">
ProxyPass ${funkwhale-api}/api/subsonic/rest
ProxyPassReverse ${funkwhale-api}/api/subsonic/rest
</Location>
0.13 (2018-05-19)
-----------------
......
......@@ -38,6 +38,10 @@ v1_patterns += [
include(
('funkwhale_api.instance.urls', 'instance'),
namespace='instance')),
url(r'^manage/',
include(
('funkwhale_api.manage.urls', 'manage'),
namespace='manage')),
url(r'^federation/',
include(
('funkwhale_api.federation.api_urls', 'federation'),
......
......@@ -97,6 +97,7 @@ THIRD_PARTY_APPS = (
'dynamic_preferences',
'django_filters',
'cacheops',
'django_cleanup',
)
......@@ -302,6 +303,9 @@ ROOT_URLCONF = 'config.urls'
WSGI_APPLICATION = 'config.wsgi.application'
ASGI_APPLICATION = "config.routing.application"
# This ensures that Django will be able to detect a secure connection
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
# AUTHENTICATION CONFIGURATION
# ------------------------------------------------------------------------------
AUTHENTICATION_BACKENDS = (
......@@ -433,12 +437,6 @@ USE_X_FORWARDED_PORT = True
REVERSE_PROXY_TYPE = env('REVERSE_PROXY_TYPE', default='nginx')
assert REVERSE_PROXY_TYPE in ['apache2', 'nginx'], 'Unsupported REVERSE_PROXY_TYPE'
# Wether we should check user permission before serving audio files (meaning
# return an obfuscated url)
# This require a special configuration on the reverse proxy side
# See https://wellfire.co/learn/nginx-django-x-accel-redirects/ for example
PROTECT_AUDIO_FILES = env.bool('PROTECT_AUDIO_FILES', default=True)
# Which path will be used to process the internal redirection
# **DO NOT** put a slash at the end
PROTECT_FILES_PATH = env('PROTECT_FILES_PATH', default='/_protected')
......
......@@ -76,3 +76,4 @@ LOGGING = {
},
},
}
CSRF_TRUSTED_ORIGINS = [o for o in ALLOWED_HOSTS]
......@@ -22,10 +22,6 @@ from .common import * # noqa
# Raises ImproperlyConfigured exception if DJANGO_SECRET_KEY not in os.environ
SECRET_KEY = env("DJANGO_SECRET_KEY")
# This ensures that Django will be able to detect a secure connection
# properly on Heroku.
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
# django-secure
# ------------------------------------------------------------------------------
# INSTALLED_APPS += ("djangosecure", )
......
# -*- coding: utf-8 -*-
__version__ = '0.13'
__version__ = '0.14'
__version_info__ = tuple([int(num) if num.isdigit() else num for num in __version__.replace('-', '.', 1).split('.')])
......@@ -3,4 +3,4 @@ from rest_framework.pagination import PageNumberPagination
class FunkwhalePagination(PageNumberPagination):
page_size_query_param = 'page_size'
max_page_size = 25
max_page_size = 50
from django.conf import settings
from django import forms
from dynamic_preferences import serializers
from dynamic_preferences import types
from dynamic_preferences.registries import global_preferences_registry
......@@ -10,3 +14,38 @@ class DefaultFromSettingMixin(object):
def get(pref):
manager = global_preferences_registry.manager()
return manager[pref]
class StringListSerializer(serializers.BaseSerializer):
separator = ','
sort = True
@classmethod
def to_db(cls, value, **kwargs):
if not value:
return
if type(value) not in [list, tuple]:
raise cls.exception(
"Cannot serialize, value {} is not a list or a tuple".format(
value))
if cls.sort:
value = sorted(value)
return cls.separator.join(value)
@classmethod
def to_python(cls, value, **kwargs):
if not value:
return []
return value.split(',')
class StringListPreference(types.BasePreferenceType):
serializer = StringListSerializer
field_class = forms.MultipleChoiceField
def get_api_additional_data(self):
d = super(StringListPreference, self).get_api_additional_data()
d['choices'] = self.get('choices')
return d
from rest_framework import serializers
class ActionSerializer(serializers.Serializer):
"""
A special serializer that can operate on a list of objects
and apply actions on it.
"""
action = serializers.CharField(required=True)
objects = serializers.JSONField(required=True)
filters = serializers.DictField(required=False)
actions = None
filterset_class = None
# those are actions identifier where we don't want to allow the "all"
# selector because it's to dangerous. Like object deletion.
dangerous_actions = []
def __init__(self, *args, **kwargs):
self.queryset = kwargs.pop('queryset')
if self.actions is None:
raise ValueError(
'You must declare a list of actions on '
'the serializer class')
for action in self.actions:
handler_name = 'handle_{}'.format(action)
assert hasattr(self, handler_name), (
'{} miss a {} method'.format(
self.__class__.__name__, handler_name)
)
super().__init__(self, *args, **kwargs)
def validate_action(self, value):
if value not in self.actions:
raise serializers.ValidationError(
'{} is not a valid action. Pick one of {}.'.format(
value, ', '.join(self.actions)
)
)
return value
def validate_objects(self, value):
qs = None
if value == 'all':
return self.queryset.all().order_by('id')
if type(value) in [list, tuple]:
return self.queryset.filter(pk__in=value).order_by('id')
raise serializers.ValidationError(
'{} is not a valid value for objects. You must provide either a '
'list of identifiers or the string "all".'.format(value))
def validate(self, data):
dangerous = data['action'] in self.dangerous_actions
if dangerous and self.initial_data['objects'] == 'all':
raise serializers.ValidationError(
'This action is to dangerous to be applied to all objects')
if self.filterset_class and 'filters' in data:
qs_filterset = self.filterset_class(
data['filters'], queryset=data['objects'])
try:
assert qs_filterset.form.is_valid()
except (AssertionError, TypeError):
raise serializers.ValidationError('Invalid filters')
data['objects'] = qs_filterset.qs
data['count'] = data['objects'].count()
if data['count'] < 1:
raise serializers.ValidationError(
'No object matching your request')
return data
def save(self):
handler_name = 'handle_{}'.format(self.validated_data['action'])
handler = getattr(self, handler_name)
result = handler(self.validated_data['objects'])
payload = {
'updated': self.validated_data['count'],
'action': self.validated_data['action'],
'result': result,
}
return payload
......@@ -3,7 +3,6 @@ from django.conf import settings
from rest_framework import serializers
from funkwhale_api.activity import serializers as activity_serializers
from funkwhale_api.music.serializers import TrackSerializerNested
from funkwhale_api.music.serializers import TrackActivitySerializer
from funkwhale_api.users.serializers import UserActivitySerializer
......@@ -35,7 +34,6 @@ class TrackFavoriteActivitySerializer(activity_serializers.ModelSerializer):
class UserTrackFavoriteSerializer(serializers.ModelSerializer):
# track = TrackSerializerNested(read_only=True)
class Meta:
model = models.TrackFavorite
fields = ('id', 'track', 'creation_date')
......@@ -12,12 +12,6 @@ from . import models
from . import serializers
class CustomLimitPagination(pagination.PageNumberPagination):
page_size = 100
page_size_query_param = 'page_size'
max_page_size = 100
class TrackFavoriteViewSet(mixins.CreateModelMixin,
mixins.DestroyModelMixin,
mixins.ListModelMixin,
......@@ -26,7 +20,6 @@ class TrackFavoriteViewSet(mixins.CreateModelMixin,
serializer_class = serializers.UserTrackFavoriteSerializer
queryset = (models.TrackFavorite.objects.all())
permission_classes = [ConditionalAuthentication]
pagination_class = CustomLimitPagination
def create(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
......
......@@ -24,7 +24,7 @@ class LibraryFilter(django_filters.FilterSet):
class LibraryTrackFilter(django_filters.FilterSet):
library = django_filters.CharFilter('library__uuid')
imported = django_filters.CharFilter(method='filter_imported')
status = django_filters.CharFilter(method='filter_status')
q = fields.SearchFilter(search_fields=[
'artist_name',
'title',
......@@ -32,11 +32,15 @@ class LibraryTrackFilter(django_filters.FilterSet):
'library__actor__domain',
])
def filter_imported(self, queryset, field_name, value):
if value.lower() in ['true', '1', 'yes']:
queryset = queryset.filter(local_track_file__isnull=False)
elif value.lower() in ['false', '0', 'no']:
queryset = queryset.filter(local_track_file__isnull=True)
def filter_status(self, queryset, field_name, value):
if value == 'imported':
return queryset.filter(local_track_file__isnull=False)
elif value == 'not_imported':
return queryset.filter(
local_track_file__isnull=True
).exclude(import_jobs__status='pending')
elif value == 'import_pending':
return queryset.filter(import_jobs__status='pending')
return queryset
class Meta:
......
# Generated by Django 2.0.4 on 2018-05-21 17:02
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('federation', '0005_auto_20180413_1723'),
]
operations = [
migrations.AlterField(
model_name='library',
name='url',
field=models.URLField(max_length=500),
),
migrations.AlterField(
model_name='librarytrack',
name='audio_url',
field=models.URLField(max_length=500),
),
migrations.AlterField(
model_name='librarytrack',
name='url',
field=models.URLField(max_length=500, unique=True),
),
]
......@@ -139,7 +139,7 @@ class Library(models.Model):
on_delete=models.CASCADE,
related_name='library')
uuid = models.UUIDField(default=uuid.uuid4)
url = models.URLField()
url = models.URLField(max_length=500)
# use this flag to disable federation with a library
federation_enabled = models.BooleanField()
......@@ -166,8 +166,8 @@ def get_file_path(instance, filename):
class LibraryTrack(models.Model):
url = models.URLField(unique=True)
audio_url = models.URLField()
url = models.URLField(unique=True, max_length=500)
audio_url = models.URLField(max_length=500)
audio_mimetype = models.CharField(max_length=200)
audio_file = models.FileField(
upload_to=get_file_path,
......
......@@ -10,8 +10,11 @@ from rest_framework import serializers
from dynamic_preferences.registries import global_preferences_registry
from funkwhale_api.common import utils as funkwhale_utils
from funkwhale_api.common import serializers as common_serializers
from funkwhale_api.music import models as music_models
from funkwhale_api.music import tasks as music_tasks
from . import activity
from . import filters
from . import models
from . import utils
......@@ -26,16 +29,16 @@ logger = logging.getLogger(__name__)
class ActorSerializer(serializers.Serializer):
id = serializers.URLField()
outbox = serializers.URLField()
inbox = serializers.URLField()
id = serializers.URLField(max_length=500)
outbox = serializers.URLField(max_length=500)
inbox = serializers.URLField(max_length=500)
type = serializers.ChoiceField(choices=models.TYPE_CHOICES)
preferredUsername = serializers.CharField()
manuallyApprovesFollowers = serializers.NullBooleanField(required=False)
name = serializers.CharField(required=False, max_length=200)
summary = serializers.CharField(max_length=None, required=False)
followers = serializers.URLField(required=False, allow_null=True)
following = serializers.URLField(required=False, allow_null=True)
followers = serializers.URLField(max_length=500, required=False, allow_null=True)
following = serializers.URLField(max_length=500, required=False, allow_null=True)
publicKey = serializers.JSONField(required=False)
def to_representation(self, instance):
......@@ -224,7 +227,7 @@ class APILibraryFollowUpdateSerializer(serializers.Serializer):
class APILibraryCreateSerializer(serializers.ModelSerializer):
actor = serializers.URLField()
actor = serializers.URLField(max_length=500)
federation_enabled = serializers.BooleanField()
uuid = serializers.UUIDField(read_only=True)
......@@ -293,6 +296,7 @@ class APILibraryCreateSerializer(serializers.ModelSerializer):
class APILibraryTrackSerializer(serializers.ModelSerializer):
library = APILibrarySerializer()
status = serializers.SerializerMethodField()
class Meta:
model = models.LibraryTrack
......@@ -311,13 +315,25 @@ class APILibraryTrackSerializer(serializers.ModelSerializer):
'title',
'library',
'local_track_file',
'status',
]
def get_status(self, o):
try:
if o.local_track_file is not None:
return 'imported'
except music_models.TrackFile.DoesNotExist:
pass
for job in o.import_jobs.all():
if job.status == 'pending':
return 'import_pending'
return 'not_imported'