Merge branch 'release/0.16.1'

.env.dev

@@ -7,7 +7,7 @@ C_FORCE_ROOT=true
 FUNKWHALE_HOSTNAME=localhost
 FUNKWHALE_PROTOCOL=http
 PYTHONDONTWRITEBYTECODE=true
-WEBPACK_DEVSERVER_PORT=8080
+VUE_PORT=8080
 MUSIC_DIRECTORY_PATH=/music
 BROWSABLE_API_ENABLED=True
 FORWARDED_PROTO=http

.gitignore

@@ -72,16 +72,17 @@ api/media
 api/staticfiles
 api/static
 api/.pytest_cache
-
+api/celerybeat-*
 # Front
+oldfront/node_modules/
 front/static/translations
 front/node_modules/
 front/dist/
 front/npm-debug.log*
 front/yarn-debug.log*
 front/yarn-error.log*
-front/test/unit/coverage
-front/test/e2e/reports
+front/tests/unit/coverage
+front/tests/e2e/reports
 front/selenium-debug.log
 docs/_build
 

.gitlab-ci.yml

@@ -156,7 +156,6 @@ test_api:
   tags:
     - docker
 
-
 test_front:
   stage: test
   image: node:9
@@ -166,7 +165,7 @@ test_front:
     - branches
   script:
     - yarn install
-    - yarn run unit
+    - yarn test:unit
   cache:
     key: "funkwhale__front_dependencies"
     paths:
@@ -179,7 +178,6 @@ test_front:
   tags:
     - docker
 
-
 build_front:
   stage: build
   image: node:9
@@ -192,8 +190,8 @@ build_front:
     - yarn run i18n-compile
     # this is to ensure we don't have any errors in the output,
     # cf https://code.eliotberriot.com/funkwhale/funkwhale/issues/169
-    - yarn run build | tee /dev/stderr | (! grep -i 'ERROR in')
-    - chmod -R 750 dist
+    - yarn build | tee /dev/stderr | (! grep -i 'ERROR in')
+    - chmod -R 755 dist
   cache:
     key: "funkwhale__front_dependencies"
     paths:
@@ -210,7 +208,6 @@ build_front:
   tags:
     - docker
 
-
 pages:
   stage: test
   image: python:3.6

CHANGELOG

@@ -10,7 +10,82 @@ This changelog is viewable on the web at https://docs.funkwhale.audio/changelog.
 
 .. towncrier
 
-0.16 (unreleased)
+0.16.1 (2018-08-19)
+-------------------
+
+Upgrade instructions are available at
+https://docs.funkwhale.audio/upgrading.html
+
+Features:
+
+- Make funkwhale themable by loading external stylesheets (#456)
+
+Enhancements:
+
+- Add link to admin on "Staff member" button (#202)
+- Can now add a description to radios and better radio cards (#331)
+- Display track duration in track tables (#461)
+- More permissive default permissions for front-end files (#388)
+- Simpler configuration and toolchain for the front-end using vue-cli (!375)
+- Use Howler to manage audio instead of our own dirty/untested code (#392)
+
+
+Bugfixes:
+
+- Fix alignment issue on top bar in Admin tabs (#395)
+- Fix Apache2 permission issue preventing `/media` folder from being served
+  correctly (#389)
+- Fix loading on browse page lists causing them to go down, and dimming over
+  the top bar (#468)
+- Fixed (again): administration section not showing up in sidebar after login
+  (#245)
+- Fixed audio mimetype not showing up on track detail and list (#459)
+- Fixed broken audio playback on Chrome and invisible volume control (#390)
+- Fixed broken federation import on big imports due to missing transaction
+  logic (#397)
+- Fixed crash on artist pages when no cover is available (#457)
+- Fixed favorited status of tracks not appearing in interface (#398)
+- Fixed invitation code not prefilled in form when accessing invitation link
+  (#476)
+- Fixed typos in scheduled tasks configuration (#487)
+- Removed release date error in case of empty date (#478)
+- Removed white on white artist button on hover, on Album page (#393)
+- Smarter date parsing during import by replacing arrow with pendulum (#376)
+- Display public playlists properly for anonymous users (#488)
+
+
+i18n:
+
+- Added portuguese, spanish and german translations
+
+
+Custom themes for Funkwhale
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you ever wanted to give a custom look and feel to your instance, this is now possible.
+
+Check https://docs.funkwhale.audio/configuration.html#theming if you want to know more!
+
+
+Fix Apache2 configuration file for media block [Manual action required]
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The permission scope on the current Apache2 configuration file is too narrow, preventing thumbnails from being served.
+
+On Apache2 setups, you have to replace the following line::
+
+    <Directory /srv/funkwhale/data/media/albums>
+
+with::
+
+    <Directory /srv/funkwhale/data/media>
+
+You can now restart your server::
+
+    sudo systemctl restart apache2
+
+
+0.16 (2018-07-22)
 -----------------
 
 Upgrade instructions are available at
@@ -232,7 +307,7 @@ In the end, the ``volumes`` directives of your containers should look like that:
 
 
 Removed Cacheops dependency
----------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 We removed one of our dependency named django-cacheops. It was unly used in a few places,
 and not playing nice with other dependencies.

CONTRIBUTING → CONTRIBUTING.rst

@@ -23,7 +23,7 @@ As the front-end can work with any Funkwhale server, you can work with the front
 and make it talk with an existing instance (like the demo one, or you own instance, if you have one).
 
 If even that is too much for you, you can also make your changes without any development environment,
-and open a merge request. We will be able to to review your work easily by spawning automatically a
+and open a merge request. We will be able to review your work easily by spawning automatically a
 live version of your changes, thanks to Gitlab Review apps.
 
 Setup front-end only development environment
@@ -43,7 +43,7 @@ Setup front-end only development environment
 4. Launch the development server::
 
     # this will serve the front-end on http://localhost:8000
-    WEBPACK_DEVSERVER_PORT=8000 yarn dev
+    VUE_PORT=8000 yarn serve
 
 5. Make the front-end talk with an existing server (like https://demo.funkwhale.audio),
    by clicking on the corresponding link in the footer
@@ -264,7 +264,7 @@ When working on federation with traefik, ensure you have this in your ``env``::
 
     # This will ensure we don't bind any port on the host, and thus enable
     # multiple instances of funkwhale to be spawned concurrently.
-    WEBPACK_DEVSERVER_PORT_BINDING=
+    VUE_PORT_BINDING=
     # This disable certificate verification
     EXTERNAL_REQUESTS_VERIFY_SSL=false
     # this ensure you don't have incorrect urls pointing to http resources
@@ -466,12 +466,12 @@ Running tests
 
 To run the front-end test suite, use the following command::
 
-    docker-compose -f dev.yml run --rm front yarn run unit
+    docker-compose -f dev.yml run --rm front yarn test:unit
 
 We also support a "watch and test" mode were we continually relaunch
 tests when changes are recorded on the file system::
 
-    docker-compose -f dev.yml run --rm front yarn run unit-watch
+    docker-compose -f dev.yml run --rm front yarn test:unit -w
 
 The latter is especially useful when you are debugging failing tests.
 

README.rst

@@ -26,7 +26,7 @@ Contribute
 ----------
 
 Contribution guidelines as well as development installation instructions
-are outlined in `CONTRIBUTING <CONTRIBUTING>`_.
+are outlined in `CONTRIBUTING <CONTRIBUTING.rst>`_.
 
 Translate
 ^^^^^^^^^

api/config/settings/common.py

@@ -344,9 +344,9 @@ CELERY_BROKER_URL = env(
 # Your common stuff: Below this line define 3rd party library settings
 CELERY_TASK_DEFAULT_RATE_LIMIT = 1
 CELERY_TASK_TIME_LIMIT = 300
-CELERYBEAT_SCHEDULE = {
+CELERY_BEAT_SCHEDULE = {
     "federation.clean_music_cache": {
-        "task": "funkwhale_api.federation.tasks.clean_music_cache",
+        "task": "federation.clean_music_cache",
         "schedule": crontab(hour="*/2"),
         "options": {"expires": 60 * 2},
     }

api/funkwhale_api/__init__.py

 # -*- coding: utf-8 -*-
-__version__ = "0.16"
+__version__ = "0.16.1"
 __version_info__ = tuple(
     [
         int(num) if num.isdigit() else num

api/funkwhale_api/federation/serializers.py

@@ -796,6 +796,8 @@ class LibraryTrackActionSerializer(common_serializers.ActionSerializer):
             jobs.append(job)
 
         music_models.ImportJob.objects.bulk_create(jobs)
-        music_tasks.import_batch_run.delay(import_batch_id=batch.pk)
+        funkwhale_utils.on_commit(
+            music_tasks.import_batch_run.delay, import_batch_id=batch.pk
+        )
 
         return {"batch": {"id": batch.pk}}

api/funkwhale_api/music/metadata.py

-import arrow
+import datetime
 import mutagen
+import pendulum
 from django import forms
 
 NODEFAULT = object()
@@ -101,6 +102,11 @@ class FirstUUIDField(forms.UUIDField):
         return super().to_python(value)
 
 
+def get_date(value):
+    parsed = pendulum.parse(str(value))
+    return datetime.date(parsed.year, parsed.month, parsed.day)
+
+
 VALIDATION = {
     "musicbrainz_artistid": FirstUUIDField(),
     "musicbrainz_albumid": FirstUUIDField(),
@@ -118,7 +124,7 @@ CONF = {
             "title": {},
             "artist": {},
             "album": {},
-            "date": {"field": "date", "to_application": lambda v: arrow.get(v).date()},
+            "date": {"field": "date", "to_application": get_date},
             "musicbrainz_albumid": {},
             "musicbrainz_artistid": {},
             "musicbrainz_recordingid": {"field": "musicbrainz_trackid"},
@@ -134,7 +140,7 @@ CONF = {
             "title": {},
             "artist": {},
             "album": {},
-            "date": {"field": "date", "to_application": lambda v: arrow.get(v).date()},
+            "date": {"field": "date", "to_application": get_date},
             "musicbrainz_albumid": {"field": "MusicBrainz Album Id"},
             "musicbrainz_artistid": {"field": "MusicBrainz Artist Id"},
             "musicbrainz_recordingid": {"field": "MusicBrainz Track Id"},
@@ -148,10 +154,7 @@ CONF = {
             "title": {"field": "TIT2"},
             "artist": {"field": "TPE1"},
             "album": {"field": "TALB"},
-            "date": {
-                "field": "TDRC",
-                "to_application": lambda v: arrow.get(str(v)).date(),
-            },
+            "date": {"field": "TDRC", "to_application": get_date},
             "musicbrainz_albumid": {"field": "MusicBrainz Album Id"},
             "musicbrainz_artistid": {"field": "MusicBrainz Artist Id"},
             "musicbrainz_recordingid": {
@@ -172,10 +175,7 @@ CONF = {
             "title": {},
             "artist": {},
             "album": {},
-            "date": {
-                "field": "date",
-                "to_application": lambda v: arrow.get(str(v)).date(),
-            },
+            "date": {"field": "date", "to_application": get_date},
             "musicbrainz_albumid": {},
             "musicbrainz_artistid": {},
             "musicbrainz_recordingid": {"field": "musicbrainz_trackid"},

api/funkwhale_api/music/models.py

-import datetime
 import os
 import shutil
 import tempfile
 import uuid
 
-import arrow
 import markdown
+import pendulum
 from django.conf import settings
 from django.core.files import File
 from django.core.files.base import ContentFile
@@ -125,9 +124,7 @@ def import_artist(v):
 
 
 def parse_date(v):
-    if len(v) == 4:
-        return datetime.date(int(v), 1, 1)
-    d = arrow.get(v).date()
+    d = pendulum.parse(v).date()
     return d
 
 
@@ -147,7 +144,7 @@ class AlbumQuerySet(models.QuerySet):
 class Album(APIModelMixin):
     title = models.CharField(max_length=255)
     artist = models.ForeignKey(Artist, related_name="albums", on_delete=models.CASCADE)
-    release_date = models.DateField(null=True)
+    release_date = models.DateField(null=True, blank=True)
     release_group_id = models.UUIDField(null=True, blank=True)
     cover = VersatileImageField(
         upload_to="albums/covers/%Y/%m/%d", null=True, blank=True

api/requirements/base.txt

@@ -35,7 +35,7 @@ djangorestframework>=3.7,<3.8
 djangorestframework-jwt>=1.11,<1.12
 oauth2client<4
 google-api-python-client>=1.6,<1.7
-arrow>=0.12,<0.13
+pendulum>=2,<3
 persisting-theory>=0.2,<0.3
 django-versatileimagefield>=1.9,<1.10
 django-filter>=1.1,<1.2

api/tests/federation/test_actors.py

-import arrow
+import pendulum
 import pytest
 from django.urls import reverse
 from django.utils import timezone
@@ -455,7 +455,7 @@ def test_library_actor_handle_create_audio(mocker, factories):
         assert lt.title == a["metadata"]["recording"]["title"]
         assert lt.artist_name == a["metadata"]["artist"]["name"]
         assert lt.album_title == a["metadata"]["release"]["title"]
-        assert lt.published_date == arrow.get(a["published"])
+        assert lt.published_date == pendulum.parse(a["published"])
 
 
 def test_library_actor_handle_create_audio_autoimport(mocker, factories):
@@ -494,7 +494,7 @@ def test_library_actor_handle_create_audio_autoimport(mocker, factories):
         assert lt.title == a["metadata"]["recording"]["title"]
         assert lt.artist_name == a["metadata"]["artist"]["name"]
         assert lt.album_title == a["metadata"]["release"]["title"]
-        assert lt.published_date == arrow.get(a["published"])
+        assert lt.published_date == pendulum.parse(a["published"])
 
     batch = music_models.ImportBatch.objects.latest("id")
 

api/tests/federation/test_serializers.py

-import arrow
+import pendulum
 import pytest
 from django.core.paginator import Paginator
 
@@ -492,7 +492,7 @@ def test_activity_pub_audio_serializer_to_library_track(factories):
     assert lt.title == audio["metadata"]["recording"]["title"]
     assert lt.artist_name == audio["metadata"]["artist"]["name"]
     assert lt.album_title == audio["metadata"]["release"]["title"]
-    assert lt.published_date == arrow.get(audio["published"])
+    assert lt.published_date == pendulum.parse(audio["published"])
 
 
 def test_activity_pub_audio_serializer_to_library_track_no_duplicate(factories):

api/tests/federation/test_views.py

@@ -12,6 +12,7 @@ from funkwhale_api.federation import (
     views,
     webfinger,
 )
+from funkwhale_api.music import tasks as music_tasks
 
 
 @pytest.mark.parametrize(
@@ -398,7 +399,7 @@ def test_library_track_action_import(factories, superuser_api_client, mocker):
     lt2 = factories["federation.LibraryTrack"](library=lt1.library)
     lt3 = factories["federation.LibraryTrack"]()
     factories["federation.LibraryTrack"](library=lt3.library)
-    mocked_run = mocker.patch("funkwhale_api.music.tasks.import_batch_run.delay")
+    mocked_run = mocker.patch("funkwhale_api.common.utils.on_commit")
 
     payload = {
         "objects": "all",
@@ -416,7 +417,9 @@ def test_library_track_action_import(factories, superuser_api_client, mocker):
     assert batch.jobs.count() == 2
     for i, job in enumerate(batch.jobs.all()):
         assert job.library_track == imported_lts[i]
-    mocked_run.assert_called_once_with(import_batch_id=batch.pk)
+    mocked_run.assert_called_once_with(
+        music_tasks.import_batch_run.delay, import_batch_id=batch.pk
+    )
 
 
 def test_local_actor_detail(factories, api_client):

api/tests/music/test_metadata.py

@@ -122,3 +122,11 @@ def test_mbid_clean_keeps_only_first(field_name):
     result = field.to_python("/".join([u1, u2]))
 
     assert str(result) == u1
+
+
+@pytest.mark.parametrize(
+    "raw,expected",
+    [("2017", datetime.date(2017, 1, 1)), ("2017-12-31", datetime.date(2017, 12, 31))],
+)
+def test_date_parsing(raw, expected):
+    assert metadata.get_date(raw) == expected

deploy/apache.conf

@@ -111,7 +111,7 @@ Define MUSIC_DIRECTORY_PATH /srv/funkwhale/data/music
       Require all granted
    </Directory>
 
-   <Directory /srv/funkwhale/data/media/albums>
+   <Directory /srv/funkwhale/data/media>
       Options FollowSymLinks
       AllowOverride None
       Require all granted

dev.yml

-version: '3'
+version: "3"
 
 services:
   front:
@@ -8,22 +8,22 @@ services:
       - .env
     environment:
       - "HOST=0.0.0.0"
-      - "WEBPACK_DEVSERVER_PORT=${WEBPACK_DEVSERVER_PORT-8080}"
+      - "VUE_PORT=${VUE_PORT-8080}"
     ports:
-      - "${WEBPACK_DEVSERVER_PORT_BINDING-8080:}${WEBPACK_DEVSERVER_PORT-8080}"
+      - "${VUE_PORT_BINDING-8080:}${VUE_PORT-8080}"
     volumes:
-      - './front:/app'
-      - '/app/node_modules'
-      - './po:/po'
+      - "./front:/app"
+      - "/app/node_modules"
+      - "./po:/po"
     networks:
       - federation
       - internal
     labels:
       traefik.backend: "${COMPOSE_PROJECT_NAME-node1}"
       traefik.frontend.rule: "Host:${COMPOSE_PROJECT_NAME-node1}.funkwhale.test,${NODE_IP-127.0.0.1}"
-      traefik.enable: 'true'
-      traefik.federation.protocol: 'http'
-      traefik.federation.port: "${WEBPACK_DEVSERVER_PORT-8080}"
+      traefik.enable: "true"
+      traefik.federation.protocol: "http"
+      traefik.federation.port: "${VUE_PORT-8080}"
 
   postgres:
     env_file:
@@ -53,9 +53,9 @@ services:
       context: ./api
       dockerfile: docker/Dockerfile.test
     links:
-     - postgres
-     - redis
-    command: celery -A funkwhale_api.taskapp worker -l debug
+      - postgres
+      - redis
+    command: celery -A funkwhale_api.taskapp worker -l debug -B
     environment:
       - "FUNKWHALE_HOSTNAME=${FUNKWHALE_HOSTNAME-localhost}"
       - "FUNKWHALE_HOSTNAME_SUFFIX=funkwhale.test"
@@ -98,7 +98,7 @@ services:
       - .env
     image: nginx
     environment:
-      - "WEBPACK_DEVSERVER_PORT=${WEBPACK_DEVSERVER_PORT-8080}"
+      - "VUE_PORT=${VUE_PORT-8080}"
       - "COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME- }"
       - "FUNKWHALE_HOSTNAME=${FUNKWHALE_HOSTNAME-localhost}"
     links:
@@ -120,20 +120,20 @@ services:
     volumes:
       - ".:/app/"
     ports:
-      - '35730:35730'
-      - '8001:8001'
+      - "35730:35730"
+      - "8001:8001"
 
   api-docs:
     image: swaggerapi/swagger-ui
     environment:
       - "API_URL=/swagger.yml"
     ports:
-      - '8002:8080'
+      - "8002:8080"
     volumes:
       - "./docs/swagger.yml:/usr/share/nginx/html/swagger.yml"
 
 networks:
-  internal:
+  ? internal
   federation:
     external:
       name: federation

docker/nginx/entrypoint.sh

 #!/bin/bash -eux
 
-FORWARDED_PORT="$WEBPACK_DEVSERVER_PORT"
+FORWARDED_PORT="$VUE_PORT"
 COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME// /}"
 if [ -n "$COMPOSE_PROJECT_NAME" ]; then
   echo

docs/configuration.rst

@@ -158,3 +158,79 @@ permissions are:
 There is no dedicated interface to manage users permissions, but superusers
 can login on the Django's admin at ``/api/admin/`` and grant permissions
 to users at ``/api/admin/users/user/``.
+
+Theming
+-------
+
+Funkwhale supports custom themes, which are great if you want to personnalize the
+look and feel of your instance. Theming is achieved by declaring
+additionnal stylesheets you want to load in the front-end.
+
+Customize the settings
+^^^^^^^^^^^^^^^^^^^^^^
+
+In order to know what stylesheets to load, the front-end requests the following
+url: ``https://your.instance/settings.json``. On typical deployments, this url
+returns a 404 error, which is simply ignored.
+
+However, if you return the appropriate payload on this url, you can make the magic
+work. We will store the necessary files in the ``/srv/funkwhale/custom`` directory:
+
+.. code-block:: shell
+
+    cd /srv/funkwhale/
+    mkdir custom
+    cat <<EOF > custom/settings.json
+    {
+      "additionalStylesheets": ["/custom/custom.css"]
+    }
+    EOF
+    cat <<EOF > custom/custom.css
+    body {
+      background-color: red;
+    }
+    EOF
+
+By executing the previous commands, you will end up with two files in your ``/srv/funkwhale/custom``
+directory:
+
+- ``settings.json`` will tell the front-end what stylesheets you want to load (``/custom/custom.css`` in this example)
+- ``custom.css`` will hold your custom CSS
+
+The last step to make this work is to ensure both files are served by the reverse proxy.
+
+On nginx, add the following snippet to your vhost config::
+
+    location /settings.json {
+        alias /srv/funkwhale/custom/settings.json;
+    }
+    location /custom {
+        alias /srv/funkwhale/custom;
+    }
+
+On apache, use the following one::
+
+    Alias /settings.json /srv/funkwhale/custom/settings.json
+    Alias /custom /srv/funkwhale/custom
+
+    <Directory "/srv/funkwhale/custom">
+      Options FollowSymLinks
+      AllowOverride None
+      Require all granted
+    </Directory>
+
+Once done, reload your reverse proxy, refresh Funkwhale in your web browser, and you should see
+a red background.
+
+.. note::
+
+    You can reference external urls as well in ``settings.json``, simply use
+    the full urls. Be especially careful with external urls as they may affect your users
+    privacy.
+
+.. warning::
+
+    Loading additional stylesheets and CSS rules can affect the performance and
+    usability of your instance. If you encounter issues with the interfaces and use
+    custom stylesheets, try to disable those to ensure the issue is not caused
+    by your customizations.