Merge branch 'release/0.9'

.env.dev

 API_AUTHENTICATION_REQUIRED=True
 RAVEN_ENABLED=false
 RAVEN_DSN=https://44332e9fdd3d42879c7d35bf8562c6a4:0062dc16a22b41679cd5765e5342f716@sentry.eliotberriot.com/5
-DJANGO_ALLOWED_HOSTS=localhost,nginx
+DJANGO_ALLOWED_HOSTS=.funkwhale.test,localhost,nginx,0.0.0.0,127.0.0.1
 DJANGO_SETTINGS_MODULE=config.settings.local
 DJANGO_SECRET_KEY=dev
 C_FORCE_ROOT=true
-FUNKWHALE_URL=http://localhost
+FUNKWHALE_HOSTNAME=localhost
+FUNKWHALE_PROTOCOL=http
 PYTHONDONTWRITEBYTECODE=true
+WEBPACK_DEVSERVER_PORT=8080

.gitignore

@@ -35,7 +35,6 @@ htmlcov
 
 # Translations
 *.mo
-*.pot
 
 # Pycharm
 .idea
@@ -75,6 +74,7 @@ api/static
 api/.pytest_cache
 
 # Front
+front/static/translations
 front/node_modules/
 front/dist/
 front/npm-debug.log*
@@ -87,3 +87,5 @@ docs/_build
 
 data/
 .env
+
+po/*.po

.gitlab-ci.yml

@@ -68,6 +68,8 @@ build_front:
 
   script:
     - yarn install
+    - yarn run i18n-extract
+    - yarn run i18n-compile
     - yarn run build
   cache:
     key: "$CI_PROJECT_ID__front_dependencies"

CHANGELOG

@@ -3,6 +3,79 @@ Changelog
 
 .. towncrier
 
+0.9 (2018-04-17)
+----------------
+
+Features:
+
+- Add internationalization support (#5)
+- Can now follow and import music from remote libraries (#136, #137)
+
+
+Enhancements:
+
+- Added a i18n-extract yarn script to extract strings to PO files (#162)
+- User admin now includes signup and last login dates (#148)
+- We now use a proper user agent including instance version and url during
+  outgoing requests
+
+
+Federation is here!
+^^^^^^^^^^^^^^^^^^^
+
+This is for real this time, and includes:
+
+- Following other Funkwhale libraries
+- Importing tracks from remote libraries (tracks are hotlinked, and only cached for a short amount of time)
+- Searching accross federated catalogs
+
+Note that by default, federation is opt-in, on a per-instance basis:
+instances will request access to your catalog, and you can accept or refuse
+those requests. You can also revoke the access at any time.
+
+Documentation was updated with relevant instructions to use and benefit
+from this new feature: https://docs.funkwhale.audio/federation.html
+
+Preparing internationalization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Funkwhale's front-end as always been english-only, and this is a barrier
+to new users. The work make Funkwhale's interface translatable was started
+in this release by Baptiste. Although nothing is translated yet,
+this release includes behind the stage changes that will make it possible in
+the near future.
+
+Many thank to Baptiste for the hard work and for figuring out a proper solution
+to this difficult problem.
+
+Upgrade path
+^^^^^^^^^^^^
+
+In addition to the usual instructions from
+https://docs.funkwhale.audio/upgrading.html, non-docker users will have
+to setup an additional systemd unit file for recurrent tasks.
+
+This was forgotten in the deployment documentation, but recurrent tasks,
+managed by the celery beat process, will be needed more and more in subsequent
+releases. Right now, we'll be using to clear the cache for federated music files
+and keep disk usage to a minimum.
+
+In the future, they will also be needed to refetch music metadata or federated
+information periodically.
+
+Celery beat can be enabled easily::
+
+    curl -L -o "/etc/systemd/system/funkwhale-beat.service" "https://code.eliotberriot.com/funkwhale/funkwhale/raw/develop/deploy/funkwhale-beat.service"
+    # Also edit /etc/systemd/system/funkwhale.target
+    # and ensure the Wants= line contains the following:
+    # Wants=funkwhale-server.service funkwhale-worker.service funkwhale-beat.service
+    nano /etc/systemd/system/funkwhale.target
+    # reload configuration
+    systemctl daemon-reload
+
+Docker users already have celerybeat enabled.
+
+
 0.8 (2018-04-02)
 ----------------
 
@@ -71,27 +144,16 @@ and add the following snippets::
 This will ensure federation endpoints will be reachable in the future.
 You can of course skip this part if you know you will not federate your instance.
 
-A new ``FEDERATION_ENABLED`` env var have also been added to control wether
+A new ``FEDERATION_ENABLED`` env var have also been added to control whether
 federation is enabled or not on the application side. This settings defaults
-to True, which should have no consequencies at the moment, since actual
+to True, which should have no consequences at the moment, since actual
 federation is not implemented and the only available endpoints are for
 testing purposes.
 
 Add ``FEDERATION_ENABLED=false`` to your .env file to disable federation
 on the application side.
 
-The last step involves generating RSA private and public keys for signing
-your instance requests on the federation. This can be done via::
-
-    # on docker setups
-    docker-compose run --rm api python manage.py generate_keys --no-input
-
-    # on non-docker setups
-    source /srv/funkwhale/virtualenv/bin/activate
-    source /srv/funkwhale/load_env
-    python manage.py generate_keys --no-input
-
-To test and troobleshoot federation, we've added a bot account. This bot is available at @test@yourinstancedomain,
+To test and troubleshoot federation, we've added a bot account. This bot is available at @test@yourinstancedomain,
 and sending it "/ping", for example, via Mastodon, should trigger
 a response.
 

README.rst

@@ -206,3 +206,91 @@ Typical workflow for a merge request
 6. Push your branch
 7. Create your merge request
 8. Take a step back and enjoy, we're really grateful you did all of this and took the time to contribute!
+
+
+Internationalization
+--------------------
+
+When working on the front-end, any end-user string should be translated
+using either ``<i18next path="yourstring">`` or the ``$t('yourstring')``
+function.
+
+Extraction is done by calling ``yarn run i18n-extract``, which
+will pull all the strings from source files and put them in a PO file.
+
+
+Working with federation locally
+-------------------------------
+
+To achieve that, you'll need:
+
+1. to update your dns resolver to resolve all your .dev hostnames locally
+2. a reverse proxy (such as traefik) to catch those .dev requests and
+   and with https certificate
+3. two instances (or more) running locally, following the regular dev setup
+
+Resolve .dev names locally
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you use dnsmasq, this is as simple as doing::
+
+    echo "address=/test/172.17.0.1" | sudo tee /etc/dnsmasq.d/test.conf
+    sudo systemctl restart dnsmasq
+
+If you use NetworkManager with dnsmasq integration, use this instead::
+
+    echo "address=/test/172.17.0.1" | sudo tee /etc/NetworkManager/dnsmasq.d/test.conf
+    sudo systemctl restart NetworkManager
+
+Add wildcard certificate to the trusted certificates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Simply copy bundled certificates::
+
+    sudo cp docker/ssl/test.crt /usr/local/share/ca-certificates/
+    sudo update-ca-certificates
+
+This certificate is a wildcard for ``*.funkwhale.test``
+
+Run a reverse proxy for your instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+Create docker network
+^^^^^^^^^^^^^^^^^^^^
+
+Create the federation network::
+
+    docker network create federation
+
+Launch everything
+^^^^^^^^^^^^^^^^^
+
+Launch the traefik proxy::
+
+    docker-compose -f docker/traefik.yml up -d
+
+Then, in separate terminals, you can setup as many different instances as you
+need::
+
+    export COMPOSE_PROJECT_NAME=node2
+    docker-compose -f dev.yml run --rm api python manage.py migrate
+    docker-compose -f dev.yml run --rm api python manage.py createsuperuser
+    docker-compose -f dev.yml up nginx api front nginx api celeryworker
+
+Note that by default, if you don't export the COMPOSE_PROJECT_NAME,
+we will default to node1 as the name of your instance.
+
+Assuming your project name is ``node1``, your server will be reachable
+at ``https://node1.funkwhale.test/``. Not that you'll have to trust
+the SSL Certificate as it's self signed.
+
+When working on federation with traefik, ensure you have this in your ``env``::
+
+    # This will ensure we don't bind any port on the host, and thus enable
+    # multiple instances of funkwhale to be spawned concurrently.
+    WEBPACK_DEVSERVER_PORT_BINDING=
+    # This disable certificate verification
+    EXTERNAL_REQUESTS_VERIFY_SSL=false
+    # this ensure you don't have incorrect urls pointing to http resources
+    FUNKWHALE_PROTOCOL=https

api/config/api_urls.py

@@ -32,6 +32,10 @@ v1_patterns += [
         include(
             ('funkwhale_api.instance.urls', 'instance'),
             namespace='instance')),
+    url(r'^federation/',
+        include(
+            ('funkwhale_api.federation.api_urls', 'federation'),
+            namespace='federation')),
     url(r'^providers/',
         include(
             ('funkwhale_api.providers.urls', 'providers'),

api/config/settings/common.py

@@ -13,6 +13,8 @@ from __future__ import absolute_import, unicode_literals
 from urllib.parse import urlsplit
 import os
 import environ
+from celery.schedules import crontab
+
 from funkwhale_api import __version__
 
 ROOT_DIR = environ.Path(__file__) - 3  # (/a/b/myfile.py - 3 = /)
@@ -25,12 +27,35 @@ try:
 except FileNotFoundError:
     pass
 
-FUNKWHALE_URL = env('FUNKWHALE_URL')
-FUNKWHALE_HOSTNAME = urlsplit(FUNKWHALE_URL).netloc
+FUNKWHALE_HOSTNAME = None
+FUNKWHALE_HOSTNAME_SUFFIX = env('FUNKWHALE_HOSTNAME_SUFFIX', default=None)
+FUNKWHALE_HOSTNAME_PREFIX = env('FUNKWHALE_HOSTNAME_PREFIX', default=None)
+if FUNKWHALE_HOSTNAME_PREFIX and FUNKWHALE_HOSTNAME_SUFFIX:
+    # We're in traefik case, in development
+    FUNKWHALE_HOSTNAME = '{}.{}'.format(
+        FUNKWHALE_HOSTNAME_PREFIX, FUNKWHALE_HOSTNAME_SUFFIX)
+    FUNKWHALE_PROTOCOL = env('FUNKWHALE_PROTOCOL', default='https')
+else:
+    try:
+        FUNKWHALE_HOSTNAME = env('FUNKWHALE_HOSTNAME')
+        FUNKWHALE_PROTOCOL = env('FUNKWHALE_PROTOCOL', default='https')
+    except Exception:
+        FUNKWHALE_URL = env('FUNKWHALE_URL')
+        _parsed = urlsplit(FUNKWHALE_URL)
+        FUNKWHALE_HOSTNAME = _parsed.netloc
+        FUNKWHALE_PROTOCOL = _parsed.scheme
+
+FUNKWHALE_URL = '{}://{}'.format(FUNKWHALE_PROTOCOL, FUNKWHALE_HOSTNAME)
+
 
 FEDERATION_ENABLED = env.bool('FEDERATION_ENABLED', default=True)
 FEDERATION_HOSTNAME = env('FEDERATION_HOSTNAME', default=FUNKWHALE_HOSTNAME)
-
+FEDERATION_COLLECTION_PAGE_SIZE = env.int(
+    'FEDERATION_COLLECTION_PAGE_SIZE', default=50
+)
+FEDERATION_MUSIC_NEEDS_APPROVAL = env.bool(
+    'FEDERATION_MUSIC_NEEDS_APPROVAL', default=True
+)
 ALLOWED_HOSTS = env.list('DJANGO_ALLOWED_HOSTS')
 
 # APP CONFIGURATION
@@ -144,16 +169,6 @@ FIXTURE_DIRS = (
 # ------------------------------------------------------------------------------
 EMAIL_BACKEND = env('DJANGO_EMAIL_BACKEND', default='django.core.mail.backends.smtp.EmailBackend')
 
-# MANAGER CONFIGURATION
-# ------------------------------------------------------------------------------
-# See: https://docs.djangoproject.com/en/dev/ref/settings/#admins
-ADMINS = (
-    ("""Eliot Berriot""", 'contact@eliotberriot.om'),
-)
-
-# See: https://docs.djangoproject.com/en/dev/ref/settings/#managers
-MANAGERS = ADMINS
-
 # DATABASE CONFIGURATION
 # ------------------------------------------------------------------------------
 # See: https://docs.djangoproject.com/en/dev/ref/settings/#databases
@@ -321,6 +336,16 @@ CELERY_BROKER_URL = env(
 # Your common stuff: Below this line define 3rd party library settings
 CELERY_TASK_DEFAULT_RATE_LIMIT = 1
 CELERY_TASK_TIME_LIMIT = 300
+CELERYBEAT_SCHEDULE = {
+    'federation.clean_music_cache': {
+        'task': 'funkwhale_api.federation.tasks.clean_music_cache',
+        'schedule': crontab(hour='*/2'),
+        'options': {
+            'expires': 60 * 2,
+        },
+    }
+}
+
 import datetime
 JWT_AUTH = {
     'JWT_ALLOW_REFRESH': True,
@@ -411,3 +436,8 @@ ACCOUNT_USERNAME_BLACKLIST = [
     'staff',
     'service',
 ] + env.list('ACCOUNT_USERNAME_BLACKLIST', default=[])
+
+EXTERNAL_REQUESTS_VERIFY_SSL = env.bool(
+    'EXTERNAL_REQUESTS_VERIFY_SSL',
+    default=True
+)

api/funkwhale_api/__init__.py

 # -*- coding: utf-8 -*-
-__version__ = '0.7'
+__version__ = '0.9'
 __version_info__ = tuple([int(num) if num.isdigit() else num for num in __version__.replace('-', '.', 1).split('.')])

api/funkwhale_api/common/fields.py

+import django_filters
+
 from django.db import models
 
+from funkwhale_api.music import utils
+
 
 PRIVACY_LEVEL_CHOICES = [
     ('me', 'Only me'),
@@ -25,3 +29,15 @@ def privacy_level_query(user, lookup_field='privacy_level'):
             'followers', 'instance', 'everyone'
         ]
     })
+
+
+class SearchFilter(django_filters.CharFilter):
+    def __init__(self, *args, **kwargs):
+        self.search_fields = kwargs.pop('search_fields')
+        super().__init__(*args, **kwargs)
+
+    def filter(self, qs, value):
+        if not value:
+            return qs
+        query = utils.get_query(value, self.search_fields)
+        return qs.filter(query)

api/funkwhale_api/common/session.py

+import requests
+
+from django.conf import settings
+
+import funkwhale_api
+
+
+def get_user_agent():
+    return 'python-requests (funkwhale/{}; +{})'.format(
+        funkwhale_api.__version__,
+        settings.FUNKWHALE_URL
+    )
+
+
+def get_session():
+    s = requests.Session()
+    s.headers['User-Agent'] = get_user_agent()
+    return s

api/funkwhale_api/common/utils.py

+from urllib.parse import urlencode, parse_qs, urlsplit, urlunsplit
 import os
 import shutil
 
@@ -25,3 +26,20 @@ def on_commit(f, *args, **kwargs):
     return transaction.on_commit(
         lambda: f(*args, **kwargs)
     )
+
+
+def set_query_parameter(url, **kwargs):
+    """Given a URL, set or replace a query parameter and return the
+    modified URL.
+
+    >>> set_query_parameter('http://example.com?foo=bar&biz=baz', 'foo', 'stuff')
+    'http://example.com?foo=stuff&biz=baz'
+    """
+    scheme, netloc, path, query_string, fragment = urlsplit(url)
+    query_params = parse_qs(query_string)
+
+    for param_name, param_value in kwargs.items():
+        query_params[param_name] = [param_value]
+    new_query_string = urlencode(query_params, doseq=True)
+
+    return urlunsplit((scheme, netloc, path, new_query_string, fragment))

api/funkwhale_api/downloader/downloader.py

 import os
-import requests
 import json
 from urllib.parse import quote_plus
 import youtube_dl

api/funkwhale_api/federation/activity.py

-import logging
-import json
-import requests
-import requests_http_signature
-
-from . import signing
-
-logger = logging.getLogger(__name__)
+from . import serializers
+from . import tasks
 
 ACTIVITY_TYPES = [
     'Accept',
@@ -42,44 +36,32 @@ ACTIVITY_TYPES = [
 OBJECT_TYPES = [
     'Article',
     'Audio',
+    'Collection',
     'Document',
     'Event',
     'Image',
     'Note',
+    'OrderedCollection',
     'Page',
     'Place',
     'Profile',
     'Relationship',
     'Tombstone',
     'Video',
-]
+] + ACTIVITY_TYPES
+
 
 def deliver(activity, on_behalf_of, to=[]):
-    from . import actors
-    logger.info('Preparing activity delivery to %s', to)
-    auth = requests_http_signature.HTTPSignatureAuth(
-        use_auth_header=False,
-        headers=[
-            '(request-target)',
-            'user-agent',
-            'host',
-            'date',
-            'content-type',],
-        algorithm='rsa-sha256',
-        key=on_behalf_of.private_key.encode('utf-8'),
-        key_id=on_behalf_of.private_key_id,
+    return tasks.send.delay(
+        activity=activity,
+        actor_id=on_behalf_of.pk,
+        to=to
     )
-    for url in to:
-        recipient_actor = actors.get_actor(url)
-        logger.debug('delivering to %s', recipient_actor.inbox_url)
-        logger.debug('activity content: %s', json.dumps(activity))
-        response = requests.post(
-            auth=auth,
-            json=activity,
-            url=recipient_actor.inbox_url,
-            headers={
-                'Content-Type': 'application/activity+json'
-            }
-        )
-        response.raise_for_status()
-        logger.debug('Remote answered with %s', response.status_code)
+
+
+def accept_follow(follow):
+    serializer = serializers.AcceptFollowSerializer(follow)
+    return deliver(
+        serializer.data,
+        to=[follow.actor.url],
+        on_behalf_of=follow.target)

api/funkwhale_api/federation/actors.py

 import logging
-import requests
+import uuid
 import xml
 
 from django.conf import settings
+from django.db import transaction
 from django.urls import reverse
 from django.utils import timezone
 
@@ -10,9 +11,16 @@ from rest_framework.exceptions import PermissionDenied
 
 from dynamic_preferences.registries import global_preferences_registry
 
+from funkwhale_api.common import session
+from funkwhale_api.common import utils as funkwhale_utils
+from funkwhale_api.music import models as music_models
+from funkwhale_api.music import tasks as music_tasks
+
 from . import activity
+from . import keys
 from . import models
 from . import serializers
+from . import signing
 from . import utils
 
 logger = logging.getLogger(__name__)
@@ -24,8 +32,10 @@ def remove_tags(text):
 
 
 def get_actor_data(actor_url):
-    response = requests.get(
+    response = session.get_session().get(
         actor_url,
+        timeout=5,
+        verify=settings.EXTERNAL_REQUESTS_VERIFY_SSL,
         headers={
             'Accept': 'application/activity+json',
         }
@@ -37,6 +47,7 @@ def get_actor_data(actor_url):
         raise ValueError(
             'Invalid actor payload: {}'.format(response.text))
 
+
 def get_actor(actor_url):
     data = get_actor_data(actor_url)
     serializer = serializers.ActorSerializer(data=data)
@@ -47,31 +58,48 @@ def get_actor(actor_url):
 
 class SystemActor(object):
     additional_attributes = {}
+    manually_approves_followers = False
+
+    def get_request_auth(self):
+        actor = self.get_actor_instance()
+        return signing.get_auth(
+            actor.private_key, actor.private_key_id)
+
+    def serialize(self):
+        actor = self.get_actor_instance()
+        serializer = serializers.ActorSerializer(actor)
+        return serializer.data
 
     def get_actor_instance(self):
-        a = models.Actor(
-            **self.get_instance_argument(
-                self.id,
-                name=self.name,
-                summary=self.summary,
-                **self.additional_attributes
-            )
+        try:
+            return models.Actor.objects.get(url=self.get_actor_url())
+        except models.Actor.DoesNotExist:
+            pass
+        private, public = keys.get_key_pair()
+        args = self.get_instance_argument(
+            self.id,
+            name=self.name,
+            summary=self.summary,
+            **self.additional_attributes
         )
-        a.pk = self.id
-        return a
+        args['private_key'] = private.decode('utf-8')
+        args['public_key'] = public.decode('utf-8')
+        return models.Actor.objects.create(**args)
+
+    def get_actor_url(self):
+        return utils.full_url(
+            reverse(
+                'federation:instance-actors-detail',
+                kwargs={'actor': self.id}))
 
     def get_instance_argument(self, id, name, summary, **kwargs):
-        preferences = global_preferences_registry.manager()
         p = {
             'preferred_username': id,
             'domain': settings.FEDERATION_HOSTNAME,
             'type': 'Person',
             'name': name.format(host=settings.FEDERATION_HOSTNAME),
             'manually_approves_followers': True,
-            'url': utils.full_url(
-                reverse(
-                    'federation:instance-actors-detail',
-                    kwargs={'actor': id})),
+            'url': self.get_actor_url(),
             'shared_inbox_url': utils.full_url(
                 reverse(
                     'federation:instance-actors-inbox',
@@ -84,8 +112,6 @@ class SystemActor(object):
                 reverse(
                     'federation:instance-actors-outbox',
                     kwargs={'actor': id})),
-            'public_key': preferences['federation__public_key'],
-            'private_key': preferences['federation__private_key'],
             'summary': summary.format(host=settings.FEDERATION_HOSTNAME)
         }
         p.update(kwargs)
@@ -95,7 +121,7 @@ class SystemActor(object):
         raise NotImplementedError
 
     def post_inbox(self, data, actor=None):
-        raise NotImplementedError
+        return self.handle(data, actor=actor)