Merge branch 'release/0.9'

.env.dev

 API_AUTHENTICATION_REQUIRED=True
 RAVEN_ENABLED=false
 RAVEN_DSN=https://44332e9fdd3d42879c7d35bf8562c6a4:0062dc16a22b41679cd5765e5342f716@sentry.eliotberriot.com/5
-DJANGO_ALLOWED_HOSTS=localhost,nginx
+DJANGO_ALLOWED_HOSTS=.funkwhale.test,localhost,nginx,0.0.0.0,127.0.0.1
 DJANGO_SETTINGS_MODULE=config.settings.local
 DJANGO_SECRET_KEY=dev
 C_FORCE_ROOT=true
-FUNKWHALE_URL=http://localhost
+FUNKWHALE_HOSTNAME=localhost
+FUNKWHALE_PROTOCOL=http
 PYTHONDONTWRITEBYTECODE=true
+WEBPACK_DEVSERVER_PORT=8080

.gitignore

@@ -35,7 +35,6 @@ htmlcov
 
 # Translations
 *.mo
-*.pot
 
 # Pycharm
 .idea
@@ -75,6 +74,7 @@ api/static
 api/.pytest_cache
 
 # Front
+front/static/translations
 front/node_modules/
 front/dist/
 front/npm-debug.log*
@@ -87,3 +87,5 @@ docs/_build
 
 data/
 .env
+
+po/*.po

.gitlab-ci.yml

@@ -68,6 +68,8 @@ build_front:
 
   script:
     - yarn install
+    - yarn run i18n-extract
+    - yarn run i18n-compile
     - yarn run build
   cache:
     key: "$CI_PROJECT_ID__front_dependencies"

CHANGELOG

@@ -3,6 +3,79 @@ Changelog
 
 .. towncrier
 
+0.9 (2018-04-17)
+----------------
+
+Features:
+
+- Add internationalization support (#5)
+- Can now follow and import music from remote libraries (#136, #137)
+
+
+Enhancements:
+
+- Added a i18n-extract yarn script to extract strings to PO files (#162)
+- User admin now includes signup and last login dates (#148)
+- We now use a proper user agent including instance version and url during
+  outgoing requests
+
+
+Federation is here!
+^^^^^^^^^^^^^^^^^^^
+
+This is for real this time, and includes:
+
+- Following other Funkwhale libraries
+- Importing tracks from remote libraries (tracks are hotlinked, and only cached for a short amount of time)
+- Searching accross federated catalogs
+
+Note that by default, federation is opt-in, on a per-instance basis:
+instances will request access to your catalog, and you can accept or refuse
+those requests. You can also revoke the access at any time.
+
+Documentation was updated with relevant instructions to use and benefit
+from this new feature: https://docs.funkwhale.audio/federation.html
+
+Preparing internationalization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Funkwhale's front-end as always been english-only, and this is a barrier
+to new users. The work make Funkwhale's interface translatable was started
+in this release by Baptiste. Although nothing is translated yet,
+this release includes behind the stage changes that will make it possible in
+the near future.
+
+Many thank to Baptiste for the hard work and for figuring out a proper solution
+to this difficult problem.
+
+Upgrade path
+^^^^^^^^^^^^
+
+In addition to the usual instructions from
+https://docs.funkwhale.audio/upgrading.html, non-docker users will have
+to setup an additional systemd unit file for recurrent tasks.
+
+This was forgotten in the deployment documentation, but recurrent tasks,
+managed by the celery beat process, will be needed more and more in subsequent
+releases. Right now, we'll be using to clear the cache for federated music files
+and keep disk usage to a minimum.
+
+In the future, they will also be needed to refetch music metadata or federated
+information periodically.
+
+Celery beat can be enabled easily::
+
+    curl -L -o "/etc/systemd/system/funkwhale-beat.service" "https://code.eliotberriot.com/funkwhale/funkwhale/raw/develop/deploy/funkwhale-beat.service"
+    # Also edit /etc/systemd/system/funkwhale.target
+    # and ensure the Wants= line contains the following:
+    # Wants=funkwhale-server.service funkwhale-worker.service funkwhale-beat.service
+    nano /etc/systemd/system/funkwhale.target
+    # reload configuration
+    systemctl daemon-reload
+
+Docker users already have celerybeat enabled.
+
+
 0.8 (2018-04-02)
 ----------------
 
@@ -71,27 +144,16 @@ and add the following snippets::
 This will ensure federation endpoints will be reachable in the future.
 You can of course skip this part if you know you will not federate your instance.
 
-A new ``FEDERATION_ENABLED`` env var have also been added to control wether
+A new ``FEDERATION_ENABLED`` env var have also been added to control whether
 federation is enabled or not on the application side. This settings defaults
-to True, which should have no consequencies at the moment, since actual
+to True, which should have no consequences at the moment, since actual
 federation is not implemented and the only available endpoints are for
 testing purposes.
 
 Add ``FEDERATION_ENABLED=false`` to your .env file to disable federation
 on the application side.
 
-The last step involves generating RSA private and public keys for signing
-your instance requests on the federation. This can be done via::
-
-    # on docker setups
-    docker-compose run --rm api python manage.py generate_keys --no-input
-
-    # on non-docker setups
-    source /srv/funkwhale/virtualenv/bin/activate
-    source /srv/funkwhale/load_env
-    python manage.py generate_keys --no-input
-
-To test and troobleshoot federation, we've added a bot account. This bot is available at @test@yourinstancedomain,
+To test and troubleshoot federation, we've added a bot account. This bot is available at @test@yourinstancedomain,
 and sending it "/ping", for example, via Mastodon, should trigger
 a response.
 

README.rst

@@ -206,3 +206,91 @@ Typical workflow for a merge request
 6. Push your branch
 7. Create your merge request
 8. Take a step back and enjoy, we're really grateful you did all of this and took the time to contribute!
+
+
+Internationalization
+--------------------
+
+When working on the front-end, any end-user string should be translated
+using either ``<i18next path="yourstring">`` or the ``$t('yourstring')``
+function.
+
+Extraction is done by calling ``yarn run i18n-extract``, which
+will pull all the strings from source files and put them in a PO file.
+
+
+Working with federation locally
+-------------------------------
+
+To achieve that, you'll need:
+
+1. to update your dns resolver to resolve all your .dev hostnames locally
+2. a reverse proxy (such as traefik) to catch those .dev requests and
+   and with https certificate
+3. two instances (or more) running locally, following the regular dev setup
+
+Resolve .dev names locally
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you use dnsmasq, this is as simple as doing::
+
+    echo "address=/test/172.17.0.1" | sudo tee /etc/dnsmasq.d/test.conf
+    sudo systemctl restart dnsmasq
+
+If you use NetworkManager with dnsmasq integration, use this instead::
+
+    echo "address=/test/172.17.0.1" | sudo tee /etc/NetworkManager/dnsmasq.d/test.conf
+    sudo systemctl restart NetworkManager
+
+Add wildcard certificate to the trusted certificates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Simply copy bundled certificates::
+
+    sudo cp docker/ssl/test.crt /usr/local/share/ca-certificates/
+    sudo update-ca-certificates
+
+This certificate is a wildcard for ``*.funkwhale.test``
+
+Run a reverse proxy for your instances
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+Create docker network
+^^^^^^^^^^^^^^^^^^^^
+
+Create the federation network::
+
+    docker network create federation
+
+Launch everything
+^^^^^^^^^^^^^^^^^
+
+Launch the traefik proxy::
+
+    docker-compose -f docker/traefik.yml up -d
+
+Then, in separate terminals, you can setup as many different instances as you
+need::
+
+    export COMPOSE_PROJECT_NAME=node2
+    docker-compose -f dev.yml run --rm api python manage.py migrate
+    docker-compose -f dev.yml run --rm api python manage.py createsuperuser
+    docker-compose -f dev.yml up nginx api front nginx api celeryworker
+
+Note that by default, if you don't export the COMPOSE_PROJECT_NAME,
+we will default to node1 as the name of your instance.
+
+Assuming your project name is ``node1``, your server will be reachable
+at ``https://node1.funkwhale.test/``. Not that you'll have to trust
+the SSL Certificate as it's self signed.
+
+When working on federation with traefik, ensure you have this in your ``env``::
+
+    # This will ensure we don't bind any port on the host, and thus enable
+    # multiple instances of funkwhale to be spawned concurrently.
+    WEBPACK_DEVSERVER_PORT_BINDING=
+    # This disable certificate verification
+    EXTERNAL_REQUESTS_VERIFY_SSL=false
+    # this ensure you don't have incorrect urls pointing to http resources
+    FUNKWHALE_PROTOCOL=https

api/config/api_urls.py

@@ -32,6 +32,10 @@ v1_patterns += [
         include(
             ('funkwhale_api.instance.urls', 'instance'),
             namespace='instance')),
+    url(r'^federation/',
+        include(
+            ('funkwhale_api.federation.api_urls', 'federation'),
+            namespace='federation')),
     url(r'^providers/',
         include(
             ('funkwhale_api.providers.urls', 'providers'),

api/config/settings/common.py

@@ -13,6 +13,8 @@ from __future__ import absolute_import, unicode_literals
 from urllib.parse import urlsplit
 import os
 import environ
+from celery.schedules import crontab
+
 from funkwhale_api import __version__
 
 ROOT_DIR = environ.Path(__file__) - 3  # (/a/b/myfile.py - 3 = /)
@@ -25,12 +27,35 @@ try:
 except FileNotFoundError:
     pass
 
-FUNKWHALE_URL = env('FUNKWHALE_URL')
-FUNKWHALE_HOSTNAME = urlsplit(FUNKWHALE_URL).netloc
+FUNKWHALE_HOSTNAME = None
+FUNKWHALE_HOSTNAME_SUFFIX = env('FUNKWHALE_HOSTNAME_SUFFIX', default=None)
+FUNKWHALE_HOSTNAME_PREFIX = env('FUNKWHALE_HOSTNAME_PREFIX', default=None)
+if FUNKWHALE_HOSTNAME_PREFIX and FUNKWHALE_HOSTNAME_SUFFIX:
+    # We're in traefik case, in development
+    FUNKWHALE_HOSTNAME = '{}.{}'.format(
+        FUNKWHALE_HOSTNAME_PREFIX, FUNKWHALE_HOSTNAME_SUFFIX)
+    FUNKWHALE_PROTOCOL = env('FUNKWHALE_PROTOCOL', default='https')
+else:
+    try:
+        FUNKWHALE_HOSTNAME = env('FUNKWHALE_HOSTNAME')
+        FUNKWHALE_PROTOCOL = env('FUNKWHALE_PROTOCOL', default='https')
+    except Exception:
+        FUNKWHALE_URL = env('FUNKWHALE_URL')
+        _parsed = urlsplit(FUNKWHALE_URL)
+        FUNKWHALE_HOSTNAME = _parsed.netloc
+        FUNKWHALE_PROTOCOL = _parsed.scheme
+
+FUNKWHALE_URL = '{}://{}'.format(FUNKWHALE_PROTOCOL, FUNKWHALE_HOSTNAME)
+
 
 FEDERATION_ENABLED = env.bool('FEDERATION_ENABLED', default=True)
 FEDERATION_HOSTNAME = env('FEDERATION_HOSTNAME', default=FUNKWHALE_HOSTNAME)
-
+FEDERATION_COLLECTION_PAGE_SIZE = env.int(
+    'FEDERATION_COLLECTION_PAGE_SIZE', default=50
+)
+FEDERATION_MUSIC_NEEDS_APPROVAL = env.bool(
+    'FEDERATION_MUSIC_NEEDS_APPROVAL', default=True
+)
 ALLOWED_HOSTS = env.list('DJANGO_ALLOWED_HOSTS')
 
 # APP CONFIGURATION
@@ -144,16 +169,6 @@ FIXTURE_DIRS = (
 # ------------------------------------------------------------------------------
 EMAIL_BACKEND = env('DJANGO_EMAIL_BACKEND', default='django.core.mail.backends.smtp.EmailBackend')
 
-# MANAGER CONFIGURATION
-# ------------------------------------------------------------------------------
-# See: https://docs.djangoproject.com/en/dev/ref/settings/#admins
-ADMINS = (
-    ("""Eliot Berriot""", 'contact@eliotberriot.om'),
-)
-
-# See: https://docs.djangoproject.com/en/dev/ref/settings/#managers
-MANAGERS = ADMINS
-
 # DATABASE CONFIGURATION
 # ------------------------------------------------------------------------------
 # See: https://docs.djangoproject.com/en/dev/ref/settings/#databases
@@ -321,6 +336,16 @@ CELERY_BROKER_URL = env(
 # Your common stuff: Below this line define 3rd party library settings
 CELERY_TASK_DEFAULT_RATE_LIMIT = 1
 CELERY_TASK_TIME_LIMIT = 300
+CELERYBEAT_SCHEDULE = {
+    'federation.clean_music_cache': {
+        'task': 'funkwhale_api.federation.tasks.clean_music_cache',
+        'schedule': crontab(hour='*/2'),
+        'options': {
+            'expires': 60 * 2,
+        },
+    }
+}
+
 import datetime
 JWT_AUTH = {
     'JWT_ALLOW_REFRESH': True,
@@ -411,3 +436,8 @@ ACCOUNT_USERNAME_BLACKLIST = [
     'staff',
     'service',
 ] + env.list('ACCOUNT_USERNAME_BLACKLIST', default=[])
+
+EXTERNAL_REQUESTS_VERIFY_SSL = env.bool(
+    'EXTERNAL_REQUESTS_VERIFY_SSL',
+    default=True
+)

api/funkwhale_api/__init__.py

 # -*- coding: utf-8 -*-
-__version__ = '0.7'
+__version__ = '0.9'
 __version_info__ = tuple([int(num) if num.isdigit() else num for num in __version__.replace('-', '.', 1).split('.')])

api/funkwhale_api/common/fields.py

+import django_filters
+
 from django.db import models
 
+from funkwhale_api.music import utils
+
 
 PRIVACY_LEVEL_CHOICES = [
     ('me', 'Only me'),
@@ -25,3 +29,15 @@ def privacy_level_query(user, lookup_field='privacy_level'):
             'followers', 'instance', 'everyone'
         ]
     })
+
+
+class SearchFilter(django_filters.CharFilter):
+    def __init__(self, *args, **kwargs):
+        self.search_fields = kwargs.pop('search_fields')
+        super().__init__(*args, **kwargs)
+
+    def filter(self, qs, value):
+        if not value:
+            return qs
+        query = utils.get_query(value, self.search_fields)
+        return qs.filter(query)

api/funkwhale_api/common/session.py

+import requests
+
+from django.conf import settings
+
+import funkwhale_api
+
+
+def get_user_agent():
+    return 'python-requests (funkwhale/{}; +{})'.format(
+        funkwhale_api.__version__,
+        settings.FUNKWHALE_URL
+    )
+
+
+def get_session():
+    s = requests.Session()
+    s.headers['User-Agent'] = get_user_agent()
+    return s

api/funkwhale_api/common/utils.py

+from urllib.parse import urlencode, parse_qs, urlsplit, urlunsplit
 import os
 import shutil
 
@@ -25,3 +26,20 @@ def on_commit(f, *args, **kwargs):
     return transaction.on_commit(
         lambda: f(*args, **kwargs)
     )
+
+
+def set_query_parameter(url, **kwargs):
+    """Given a URL, set or replace a query parameter and return the
+    modified URL.
+
+    >>> set_query_parameter('http://example.com?foo=bar&biz=baz', 'foo', 'stuff')
+    'http://example.com?foo=stuff&biz=baz'
+    """
+    scheme, netloc, path, query_string, fragment = urlsplit(url)
+    query_params = parse_qs(query_string)
+
+    for param_name, param_value in kwargs.items():
+        query_params[param_name] = [param_value]
+    new_query_string = urlencode(query_params, doseq=True)
+
+    return urlunsplit((scheme, netloc, path, new_query_string, fragment))

api/funkwhale_api/downloader/downloader.py

 import os
-import requests
 import json
 from urllib.parse import quote_plus
 import youtube_dl

api/funkwhale_api/federation/activity.py

-import logging
-import json
-import requests
-import requests_http_signature
-
-from . import signing
-
-logger = logging.getLogger(__name__)
+from . import serializers
+from . import tasks
 
 ACTIVITY_TYPES = [
     'Accept',
@@ -42,44 +36,32 @@ ACTIVITY_TYPES = [
 OBJECT_TYPES = [
     'Article',
     'Audio',
+    'Collection',
     'Document',
     'Event',
     'Image',
     'Note',
+    'OrderedCollection',
     'Page',
     'Place',
     'Profile',
     'Relationship',
     'Tombstone',
     'Video',
-]
+] + ACTIVITY_TYPES
+
 
 def deliver(activity, on_behalf_of, to=[]):
-    from . import actors
-    logger.info('Preparing activity delivery to %s', to)
-    auth = requests_http_signature.HTTPSignatureAuth(
-        use_auth_header=False,
-        headers=[
-            '(request-target)',
-            'user-agent',
-            'host',
-            'date',
-            'content-type',],
-        algorithm='rsa-sha256',
-        key=on_behalf_of.private_key.encode('utf-8'),
-        key_id=on_behalf_of.private_key_id,
+    return tasks.send.delay(
+        activity=activity,
+        actor_id=on_behalf_of.pk,
+        to=to
     )
-    for url in to:
-        recipient_actor = actors.get_actor(url)
-        logger.debug('delivering to %s', recipient_actor.inbox_url)
-        logger.debug('activity content: %s', json.dumps(activity))
-        response = requests.post(
-            auth=auth,
-            json=activity,
-            url=recipient_actor.inbox_url,
-            headers={
-                'Content-Type': 'application/activity+json'
-            }
-        )
-        response.raise_for_status()
-        logger.debug('Remote answered with %s', response.status_code)
+
+
+def accept_follow(follow):
+    serializer = serializers.AcceptFollowSerializer(follow)
+    return deliver(
+        serializer.data,
+        to=[follow.actor.url],
+        on_behalf_of=follow.target)

api/funkwhale_api/federation/actors.py

 import logging
-import requests
+import uuid
 import xml
 
 from django.conf import settings
+from django.db import transaction
 from django.urls import reverse
 from django.utils import timezone
 
@@ -10,9 +11,16 @@ from rest_framework.exceptions import PermissionDenied
 
 from dynamic_preferences.registries import global_preferences_registry
 
+from funkwhale_api.common import session
+from funkwhale_api.common import utils as funkwhale_utils
+from funkwhale_api.music import models as music_models
+from funkwhale_api.music import tasks as music_tasks
+
 from . import activity
+from . import keys
 from . import models
 from . import serializers
+from . import signing
 from . import utils
 
 logger = logging.getLogger(__name__)
@@ -24,8 +32,10 @@ def remove_tags(text):
 
 
 def get_actor_data(actor_url):
-    response = requests.get(
+    response = session.get_session().get(
         actor_url,
+        timeout=5,
+        verify=settings.EXTERNAL_REQUESTS_VERIFY_SSL,
         headers={
             'Accept': 'application/activity+json',
         }
@@ -37,6 +47,7 @@ def get_actor_data(actor_url):
         raise ValueError(
             'Invalid actor payload: {}'.format(response.text))
 
+
 def get_actor(actor_url):
     data = get_actor_data(actor_url)
     serializer = serializers.ActorSerializer(data=data)
@@ -47,31 +58,48 @@ def get_actor(actor_url):
 
 class SystemActor(object):
     additional_attributes = {}
+    manually_approves_followers = False
+
+    def get_request_auth(self):
+        actor = self.get_actor_instance()
+        return signing.get_auth(
+            actor.private_key, actor.private_key_id)
+
+    def serialize(self):
+        actor = self.get_actor_instance()
+        serializer = serializers.ActorSerializer(actor)
+        return serializer.data
 
     def get_actor_instance(self):
-        a = models.Actor(
-            **self.get_instance_argument(
-                self.id,
-                name=self.name,
-                summary=self.summary,
-                **self.additional_attributes
-            )
+        try:
+            return models.Actor.objects.get(url=self.get_actor_url())
+        except models.Actor.DoesNotExist:
+            pass
+        private, public = keys.get_key_pair()
+        args = self.get_instance_argument(
+            self.id,
+            name=self.name,
+            summary=self.summary,
+            **self.additional_attributes
         )
-        a.pk = self.id
-        return a
+        args['private_key'] = private.decode('utf-8')
+        args['public_key'] = public.decode('utf-8')
+        return models.Actor.objects.create(**args)
+
+    def get_actor_url(self):
+        return utils.full_url(
+            reverse(
+                'federation:instance-actors-detail',
+                kwargs={'actor': self.id}))
 
     def get_instance_argument(self, id, name, summary, **kwargs):
-        preferences = global_preferences_registry.manager()
         p = {
             'preferred_username': id,
             'domain': settings.FEDERATION_HOSTNAME,
             'type': 'Person',
             'name': name.format(host=settings.FEDERATION_HOSTNAME),
             'manually_approves_followers': True,
-            'url': utils.full_url(
-                reverse(
-                    'federation:instance-actors-detail',
-                    kwargs={'actor': id})),
+            'url': self.get_actor_url(),
             'shared_inbox_url': utils.full_url(
                 reverse(
                     'federation:instance-actors-inbox',
@@ -84,8 +112,6 @@ class SystemActor(object):
                 reverse(
                     'federation:instance-actors-outbox',
                     kwargs={'actor': id})),
-            'public_key': preferences['federation__public_key'],
-            'private_key': preferences['federation__private_key'],
             'summary': summary.format(host=settings.FEDERATION_HOSTNAME)
         }
         p.update(kwargs)
@@ -95,7 +121,7 @@ class SystemActor(object):
         raise NotImplementedError
 
     def post_inbox(self, data, actor=None):
-        raise NotImplementedError
+        return self.handle(data, actor=actor)
 
     def get_outbox(self, data, actor=None):
         raise NotImplementedError
@@ -103,6 +129,70 @@ class SystemActor(object):
     def post_outbox(self, data, actor=None):
         raise NotImplementedError
 
+    def handle(self, data, actor=None):
+        """
+        Main entrypoint for handling activities posted to the
+        actor's inbox
+        """
+        logger.info('Received activity on %s inbox', self.id)
+
+        if actor is None:
+            raise PermissionDenied('Actor not authenticated')
+
+        serializer = serializers.ActivitySerializer(
+            data=data, context={'actor': actor})
+        serializer.is_valid(raise_exception=True)
+
+        ac = serializer.data
+        try:
+            handler = getattr(
+                self, 'handle_{}'.format(ac['type'].lower()))
+        except (KeyError, AttributeError):
+            logger.debug(
+                'No handler for activity %s', ac['type'])
+            return
+
+        return handler(data, actor)
+
+    def handle_follow(self, ac, sender):
+        system_actor = self.get_actor_instance()
+        serializer = serializers.FollowSerializer(
+            data=ac, context={'follow_actor': sender})
+        if not serializer.is_valid():
+            return logger.info('Invalid follow payload')
+        approved = True if not self.manually_approves_followers else None
+        follow = serializer.save(approved=approved)
+        if follow.approved:
+            return activity.accept_follow(follow)
+
+    def handle_accept(self, ac, sender):
+        system_actor = self.get_actor_instance()
+        serializer = serializers.AcceptFollowSerializer(
+            data=ac,
+            context={'follow_target': sender, 'follow_actor': system_actor})
+        if not serializer.is_valid(raise_exception=True):
+            return logger.info('Received invalid payload')
+
+        return serializer.save()
+
+    def handle_undo_follow(self, ac, sender):
+        system_actor = self.get_actor_instance()
+        serializer = serializers.UndoFollowSerializer(
+            data=ac, context={'actor': sender, 'target': system_actor})
+        if not serializer.is_valid():
+            return logger.info('Received invalid payload')
+        serializer.save()
+
+    def handle_undo(self, ac, sender):
+        if ac['object']['type'] != 'Follow':
+            return
+
+        if ac['object']['actor'] != sender.url:
+            # not the same actor, permission issue
+            return
+
+        self.handle_undo_follow(ac, sender)
+
 
 class LibraryActor(SystemActor):
     id = 'library'
@@ -112,6 +202,84 @@ class LibraryActor(SystemActor):
         'manually_approves_followers': True
     }
 
+    def serialize(self):
+        data = super().serialize()
+        urls = data.setdefault('url', [])
+        urls.append({
+            'type': 'Link',
+            'mediaType': 'application/activity+json',
+            'name': 'library',
+            'href': utils.full_url(reverse('federation:music:files-list'))
+        })
+        return data
+
+    @property
+    def manually_approves_followers(self):
+        return settings.FEDERATION_MUSIC_NEEDS_APPROVAL
+
+    @transaction.atomic
+    def handle_create(self, ac, sender):
+        try:
+            remote_library = models.Library.objects.get(
+                actor=sender,
+                federation_enabled=True,
+            )
+        except models.Library.DoesNotExist:
+            logger.info(
+                'Skipping import, we\'re not following %s', sender.url)
+            return
+
+        if ac['object']['type'] != 'Collection':
+            return
+
+        if ac['object']['totalItems'] <= 0:
+            return
+
+        try:
+            items = ac['object']['items']
+        except KeyError:
+            logger.warning('No items in collection!')
+            return
+
+        item_serializers = [
+            serializers.AudioSerializer(
+                data=i, context={'library': remote_library})
+            for i in items
+        ]
+        now = timezone.now()
+        valid_serializers = []
+        for s in item_serializers:
+            if s.is_valid():
+                valid_serializers.append(s)
+            else:
+                logger.debug(
+                    'Skipping invalid item %s, %s', s.initial_data, s.errors)
+
+        lts = []
+        for s in valid_serializers:
+            lts.append(s.save())
+
+        if remote_library.autoimport:
+            batch = music_models.ImportBatch.objects.create(
+                source='federation',
+            )
+            for lt in lts:
+                if lt.creation_date < now:
+                    # track was already in the library, we do not trigger
+                    # an import
+                    continue
+                job = music_models.ImportJob.objects.create(
+                    batch=batch,
+                    library_track=lt,
+                    mbid=lt.mbid,
+                    source=lt.url,
+                )
+                funkwhale_utils.on_commit(
+                    music_tasks.import_job_run.delay,
+                    import_job_id=job.pk,
+                    use_acoustid=False,
+                )
+
 
 class TestActor(SystemActor):
     id = 'test'
@@ -123,40 +291,24 @@ class TestActor(SystemActor):
     additional_attributes = {
         'manually_approves_followers': False
     }
+    manually_approves_followers = False
 
     def get_outbox(self, data, actor=None):
         return {
-        	"@context": [
-        		"https://www.w3.org/ns/activitystreams",
-        		"https://w3id.org/security/v1",
-        		{}
-        	],
-        	"id": utils.full_url(
+            "@context": [
+                "https://www.w3.org/ns/activitystreams",
+                "https://w3id.org/security/v1",
+                {}
+            ],
+            "id": utils.full_url(
                 reverse(
                     'federation:instance-actors-outbox',
                     kwargs={'actor': self.id})),
-        	"type": "OrderedCollection",
-        	"totalItems": 0,
-        	"orderedItems": []
+            "type": "OrderedCollection",
+            "totalItems": 0,
+            "orderedItems": []
         }
 
-    def post_inbox(self, data, actor=None):
-        if actor is None:
-            raise PermissionDenied('Actor not authenticated')
-
-        serializer = serializers.ActivitySerializer(
-            data=data, context={'actor': actor})
-        serializer.is_valid(raise_exception=True)
-
-        ac = serializer.validated_data
-        logger.info('Received activity on %s inbox', self.id)
-        if ac['type'] == 'Create' and ac['object']['type'] == 'Note':
-            # we received a toot \o/
-            command = self.parse_command(ac['object']['content'])
-            logger.debug('Parsed command: %s', command)
-            if command == 'ping':
-                self.handle_ping(ac, actor)
-
     def parse_command(self, message):
         """
         Remove any links or fancy markup to extract /command from
@@ -168,7 +320,16 @@ class TestActor(SystemActor):
         except IndexError:
             return
 
-    def handle_ping(self, ac, sender):
+    def handle_create(self, ac, sender):
+        if ac['object']['type'] != 'Note':
+            return
+
+        # we received a toot \o/
+        command = self.parse_command(ac['object']['content'])
+        logger.debug('Parsed command: %s', command)
+        if command != 'ping':
+            return
+
         now = timezone.now()
         test_actor = self.get_actor_instance()
         reply_url = 'https://{}/activities/note/{}'.format(
@@ -179,10 +340,10 @@ class TestActor(SystemActor):
         )
         reply_activity = {
             "@context": [
-        		"https://www.w3.org/ns/activitystreams",
-        		"https://w3id.org/security/v1",
-        		{}
-        	],
+                "https://www.w3.org/ns/activitystreams",
+                "https://w3id.org/security/v1",
+                {}
+            ],
             'type': 'Create',
             'actor': test_actor.url,
             'id': '{}/activity'.format(reply_url),
@@ -214,6 +375,39 @@ class TestActor(SystemActor):
             to=[ac['actor']],
             on_behalf_of=test_actor)
 
+    def handle_follow(self, ac, sender):
+        super().handle_follow(ac, sender)
+        # also, we follow back
+        test_actor = self.get_actor_instance()
+        follow_back = models.Follow.objects.get_or_create(
+            actor=test_actor,
+            target=sender,
+            approved=None,
+        )[0]
+        activity.deliver(
+            serializers.FollowSerializer(follow_back).data,
+            to=[follow_back.target.url],
+            on_behalf_of=follow_back.actor)
+
+    def handle_undo_follow(self, ac, sender):
+        super().handle_undo_follow(ac, sender)
+        actor = self.get_actor_instance()
+        # we also unfollow the sender, if possible
+        try:
+            follow = models.Follow.objects.get(
+                target=sender,
+                actor=actor,
+            )
+        except models.Follow.DoesNotExist:
+            return
+        undo = serializers.UndoFollowSerializer(follow).data
+        follow.delete()
+        activity.deliver(
+            undo,
+            to=[sender.url],
+            on_behalf_of=actor)
+
+
 SYSTEM_ACTORS = {
     'library': LibraryActor(),
     'test': TestActor(),

api/funkwhale_api/federation/admin.py

+from django.contrib import admin
+
+from . import models
+
+
+@admin.register(models.Actor)
+class ActorAdmin(admin.ModelAdmin):
+    list_display = [
+        'url',
+        'domain',
+        'preferred_username',
+        'type',
+        'creation_date',
+        'last_fetch_date']
+    search_fields = ['url', 'domain', 'preferred_username']
+    list_filter = [
+        'type'
+    ]
+
+
+@admin.register(models.Follow)
+class FollowAdmin(admin.ModelAdmin):
+    list_display = [
+        'actor',
+        'target',
+        'approved',
+        'creation_date'
+    ]
+    list_filter = [
+        'approved'
+    ]
+    search_fields = ['actor__url', 'target__url']
+    list_select_related = True
+
+
+@admin.register(models.Library)
+class LibraryAdmin(admin.ModelAdmin):
+    list_display = [
+        'actor',
+        'url',
+        'creation_date',
+        'fetched_date',
+        'tracks_count']
+    search_fields = ['actor__url', 'url']
+    list_filter = [
+        'federation_enabled',
+        'download_files',
+        'autoimport',
+    ]
+    list_select_related = True
+
+
+@admin.register(models.LibraryTrack)
+class LibraryTrackAdmin(admin.ModelAdmin):
+    list_display = [
+        'title',
+        'artist_name',
+        'album_title',
+        'url',
+        'library',
+        'creation_date',
+        'published_date',
+    ]
+    search_fields = [
+        'library__url', 'url', 'artist_name', 'title', 'album_title']
+    list_select_related = True

api/funkwhale_api/federation/api_urls.py

+from rest_framework import routers
+
+from . import views
+
+router = routers.SimpleRouter()
+router.register(
+    r'libraries',
+    views.LibraryViewSet,
+    'libraries')
+router.register(
+    r'library-tracks',
+    views.LibraryTrackViewSet,
+    'library-tracks')
+
+urlpatterns = router.urls

api/funkwhale_api/federation/authentication.py

@@ -7,6 +7,7 @@ from rest_framework import exceptions
 
 from . import actors
 from . import keys
+from . import models
 from . import serializers
 from . import signing
 from . import utils
@@ -42,11 +43,16 @@ class SignatureAuthentication(authentication.BaseAuthentication):
         except cryptography.exceptions.InvalidSignature:
             raise exceptions.AuthenticationFailed('Invalid signature')
 
-        return serializer.build()
+        try:
+            return models.Actor.objects.get(url=actor_data['id'])
+        except models.Actor.DoesNotExist:
+            return serializer.save()
 
     def authenticate(self, request):
         setattr(request, 'actor', None)
         actor = self.authenticate_actor(request)
+        if not actor:
+            return
         user = AnonymousUser()
         setattr(request, 'actor', actor)
         return (user, None)

api/funkwhale_api/federation/dynamic_preferences_registry.py

@@ -7,28 +7,14 @@ federation = types.Section('federation')
 
 
 @global_preferences_registry.register
-class FederationPrivateKey(types.StringPreference):
-    show_in_api = False
+class MusicCacheDuration(types.IntPreference):
+    show_in_api = True
     section = federation
-    name = 'private_key'
-    default = ''
+    name = 'music_cache_duration'
+    default = 60 * 24 * 2
+    verbose_name = 'Music cache duration'
     help_text = (
-        'Instance private key, used for signing federation HTTP requests'
-    )
-    verbose_name = (
-        'Instance private key (keep it secret, do not change it)'
-    )
-
-
-@global_preferences_registry.register
-class FederationPublicKey(types.StringPreference):
-    show_in_api = False
-    section = federation
-    name = 'public_key'
-    default = ''
-    help_text = (
-        'Instance public key, used for signing federation HTTP requests'
-    )
-    verbose_name = (
-        'Instance public key (do not change it)'
+        'How much minutes do you want to keep a copy of federated tracks'
+        'locally? Federated files that were not listened in this interval '
+        'will be erased and refetched from the remote on the next listening.'
     )

api/funkwhale_api/federation/factories.py

 import factory
 import requests
 import requests_http_signature
+import uuid
 
 from django.utils import timezone
+from django.conf import settings
 
 from funkwhale_api.factories import registry
 
@@ -51,9 +53,23 @@ class SignedRequestFactory(factory.Factory):
         self.headers.update(default_headers)
 
 
+@registry.register(name='federation.Link')
+class LinkFactory(factory.Factory):
+    type = 'Link'
+    href = factory.Faker('url')
+    mediaType = 'text/html'
+
+    class Meta:
+        model = dict
+
+    class Params:
+        audio = factory.Trait(
+            mediaType=factory.Iterator(['audio/mp3', 'audio/ogg'])
+        )
+
+
 @registry.register
 class ActorFactory(factory.DjangoModelFactory):
-
     public_key = None
     private_key = None
     preferred_username = factory.Faker('user_name')
@@ -66,6 +82,12 @@ class ActorFactory(factory.DjangoModelFactory):
     class Meta:
         model = models.Actor
 
+    class Params:
+        local = factory.Trait(
+            domain=factory.LazyAttribute(
+                lambda o: settings.FEDERATION_HOSTNAME)
+        )
+
     @classmethod
     def _generate(cls, create, attrs):
         has_public = attrs.get('public_key') is not None
@@ -77,6 +99,98 @@ class ActorFactory(factory.DjangoModelFactory):
         return super()._generate(create, attrs)
 
 
+@registry.register
+class FollowFactory(factory.DjangoModelFactory):
+    target = factory.SubFactory(ActorFactory)
+    actor = factory.SubFactory(ActorFactory)
+
+    class Meta:
+        model = models.Follow
+
+    class Params:
+        local = factory.Trait(
+            actor=factory.SubFactory(ActorFactory, local=True)
+        )
+
+
+@registry.register
+class LibraryFactory(factory.DjangoModelFactory):
+    actor = factory.SubFactory(ActorFactory)
+    url = factory.Faker('url')
+    federation_enabled = True
+    download_files = False
+    autoimport = False
+
+    class Meta:
+        model = models.Library
+
+
+class ArtistMetadataFactory(factory.Factory):
+    name = factory.Faker('name')
+
+    class Meta:
+        model = dict
+
+    class Params:
+        musicbrainz = factory.Trait(
+            musicbrainz_id=factory.Faker('uuid4')
+        )
+
+
+class ReleaseMetadataFactory(factory.Factory):
+    title = factory.Faker('sentence')
+
+    class Meta:
+        model = dict
+
+    class Params:
+        musicbrainz = factory.Trait(
+            musicbrainz_id=factory.Faker('uuid4')
+        )
+
+
+class RecordingMetadataFactory(factory.Factory):
+    title = factory.Faker('sentence')
+
+    class Meta:
+        model = dict
+
+    class Params:
+        musicbrainz = factory.Trait(
+            musicbrainz_id=factory.Faker('uuid4')
+        )
+
+
+@registry.register(name='federation.LibraryTrackMetadata')
+class LibraryTrackMetadataFactory(factory.Factory):
+    artist = factory.SubFactory(ArtistMetadataFactory)
+    recording = factory.SubFactory(RecordingMetadataFactory)
+    release = factory.SubFactory(ReleaseMetadataFactory)
+
+    class Meta:
+        model = dict
+
+
+@registry.register
+class LibraryTrackFactory(factory.DjangoModelFactory):
+    library = factory.SubFactory(LibraryFactory)
+    url = factory.Faker('url')
+    title = factory.Faker('sentence')
+    artist_name = factory.Faker('sentence')
+    album_title = factory.Faker('sentence')
+    audio_url = factory.Faker('url')
+    audio_mimetype = 'audio/ogg'
+    metadata = factory.SubFactory(LibraryTrackMetadataFactory)
+
+    class Meta:
+        model = models.LibraryTrack
+
+    class Params:
+        with_audio_file = factory.Trait(
+            audio_file=factory.django.FileField()
+        )
+
+
 @registry.register(name='federation.Note')
 class NoteFactory(factory.Factory):
     type = 'Note'
@@ -89,3 +203,51 @@ class NoteFactory(factory.Factory):
 
     class Meta:
         model = dict
+
+
+@registry.register(name='federation.Activity')
+class ActivityFactory(factory.Factory):
+    type = 'Create'
+    id = factory.Faker('url')
+    published = factory.LazyFunction(
+        lambda: timezone.now().isoformat()
+    )
+    actor = factory.Faker('url')
+    object = factory.SubFactory(
+        NoteFactory,
+        actor=factory.SelfAttribute('..actor'),
+        published=factory.SelfAttribute('..published'))
+
+    class Meta:
+        model = dict
+
+
+@registry.register(name='federation.AudioMetadata')
+class AudioMetadataFactory(factory.Factory):
+    recording = factory.LazyAttribute(
+        lambda o: 'https://musicbrainz.org/recording/{}'.format(uuid.uuid4())
+    )
+    artist = factory.LazyAttribute(
+        lambda o: 'https://musicbrainz.org/artist/{}'.format(uuid.uuid4())
+    )
+    release = factory.LazyAttribute(
+        lambda o: 'https://musicbrainz.org/release/{}'.format(uuid.uuid4())
+    )
+
+    class Meta:
+        model = dict
+
+
+@registry.register(name='federation.Audio')
+class AudioFactory(factory.Factory):
+    type = 'Audio'
+    id = factory.Faker('url')
+    published = factory.LazyFunction(
+        lambda: timezone.now().isoformat()
+    )
+    actor = factory.Faker('url')
+    url = factory.SubFactory(LinkFactory, audio=True)
+    metadata = factory.SubFactory(LibraryTrackMetadataFactory)
+
+    class Meta:
+        model = dict

api/funkwhale_api/federation/filters.py

+import django_filters
+
+from funkwhale_api.common import fields
+
+from . import models
+
+
+class LibraryFilter(django_filters.FilterSet):
+    approved = django_filters.BooleanFilter('following__approved')
+    q = fields.SearchFilter(search_fields=[
+        'actor__domain',
+    ])
+
+    class Meta:
+        model = models.Library
+        fields = {
+            'approved': ['exact'],
+            'federation_enabled': ['exact'],
+            'download_files': ['exact'],
+            'autoimport': ['exact'],
+            'tracks_count': ['exact'],
+        }
+
+
+class LibraryTrackFilter(django_filters.FilterSet):
+    library = django_filters.CharFilter('library__uuid')
+    imported = django_filters.CharFilter(method='filter_imported')
+    q = fields.SearchFilter(search_fields=[
+        'artist_name',
+        'title',
+        'album_title',
+        'library__actor__domain',
+    ])
+
+    def filter_imported(self, queryset, field_name, value):
+        if value.lower() in ['true', '1', 'yes']:
+            queryset = queryset.filter(local_track_file__isnull=False)
+        elif value.lower() in ['false', '0', 'no']:
+            queryset = queryset.filter(local_track_file__isnull=True)
+        return queryset
+
+    class Meta:
+        model = models.LibraryTrack
+        fields = {
+            'library': ['exact'],
+            'artist_name': ['exact', 'icontains'],
+            'title': ['exact', 'icontains'],
+            'album_title': ['exact', 'icontains'],
+            'audio_mimetype': ['exact', 'icontains'],
+        }
+
+
+class FollowFilter(django_filters.FilterSet):
+    pending = django_filters.CharFilter(method='filter_pending')
+    ordering = django_filters.OrderingFilter(
+        # tuple-mapping retains order
+        fields=(
+            ('creation_date', 'creation_date'),
+            ('modification_date', 'modification_date'),
+        ),
+    )
+    q = fields.SearchFilter(search_fields=[
+        'actor__domain',
+        'actor__preferred_username',
+    ])
+
+    class Meta:
+        model = models.Follow
+        fields = ['approved', 'pending', 'q']
+
+    def filter_pending(self, queryset, field_name, value):
+        if value.lower() in ['true', '1', 'yes']:
+            queryset = queryset.filter(approved__isnull=True)
+        return queryset