...
 
Commits (112)
......@@ -93,5 +93,6 @@ po/*.po
docs/swagger
_build
front/src/translations.json
front/src/translations/*.json
front/locales/en_US/LC_MESSAGES/app.po
*.prof
......@@ -27,7 +27,7 @@ review_front:
- yarn install
- yarn run i18n-compile
# this is to ensure we don't have any errors in the output,
# cf https://code.eliotberriot.com/funkwhale/funkwhale/issues/169
# cf https://dev.funkwhale.audio/funkwhale/funkwhale/issues/169
- VUE_APP_INSTANCE_URL=$REVIEW_INSTANCE_URL yarn run build | tee /dev/stderr | (! grep -i 'ERROR in')
- mkdir -p /static/front/$CI_PROJECT_PATH_SLUG-$CI_BUILD_REF_SLUG
- cp -r dist/* /static/front/$CI_PROJECT_PATH_SLUG-$CI_BUILD_REF_SLUG
......@@ -190,7 +190,7 @@ build_front:
- yarn install
- yarn run i18n-compile
# this is to ensure we don't have any errors in the output,
# cf https://code.eliotberriot.com/funkwhale/funkwhale/issues/169
# cf https://dev.funkwhale.audio/funkwhale/funkwhale/issues/169
- yarn build | tee /dev/stderr | (! grep -i 'ERROR in')
- chmod -R 755 dist
cache:
......@@ -229,7 +229,7 @@ pages:
paths:
- public
only:
- develop@funkwhale/funkwhale
- master@funkwhale/funkwhale
tags:
- docker
......
......@@ -4,7 +4,7 @@ Changelog
You can subscribe to release announcements by:
- Following `funkwhale@mastodon.eliotberriot.com <https://mastodon.eliotberriot.com/@funkwhale>`_ on Mastodon
- Subscribing to the following Atom feed: https://code.eliotberriot.com/funkwhale/funkwhale/commits/develop?format=atom&search=Merge+tag
- Subscribing to the following Atom feed: https://dev.funkwhale.audio/funkwhale/funkwhale/commits/develop?format=atom&search=Merge+tag
This changelog is viewable on the web at https://docs.funkwhale.audio/changelog.html.
......@@ -149,8 +149,8 @@ Download the required configuration files for the nginx container:
cd /srv/funkwhale
mkdir nginx
curl -L -o nginx/funkwhale.template "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/docker.nginx.template"
curl -L -o nginx/funkwhale_proxy.conf "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/funkwhale_proxy.conf"
curl -L -o nginx/funkwhale.template "https://dev.funkwhale.audio/funkwhale/funkwhale/raw/|version|/deploy/docker.nginx.template"
curl -L -o nginx/funkwhale_proxy.conf "https://dev.funkwhale.audio/funkwhale/funkwhale/raw/|version|/deploy/funkwhale_proxy.conf"
Update the funkwhale.conf configuration of your server's reverse-proxy::
......@@ -788,7 +788,7 @@ it to work Apache2, for instance. It was also not compatible with Subsonic clien
Based on that, we're currently removing support for transcoding
**in its current state**. The work on a better designed transcoding feature
can be tracked in https://code.eliotberriot.com/funkwhale/funkwhale/issues/272.
can be tracked in https://dev.funkwhale.audio/funkwhale/funkwhale/issues/272.
You don't have to do anything on your side, but you may want to remove
the now obsolete configuration from your reverse proxy file (nginx only)::
......@@ -1562,7 +1562,7 @@ information periodically.
Celery beat can be enabled easily::
curl -L -o "/etc/systemd/system/funkwhale-beat.service" "https://code.eliotberriot.com/funkwhale/funkwhale/raw/develop/deploy/funkwhale-beat.service"
curl -L -o "/etc/systemd/system/funkwhale-beat.service" "https://dev.funkwhale.audio/funkwhale/funkwhale/raw/develop/deploy/funkwhale-beat.service"
# Also edit /etc/systemd/system/funkwhale.target
# and ensure the Wants= line contains the following:
# Wants=funkwhale-server.service funkwhale-worker.service funkwhale-beat.service
......
......@@ -17,7 +17,7 @@ A quick path to contribute on the front-end
The next sections of this document include a full installation guide to help
you setup a local, development version of Funkwhale. If you only want to fix small things
on the front-end, and don't want to manage a full development environment, there is anoter way.
on the front-end, and don't want to manage a full development environment, there is another way.
As the front-end can work with any Funkwhale server, you can work with the front-end only,
and make it talk with an existing instance (like the demo one, or you own instance, if you have one).
......@@ -31,7 +31,7 @@ Setup front-end only development environment
1. Clone the repository::
git clone ssh://git@code.eliotberriot.com:2222/funkwhale/funkwhale.git
git clone ssh://git@dev.funkwhale.audio/funkwhale/funkwhale.git
cd funkwhale
cd front
......@@ -75,9 +75,9 @@ This is already cover in the relevant documentations:
Cloning the project
^^^^^^^^^^^^^^^^^^^
Visit https://code.eliotberriot.com/funkwhale/funkwhale and clone the repository using SSH or HTTPS. Exemple using SSH::
Visit https://dev.funkwhale.audio/funkwhale/funkwhale and clone the repository using SSH or HTTPS. Example using SSH::
git clone ssh://git@code.eliotberriot.com:2222/funkwhale/funkwhale.git
git clone ssh://git@dev.funkwhale.audio/funkwhale/funkwhale.git
cd funkwhale
......@@ -90,11 +90,11 @@ Next release development occurs on the "develop" branch, and releases are made o
Working with docker
^^^^^^^^^^^^^^^^^^^
In developpement, we use the docker-compose file named ``dev.yml``, and this is why all our docker-compose commands will look like this::
In development, we use the docker-compose file named ``dev.yml``, and this is why all our docker-compose commands will look like this::
docker-compose -f dev.yml logs
If you do not want to add the ``-f dev.yml`` snippet everytime, you can run this command before starting your work::
If you do not want to add the ``-f dev.yml`` snippet every time, you can run this command before starting your work::
export COMPOSE_FILE=dev.yml
......@@ -137,7 +137,7 @@ To setup funkwhale's database schema, run this::
docker-compose -f dev.yml run --rm api python manage.py migrate
This will create all the tables needed for the API to run proprely.
This will create all the tables needed for the API to run properly.
You will also need to run this whenever changes are made on the database
schema.
......@@ -155,7 +155,7 @@ Create an admin user with the following command::
docker-compose -f dev.yml run --rm api python manage.py createsuperuser
Injecting fake data is done by running the fllowing script::
Injecting fake data is done by running the following script::
artists=25
command="from funkwhale_api.music import fake_data; fake_data.create_data($artists)"
......@@ -298,7 +298,7 @@ sure about what to do, do not panic, open your merge request normally and we'll
figure everything during the review ;)
Changelog fragments are text files that can contain one or multiple lines
that describe the changes occuring in a bunch of commits. Those files reside
that describe the changes occurring in a bunch of commits. Those files reside
in ``changes/changelog.d``.
Content
......
......@@ -6,7 +6,7 @@ you found the proper place :)
Translation is done via our own Weblate instance at https://translate.funkwhale.audio/projects/funkwhale/front/.
You can signup/login using your Gitlab account (from https://code.eliotberriot.com).
You can signup/login using your Gitlab account (from https://dev.funkwhale.audio).
Translation workflow
--------------------
......@@ -25,4 +25,4 @@ Requesting a new language
-------------------------
If you'd like to see a new language in Funkwhale, please open an issue here:
https://code.eliotberriot.com/funkwhale/funkwhale/issues
https://dev.funkwhale.audio/funkwhale/funkwhale/issues
......@@ -31,6 +31,7 @@ subsonic_router.register(r"subsonic/rest", SubsonicViewSet, base_name="subsonic"
v1_patterns += [
url(r"^oembed/$", views.OembedView.as_view(), name="oembed"),
url(
r"^instance/",
include(("funkwhale_api.instance.urls", "instance"), namespace="instance"),
......
......@@ -13,7 +13,7 @@ from __future__ import absolute_import, unicode_literals
import datetime
import logging
from urllib.parse import urlparse, urlsplit
from urllib.parse import urlsplit
import environ
from celery.schedules import crontab
......@@ -69,12 +69,23 @@ else:
FUNKWHALE_HOSTNAME = _parsed.netloc
FUNKWHALE_PROTOCOL = _parsed.scheme
FUNKWHALE_PROTOCOL = FUNKWHALE_PROTOCOL.lower()
FUNKWHALE_HOSTNAME = FUNKWHALE_HOSTNAME.lower()
FUNKWHALE_URL = "{}://{}".format(FUNKWHALE_PROTOCOL, FUNKWHALE_HOSTNAME)
FUNKWHALE_SPA_HTML_ROOT = env(
"FUNKWHALE_SPA_HTML_ROOT", default=FUNKWHALE_URL + "/front/"
)
FUNKWHALE_SPA_HTML_CACHE_DURATION = env.int(
"FUNKWHALE_SPA_HTML_CACHE_DURATION", default=60 * 15
)
FUNKWHALE_EMBED_URL = env(
"FUNKWHALE_EMBED_URL", default=FUNKWHALE_SPA_HTML_ROOT + "embed.html"
)
APP_NAME = "Funkwhale"
# XXX: deprecated, see #186
FEDERATION_ENABLED = env.bool("FEDERATION_ENABLED", default=True)
FEDERATION_HOSTNAME = env("FEDERATION_HOSTNAME", default=FUNKWHALE_HOSTNAME)
FEDERATION_HOSTNAME = env("FEDERATION_HOSTNAME", default=FUNKWHALE_HOSTNAME).lower()
# XXX: deprecated, see #186
FEDERATION_COLLECTION_PAGE_SIZE = env.int("FEDERATION_COLLECTION_PAGE_SIZE", default=50)
# XXX: deprecated, see #186
......@@ -159,7 +170,7 @@ INSTALLED_APPS = DJANGO_APPS + THIRD_PARTY_APPS + LOCAL_APPS
# MIDDLEWARE CONFIGURATION
# ------------------------------------------------------------------------------
MIDDLEWARE = (
# Make sure djangosecure.middleware.SecurityMiddleware is listed first
"funkwhale_api.common.middleware.SPAFallbackMiddleware",
"django.contrib.sessions.middleware.SessionMiddleware",
"corsheaders.middleware.CorsMiddleware",
"django.middleware.common.CommonMiddleware",
......@@ -305,6 +316,7 @@ FILE_UPLOAD_PERMISSIONS = 0o644
# URL Configuration
# ------------------------------------------------------------------------------
ROOT_URLCONF = "config.urls"
SPA_URLCONF = "config.spa_urls"
# See: https://docs.djangoproject.com/en/dev/ref/settings/#wsgi-application
WSGI_APPLICATION = "config.wsgi.application"
ASGI_APPLICATION = "config.routing.application"
......@@ -400,15 +412,20 @@ if AUTH_LDAP_ENABLED:
AUTOSLUG_SLUGIFY_FUNCTION = "slugify.slugify"
CACHE_DEFAULT = "redis://127.0.0.1:6379/0"
CACHES = {"default": env.cache_url("CACHE_URL", default=CACHE_DEFAULT)}
CACHES = {
"default": env.cache_url("CACHE_URL", default=CACHE_DEFAULT),
"local": {
"BACKEND": "django.core.cache.backends.locmem.LocMemCache",
"LOCATION": "local-cache",
},
}
CACHES["default"]["BACKEND"] = "django_redis.cache.RedisCache"
cache_url = urlparse(CACHES["default"]["LOCATION"])
CHANNEL_LAYERS = {
"default": {
"BACKEND": "channels_redis.core.RedisChannelLayer",
"CONFIG": {"hosts": [(cache_url.hostname, cache_url.port)]},
"CONFIG": {"hosts": [CACHES["default"]["LOCATION"]]},
}
}
......
from django import urls
from funkwhale_api.music import spa_views
urlpatterns = [
urls.re_path(
r"^library/tracks/(?P<pk>\d+)/?$", spa_views.library_track, name="library_track"
),
urls.re_path(
r"^library/albums/(?P<pk>\d+)/?$", spa_views.library_album, name="library_album"
),
urls.re_path(
r"^library/artists/(?P<pk>\d+)/?$",
spa_views.library_artist,
name="library_artist",
),
]
......@@ -5,7 +5,7 @@ from asgiref.sync import async_to_sync
from channels.layers import get_channel_layer
from django.core.serializers.json import DjangoJSONEncoder
logger = logging.getLogger(__file__)
logger = logging.getLogger(__name__)
channel_layer = get_channel_layer()
group_add = async_to_sync(channel_layer.group_add)
......
import html
import requests
from django import http
from django.conf import settings
from django.core.cache import caches
from django import urls
from . import preferences
from . import utils
EXCLUDED_PATHS = ["/api", "/federation", "/.well-known"]
def should_fallback_to_spa(path):
if path == "/":
return True
return not any([path.startswith(m) for m in EXCLUDED_PATHS])
def serve_spa(request):
html = get_spa_html(settings.FUNKWHALE_SPA_HTML_ROOT)
head, tail = html.split("</head>", 1)
if not preferences.get("common__api_authentication_required"):
try:
request_tags = get_request_head_tags(request) or []
except urls.exceptions.Resolver404:
# we don't have any custom tags for this route
request_tags = []
else:
# API is not open, we don't expose any custom data
request_tags = []
default_tags = get_default_head_tags(request.path)
unique_attributes = ["name", "property"]
final_tags = request_tags
skip = []
for t in final_tags:
for attr in unique_attributes:
if attr in t:
skip.append(t[attr])
for t in default_tags:
existing = False
for attr in unique_attributes:
if t.get(attr) in skip:
existing = True
break
if not existing:
final_tags.append(t)
# let's inject our meta tags in the HTML
head += "\n" + "\n".join(render_tags(final_tags)) + "\n</head>"
return http.HttpResponse(head + tail)
def get_spa_html(spa_url):
cache_key = "spa-html:{}".format(spa_url)
cached = caches["local"].get(cache_key)
if cached:
return cached
response = requests.get(
utils.join_url(spa_url, "index.html"),
verify=settings.EXTERNAL_REQUESTS_VERIFY_SSL,
)
response.raise_for_status()
content = response.text
caches["local"].set(cache_key, content, settings.FUNKWHALE_SPA_HTML_CACHE_DURATION)
return content
def get_default_head_tags(path):
instance_name = preferences.get("instance__name")
short_description = preferences.get("instance__short_description")
app_name = settings.APP_NAME
parts = [instance_name, app_name]
return [
{"tag": "meta", "property": "og:type", "content": "website"},
{
"tag": "meta",
"property": "og:site_name",
"content": " - ".join([p for p in parts if p]),
},
{"tag": "meta", "property": "og:description", "content": short_description},
{
"tag": "meta",
"property": "og:image",
"content": utils.join_url(settings.FUNKWHALE_URL, "/front/favicon.png"),
},
{
"tag": "meta",
"property": "og:url",
"content": utils.join_url(settings.FUNKWHALE_URL, path),
},
]
def render_tags(tags):
"""
Given a dict like {'tag': 'meta', 'hello': 'world'}
return a html ready tag like
<meta hello="world" />
"""
for tag in tags:
yield "<{tag} {attrs} />".format(
tag=tag.pop("tag"),
attrs=" ".join(
[
'{}="{}"'.format(a, html.escape(str(v)))
for a, v in sorted(tag.items())
if v
]
),
)
def get_request_head_tags(request):
match = urls.resolve(request.path, urlconf=settings.SPA_URLCONF)
return match.func(request, *match.args, **match.kwargs)
class SPAFallbackMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
response = self.get_response(request)
if response.status_code == 404 and should_fallback_to_spa(request.path):
return serve_spa(request)
return response
......@@ -10,7 +10,6 @@ from funkwhale_api.users import models
mapping = {
"dynamic_preferences.change_globalpreferencemodel": "settings",
"music.add_importbatch": "library",
"federation.change_library": "federation",
}
......
import collections
import io
import PIL
import os
from rest_framework import serializers
from django.core.exceptions import ObjectDoesNotExist
from django.core.files.uploadedfile import SimpleUploadedFile
from django.utils.encoding import smart_text
from django.utils.translation import ugettext_lazy as _
......@@ -190,3 +194,25 @@ def track_fields_for_update(*fields):
return serializer_class
return decorator
class StripExifImageField(serializers.ImageField):
def to_internal_value(self, data):
file_obj = super().to_internal_value(data)
image = PIL.Image.open(file_obj)
data = list(image.getdata())
image_without_exif = PIL.Image.new(image.mode, image.size)
image_without_exif.putdata(data)
with io.BytesIO() as output:
image_without_exif.save(
output,
format=PIL.Image.EXTENSION[os.path.splitext(file_obj.name)[-1]],
quality=100,
)
content = output.getvalue()
return SimpleUploadedFile(
file_obj.name, content, content_type=file_obj.content_type
)
......@@ -3,9 +3,12 @@ from django.utils.deconstruct import deconstructible
import os
import shutil
import uuid
import xml.etree.ElementTree as ET
from urllib.parse import parse_qs, urlencode, urlsplit, urlunsplit
from django.conf import settings
from django import urls
from django.db import transaction
......@@ -107,3 +110,32 @@ def chunk_queryset(source_qs, chunk_size):
if nb_items < chunk_size:
return
def join_url(start, end):
if start.endswith("/") and end.startswith("/"):
return start + end[1:]
if not start.endswith("/") and not end.startswith("/"):
return start + "/" + end
return start + end
def spa_reverse(name, args=[], kwargs={}):
return urls.reverse(name, urlconf=settings.SPA_URLCONF, args=args, kwargs=kwargs)
def spa_resolve(path):
return urls.resolve(path, urlconf=settings.SPA_URLCONF)
def parse_meta(html):
# dirty but this is only for testing so we don't really care,
# we convert the html string to xml so it can be parsed as xml
html = '<?xml version="1.0"?>' + html
tree = ET.fromstring(html)
meta = [elem for elem in tree.iter() if elem.tag in ["meta", "link"]]
return [dict([("tag", elem.tag)] + list(elem.items())) for elem in meta]
......@@ -42,23 +42,39 @@ ACTIVITY_TYPES = [
"View",
]
OBJECT_TYPES = [
"Article",
"Audio",
"Collection",
"Document",
"Event",
"Image",
"Note",
"OrderedCollection",
"Page",
"Place",
"Profile",
"Relationship",
"Tombstone",
"Video",
] + ACTIVITY_TYPES
FUNKWHALE_OBJECT_TYPES = [
("Domain", "Domain"),
("Artist", "Artist"),
("Album", "Album"),
("Track", "Track"),
("Library", "Library"),
]
OBJECT_TYPES = (
[
"Application",
"Article",
"Audio",
"Collection",
"Document",
"Event",
"Group",
"Image",
"Note",
"Object",
"OrderedCollection",
"Organization",
"Page",
"Person",
"Place",
"Profile",
"Relationship",
"Service",
"Tombstone",
"Video",
]
+ ACTIVITY_TYPES
+ FUNKWHALE_OBJECT_TYPES
)
BROADCAST_TO_USER_ACTIVITIES = ["Follow", "Accept"]
......@@ -386,15 +402,3 @@ def get_actors_from_audience(urls):
if not final_query:
return models.Actor.objects.none()
return models.Actor.objects.filter(final_query)
def get_inbox_urls(actor_queryset):
"""
Given an actor queryset, returns a deduplicated set containing
all inbox or shared inbox urls where we should deliver our payloads for
those actors
"""
values = actor_queryset.values("inbox_url", "shared_inbox_url")
urls = set([actor["shared_inbox_url"] or actor["inbox_url"] for actor in values])
return sorted(urls)
......@@ -24,6 +24,12 @@ def redeliver_activities(modeladmin, request, queryset):
redeliver_activities.short_description = "Redeliver"
@admin.register(models.Domain)
class DomainAdmin(admin.ModelAdmin):
list_display = ["name", "creation_date"]
search_fields = ["name"]
@admin.register(models.Activity)
class ActivityAdmin(admin.ModelAdmin):
list_display = ["type", "fid", "url", "actor", "creation_date"]
......
......@@ -66,24 +66,39 @@ def create_user(actor):
return user_factories.UserFactory(actor=actor)
@registry.register
class Domain(factory.django.DjangoModelFactory):
name = factory.Faker("domain_name")
class Meta:
model = "federation.Domain"
django_get_or_create = ("name",)
@registry.register
class ActorFactory(factory.DjangoModelFactory):
public_key = None
private_key = None
preferred_username = factory.Faker("user_name")
summary = factory.Faker("paragraph")
domain = factory.Faker("domain_name")
domain = factory.SubFactory(Domain)
fid = factory.LazyAttribute(
lambda o: "https://{}/users/{}".format(o.domain, o.preferred_username)
lambda o: "https://{}/users/{}".format(o.domain.name, o.preferred_username)
)
followers_url = factory.LazyAttribute(
lambda o: "https://{}/users/{}followers".format(o.domain, o.preferred_username)
lambda o: "https://{}/users/{}followers".format(
o.domain.name, o.preferred_username
)
)
inbox_url = factory.LazyAttribute(
lambda o: "https://{}/users/{}/inbox".format(o.domain, o.preferred_username)
lambda o: "https://{}/users/{}/inbox".format(
o.domain.name, o.preferred_username
)
)
outbox_url = factory.LazyAttribute(
lambda o: "https://{}/users/{}/outbox".format(o.domain, o.preferred_username)
lambda o: "https://{}/users/{}/outbox".format(
o.domain.name, o.preferred_username
)
)
class Meta:
......@@ -95,7 +110,9 @@ class ActorFactory(factory.DjangoModelFactory):
return
from funkwhale_api.users.factories import UserFactory
self.domain = settings.FEDERATION_HOSTNAME
self.domain = models.Domain.objects.get_or_create(
name=settings.FEDERATION_HOSTNAME
)[0]
self.save(update_fields=["domain"])
if not create:
if extracted and hasattr(extracted, "pk"):
......
# Generated by Django 2.0.9 on 2018-12-26 19:35
from django.db import migrations, models
import django.db.models.deletion
import django.utils.timezone
class Migration(migrations.Migration):
dependencies = [("federation", "0012_auto_20180920_1803")]
operations = [
migrations.AlterField(
model_name="actor",
name="private_key",
field=models.TextField(blank=True, max_length=5000, null=True),
),
migrations.AlterField(
model_name="actor",
name="public_key",
field=models.TextField(blank=True, max_length=5000, null=True),
),
]
# Generated by Django 2.0.9 on 2018-12-05 09:58
from django.db import migrations, models
import django.db.models.deletion
import django.utils.timezone
class Migration(migrations.Migration):
dependencies = [("federation", "0013_auto_20181226_1935")]
operations = [
migrations.CreateModel(
name="Domain",
fields=[
(
"name",
models.CharField(max_length=255, primary_key=True, serialize=False),
),
(
"creation_date",
models.DateTimeField(default=django.utils.timezone.now),
),
],
),
migrations.AlterField(
model_name="actor",
name="domain",
field=models.CharField(max_length=1000, null=True),
),
migrations.RenameField("actor", "domain", "old_domain"),
migrations.AddField(
model_name="actor",
name="domain",
field=models.ForeignKey(
null=True,
on_delete=django.db.models.deletion.CASCADE,
related_name="actors",
to="federation.Domain",
),
),
migrations.AlterUniqueTogether(name="actor", unique_together=set()),
migrations.AlterUniqueTogether(
name="actor", unique_together={("domain", "preferred_username")}
),
]
# Generated by Django 2.0.9 on 2018-11-14 08:55
from django.db import migrations, models
import django.db.models.deletion
import django.utils.timezone
def populate_domains(apps, schema_editor):
Domain = apps.get_model("federation", "Domain")
Actor = apps.get_model("federation", "Actor")
domains = set(
[v.lower() for v in Actor.objects.values_list("old_domain", flat=True)]
)
for domain in sorted(domains):
print("Populating domain {}...".format(domain))
first_actor = (
Actor.objects.order_by("creation_date")
.exclude(creation_date=None)
.filter(old_domain__iexact=domain)
.first()
)
if first_actor:
first_seen = first_actor.creation_date
else:
first_seen = django.utils.timezone.now()
Domain.objects.update_or_create(
name=domain, defaults={"creation_date": first_seen}
)
for domain in Domain.objects.all():
Actor.objects.filter(old_domain__iexact=domain.name).update(domain=domain)
def skip(apps, schema_editor):
pass
class Migration(migrations.Migration):
dependencies = [("federation", "0014_auto_20181205_0958")]
operations = [
migrations.RunPython(populate_domains, skip),
migrations.AlterField(
model_name="actor",
name="domain",
field=models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="actors",
to="federation.Domain",
),
),
]
# Generated by Django 2.0.9 on 2018-12-27 16:05
import django.contrib.postgres.fields.jsonb
from django.db import migrations, models
import funkwhale_api.federation.models
class Migration(migrations.Migration):
dependencies = [("federation", "0015_populate_domains")]
operations = [
migrations.AddField(
model_name="domain",
name="nodeinfo",
field=django.contrib.postgres.fields.jsonb.JSONField(
default=funkwhale_api.federation.models.empty_dict, max_length=50000
),
),
migrations.AddField(
model_name="domain",
name="nodeinfo_fetch_date",
field=models.DateTimeField(blank=True, default=None, null=True),
),
]
......@@ -62,6 +62,72 @@ class ActorQuerySet(models.QuerySet):
return qs
class DomainQuerySet(models.QuerySet):
def external(self):
return self.exclude(pk=settings.FEDERATION_HOSTNAME)
def with_actors_count(self):
return self.annotate(actors_count=models.Count("actors", distinct=True))
def with_outbox_activities_count(self):
return self.annotate(
outbox_activities_count=models.Count("actors__outbox_activities")
)
class Domain(models.Model):
name = models.CharField(primary_key=True, max_length=255)
creation_date = models.DateTimeField(default=timezone.now)
nodeinfo_fetch_date = models.DateTimeField(default=None, null=True, blank=True)
nodeinfo = JSONField(default=empty_dict, max_length=50000, blank=True)
objects = DomainQuerySet.as_manager()
def __str__(self):
return self.name
def save(self, **kwargs):
lowercase_fields = ["name"]
for field in lowercase_fields:
v = getattr(self, field, None)
if v:
setattr(self, field, v.lower())
super().save(**kwargs)
def get_stats(self):
from funkwhale_api.music import models as music_models
data = Domain.objects.filter(pk=self.pk).aggregate(
actors=models.Count("actors", distinct=True),
outbox_activities=models.Count("actors__outbox_activities", distinct=True),
libraries=models.Count("actors__libraries", distinct=True),
received_library_follows=models.Count(
"actors__libraries__received_follows", distinct=True
),
emitted_library_follows=models.Count(
"actors__library_follows", distinct=True
),
)
data["artists"] = music_models.Artist.objects.filter(
from_activity__actor__domain_id=self.pk
).count()
data["albums"] = music_models.Album.objects.filter(
from_activity__actor__domain_id=self.pk
).count()
data["tracks"] = music_models.Track.objects.filter(
from_activity__actor__domain_id=self.pk
).count()
uploads = music_models.Upload.objects.filter(library__actor__domain_id=self.pk)
data["uploads"] = uploads.count()
data["media_total_size"] = uploads.aggregate(v=models.Sum("size"))["v"] or 0
data["media_downloaded_size"] = (
uploads.with_file().aggregate(v=models.Sum("size"))["v"] or 0
)
return data
class Actor(models.Model):
ap_type = "Actor"
......@@ -74,7 +140,7 @@ class Actor(models.Model):
shared_inbox_url = models.URLField(max_length=500, null=True, blank=True)
type = models.CharField(choices=TYPE_CHOICES, default="Person", max_length=25)
name = models.CharField(max_length=200, null=True, blank=True)
domain = models.CharField(max_length=1000)
domain = models.ForeignKey(Domain, on_delete=models.CASCADE, related_name="actors")
summary = models.CharField(max_length=500, null=True, blank=True)
preferred_username = models.CharField(max_length=200, null=True, blank=True)
public_key = models.TextField(max_length=5000, null=True, blank=True)
......@@ -110,36 +176,9 @@ class Actor(models.Model):
def __str__(self):
return "{}@{}".format(self.preferred_username, self.domain)
def save(self, **kwargs):
lowercase_fields = ["domain"]
for field in lowercase_fields:
v = getattr(self, field, None)
if v:
setattr(self, field, v.lower())
super().save(**kwargs)
@property
def is_local(self):
return self.domain == settings.FEDERATION_HOSTNAME
@property
def is_system(self):
from . import actors
return all(
[
settings.FEDERATION_HOSTNAME == self.domain,
self.preferred_username in actors.SYSTEM_ACTORS,
]
)
@property
def system_conf(self):
from . import actors
if self.is_system:
return actors.SYSTEM_ACTORS[self.preferred_username]
return self.domain_id == settings.FEDERATION_HOSTNAME
def get_approved_followers(self):
follows = self.received_follows.filter(approved=True)
......
......@@ -114,7 +114,7 @@ class ActorSerializer(serializers.Serializer):
if maf is not None:
kwargs["manually_approves_followers"] = maf
domain = urllib.parse.urlparse(kwargs["fid"]).netloc
kwargs["domain"] = domain
kwargs["domain"] = models.Domain.objects.get_or_create(pk=domain)[0]
for endpoint, url in self.initial_data.get("endpoints", {}).items():
if endpoint == "sharedInbox":
kwargs["shared_inbox_url"] = url
......@@ -888,3 +888,12 @@ class CollectionSerializer(serializers.Serializer):
if self.context.get("include_ap_context", True):
d["@context"] = AP_CONTEXT
return d
class NodeInfoLinkSerializer(serializers.Serializer):
href = serializers.URLField()
rel = serializers.URLField()
class NodeInfoSerializer(serializers.Serializer):
links = serializers.ListField(child=NodeInfoLinkSerializer(), min_length=1)
import datetime
import logging
import os
import requests
from django.conf import settings
from django.db.models import Q, F
......@@ -14,6 +15,7 @@ from funkwhale_api.music import models as music_models
from funkwhale_api.taskapp import celery
from . import models, signing
from . import serializers
from . import routes
logger = logging.getLogger(__name__)
......@@ -147,3 +149,40 @@ def deliver_to_remote(delivery):
delivery.attempts = F("attempts") + 1
delivery.is_delivered = True
delivery.save(update_fields=["last_attempt_date", "attempts", "is_delivered"])
def fetch_nodeinfo(domain_name):
s = session.get_session()
wellknown_url = "https://{}/.well-known/nodeinfo".format(domain_name)
response = s.get(
url=wellknown_url, timeout=5, verify=settings.EXTERNAL_REQUESTS_VERIFY_SSL
)
response.raise_for_status()
serializer = serializers.NodeInfoSerializer(data=response.json())
serializer.is_valid(raise_exception=True)
nodeinfo_url = None
for link in serializer.validated_data["links"]:
if link["rel"] == "http://nodeinfo.diaspora.software/ns/schema/2.0":
nodeinfo_url = link["href"]
break
response = s.get(
url=nodeinfo_url, timeout=5, verify=settings.EXTERNAL_REQUESTS_VERIFY_SSL
)
response.raise_for_status()
return response.json()
@celery.app.task(name="federation.update_domain_nodeinfo")
@celery.require_instance(
models.Domain.objects.external(), "domain", id_kwarg_name="domain_name"
)
def update_domain_nodeinfo(domain):
now = timezone.now()
try:
nodeinfo = {"status": "ok", "payload": fetch_nodeinfo(domain.name)}
except (requests.RequestException, serializers.serializers.ValidationError) as e:
nodeinfo = {"status": "error", "error": str(e)}
domain.nodeinfo_fetch_date = now
domain.nodeinfo = nodeinfo
domain.save(update_fields=["nodeinfo", "nodeinfo_fetch_date"])
......@@ -58,15 +58,6 @@ class RavenDSN(types.StringPreference):
field_kwargs = {"required": False}
@global_preferences_registry.register
class RavenEnabled(types.BooleanPreference):
show_in_api = True
section = raven
name = "front_enabled"
default = False
verbose_name = "Report front-end errors with Raven"
@global_preferences_registry.register
class InstanceNodeinfoEnabled(types.BooleanPreference):
show_in_api = False
......
from django_filters import rest_framework as filters
from funkwhale_api.common import fields
from funkwhale_api.federation import models as federation_models
from funkwhale_api.music import models as music_models
from funkwhale_api.users import models as users_models
......@@ -20,6 +21,14 @@ class ManageUploadFilterSet(filters.FilterSet):
fields = ["q", "track__album", "track__artist", "track"]
class ManageDomainFilterSet(filters.FilterSet):
q = fields.SearchFilter(search_fields=["name"])
class Meta:
model = federation_models.Domain
fields = ["name"]
class ManageUserFilterSet(filters.FilterSet):
q = fields.SearchFilter(search_fields=["username", "email", "name"])
......@@ -31,10 +40,9 @@ class ManageUserFilterSet(filters.FilterSet):
"privacy_level",
"is_staff",
"is_superuser",
"permission_upload",
"permission_library",
"permission_settings",
"permission_federation",
"permission_moderation",
]
......
......@@ -3,6 +3,7 @@ from django.db import transaction
from rest_framework import serializers
from funkwhale_api.common import serializers as common_serializers
from funkwhale_api.federation import models as federation_models
from funkwhale_api.music import models as music_models
from funkwhale_api.users import models as users_models
......@@ -168,3 +169,25 @@ class ManageInvitationActionSerializer(common_serializers.ActionSerializer):
@transaction.atomic
def handle_delete(self, objects):
return objects.delete()
class ManageDomainSerializer(serializers.ModelSerializer):
actors_count = serializers.SerializerMethodField()
outbox_activities_count = serializers.SerializerMethodField()
class Meta:
model = federation_models.Domain
fields = [
"name",
"creation_date",
"actors_count",
"outbox_activities_count",
"nodeinfo",
"nodeinfo_fetch_date",
]
def get_actors_count(self, o):
return getattr(o, "actors_count", 0)
def get_outbox_activities_count(self, o):
return getattr(o, "outbox_activities_count", 0)
......@@ -3,6 +3,8 @@ from rest_framework import routers
from . import views
federation_router = routers.SimpleRouter()
federation_router.register(r"domains", views.ManageDomainViewSet, "domains")
library_router = routers.SimpleRouter()
library_router.register(r"uploads", views.ManageUploadViewSet, "uploads")
users_router = routers.SimpleRouter()
......@@ -10,6 +12,10 @@ users_router.register(r"users", views.ManageUserViewSet, "users")
users_router.register(r"invitations", views.ManageInvitationViewSet, "invitations")
urlpatterns = [
url(
r"^federation/",
include((federation_router.urls, "federation"), namespace="federation"),
),
url(r"^library/", include((library_router.urls, "instance"), namespace="library")),
url(r"^users/", include((users_router.urls, "instance"), namespace="users")),
]
from rest_framework import mixins, response, viewsets
from rest_framework.decorators import list_route
from rest_framework.decorators import detail_route, list_route
from funkwhale_api.common import preferences
from funkwhale_api.federation import models as federation_models
from funkwhale_api.federation import tasks as federation_tasks
from funkwhale_api.music import models as music_models
from funkwhale_api.users import models as users_models
from funkwhale_api.users.permissions import HasUserPermission
......@@ -92,3 +94,38 @@ class ManageInvitationViewSet(
serializer.is_valid(raise_exception=True)
result = serializer.save()
return response.Response(result, status=200)
class ManageDomainViewSet(
mixins.ListModelMixin, mixins.RetrieveModelMixin, viewsets.GenericViewSet
):
lookup_value_regex = r"[a-zA-Z0-9\-\.]+"
queryset = (
federation_models.Domain.objects.external()
.with_actors_count()
.with_outbox_activities_count()
.order_by("name")
)
serializer_class = serializers.ManageDomainSerializer
filter_class = filters.ManageDomainFilterSet
permission_classes = (HasUserPermission,)
required_permissions = ["moderation"]
ordering_fields = [
"name",
"creation_date",
"nodeinfo_fetch_date",
"actors_count",
"outbox_activities_count",
]
@detail_route(methods=["get"])
def nodeinfo(self, request, *args, **kwargs):
domain = self.get_object()
federation_tasks.update_domain_nodeinfo(domain_name=domain.name)
domain.refresh_from_db()
return response.Response(domain.nodeinfo, status=200)
@detail_route(methods=["get"])
def stats(self, request, *args, **kwargs):
domain = self.get_object()
return response.Response(domain.get_stats(), status=200)
......@@ -143,7 +143,7 @@ class Command(BaseCommand):
for m in raw_matching:
# In some situations, the path is encoded incorrectly on the filesystem
# so we filter out faulty paths and display a warning to the user.
# see https://code.eliotberriot.com/funkwhale/funkwhale/issues/138
# see https://dev.funkwhale.audio/funkwhale/funkwhale/issues/138
try:
m.encode("utf-8")
matching.append(m)
......
......@@ -165,6 +165,9 @@ CONF = {
"field": "albumartist",
"to_application": split_and_return_first(";"),
},
"performer": {},
"arranger": {"field": "REMIXER"},
"composer": {},
"album": {},
"date": {"field": "date", "to_application": get_date},
"musicbrainz_albumid": {},
......@@ -189,6 +192,9 @@ CONF = {
"field": "albumartist",
"to_application": split_and_return_first(";"),
},
"performer": {},
"arranger": {"field": "REMIXER"},
"composer": {},
"album": {},
"date": {"field": "date", "to_application": get_date},
"musicbrainz_albumid": {},
......@@ -210,6 +216,9 @@ CONF = {
"title": {},
"artist": {},
"album_artist": {"field": "albumartist"},
"performer": {},
"arranger": {"field": "REMIXER"},
"composer": {},
"album": {},
"date": {"field": "date", "to_application": get_date},
"musicbrainz_albumid": {"field": "MusicBrainz Album Id"},
......@@ -230,6 +239,9 @@ CONF = {
"title": {"field": "TIT2"},
"artist": {"field": "TPE1"},
"album_artist": {"field": "TPE2"},
"performer": {"field": "TPE3"},
"arranger": {"field": "TPE4"},
"composer": {"field": "TCOM"},
"album": {"field": "TALB"},
"date": {"field": "TDRC", "to_application": get_date},
"musicbrainz_albumid": {"field": "MusicBrainz Album Id"},
......@@ -256,6 +268,9 @@ CONF = {
"title": {},
"artist": {},
"album_artist": {"field": "albumartist"},
"performer": {},
"arranger": {"field": "REMIXER"},
"composer": {},
"album": {},
"date": {"field": "date", "to_application": get_date},
"musicbrainz_albumid": {},
......
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import migrations, models
import django.db.models.deletion
def populate_track_credit(apps, schema_editor):
TrackCredit = apps.get_model("music", "TrackCredit")
Track = apps.get_model("music", "Track")
tracks = Track.objects.values("artist__id", "id").iterator()
credits = [
TrackCredit(artist_id=track["artist__id"], track_id=track["id"], type="main")
for track in tracks
]
TrackCredit.objects.bulk_create(credits)
def rewind(apps, schema_editor):
pass
class Migration(migrations.Migration):
dependencies = [
("music", "0036_track_disc_number"),
]
operations = [
migrations.CreateModel(
name="TrackCredit",
fields=[
("id", models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name="ID")),
("type", models.CharField(choices=[("main", "Main"), ("featured", "Featured"), ("composer", "Composer"), ("performer", "Performer"), ("arranger", "Arranger")], default="main", max_length=30)),
("artist", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name="track_credits", to="music.Artist")),
("track", models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name="track_credits", to="music.Track")),
],
),
migrations.AlterUniqueTogether(
name="trackcredit",
unique_together={("artist", "track", "type")},
),
migrations.RunPython(populate_track_credit, rewind),
]
......@@ -29,7 +29,7 @@ from funkwhale_api.federation import models as federation_models
from funkwhale_api.federation import utils as federation_utils
from . import importers, metadata, utils
logger = logging.getLogger(__file__)
logger = logging.getLogger(__name__)
def empty_dict():
......@@ -617,6 +617,40 @@ class UploadQuerySet(models.QuerySet):
def for_federation(self):
return self.filter(import_status="finished", mimetype__startswith="audio/")
def with_file(self):
return self.exclude(audio_file=None).exclude(audio_file="")
class Credit:
MAIN = "main"
FEATURED = "featured"
COMPOSER = "composer"