Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision
  • 1.4.1-upgrade-release
  • 1121-download
  • 1218-smartplaylist_backend
  • 1373-login-form-move-reset-your-password-link
  • 1381-progress-bars
  • 1481
  • 1518-update-django-allauth
  • 1645
  • 1675-widget-improperly-configured-missing-resource-id
  • 1675-widget-improperly-configured-missing-resource-id-2
  • 1704-required-props-are-not-always-passed
  • 1716-add-frontend-tests-again
  • 1749-smtp-uri-configuration
  • 1930-first-upload-in-a-batch-always-fails
  • 1976-update-documentation-links-in-readme-files
  • 2054-player-layout
  • 2063-funkwhale-connection-interrupted-every-so-often-requires-network-reset-page-refresh
  • 2091-iii-6-improve-visuals-layout
  • 2151-refused-to-load-spa-manifest-json-2
  • 2154-add-to-playlist-pop-up-hidden-by-now-playing-screen
  • 2155-can-t-see-the-episode-list-of-a-podcast-as-an-anonymous-user-with-anonymous-access-enabled
  • 2156-add-management-command-to-change-file-ref-for-in-place-imported-files-to-s3
  • 2192-clear-queue-bug-when-random-shuffle-is-enabled
  • 2205-channel-page-pagination-link-dont-working
  • 2215-custom-logger-does-not-work-at-all-with-webkit-and-blink-based-browsers
  • 2228-troi-real-world-review
  • 2274-implement-new-upload-api
  • 2303-allow-users-to-own-tagged-items
  • 2395-far-right-filter
  • 2405-front-buttont-trigger-third-party-hook
  • 2408-troi-create-missing-tracks
  • 2416-revert-library-drop
  • 2448-complete-tags
  • 2451-delete-no-user-query
  • 2452-fetch-third-party-metadata
  • 623-test
  • 653-enable-starting-embedded-player-at-a-specific-position-in-track
  • activitypub-overview
  • album-sliders
  • arne/2091-improve-visuals
  • back-option-for-edits
  • chore/2406-compose-modularity-scope
  • develop
  • develop-password-reset
  • env-file-cleanup
  • feat/2091-improve-visuals
  • fix-amd64-docker-build-gfortran
  • fix-gitpod
  • fix-plugins-dev-setup
  • fix-rate-limit-serializer
  • fix-schema-channel-metadata-choices
  • flupsi/2803-improve-visuals
  • flupsi/2804-new-upload-process
  • funkwhale-fix_pwa_manifest
  • funkwhale-petitminion-2136-bug-fix-prune-skipped-upload
  • funkwhale-ui-buttons
  • georg/add-typescript
  • gitpod/test-1866
  • global-button-experiment
  • global-buttons
  • juniorjpdj/pkg-repo
  • manage-py-reference
  • merge-review
  • minimal-python-version
  • petitminion-develop-patch-84496
  • pin-mutagen-to-1.46
  • pipenv
  • plugins
  • plugins-v2
  • plugins-v3
  • pre-release/1.3.0
  • prune_skipped_uploads_docs
  • renovate/front-all-dependencies
  • renovate/front-major-all-dependencies
  • schema-updates
  • small-gitpod-improvements
  • spectacular_schema
  • stable
  • tempArne
  • ui-buttons
  • update-frontend-dependencies
  • upload-process-spec
  • user-concept-docs
  • v2-artists
  • vite-ws-ssl-compatible
  • wip/2091-improve-visuals
  • wvffle/dependency-maintenance
  • wvffle/new-upload-process
  • wvffle/ui-rewrite-sidebar
  • wvffle/ui-sidebar
  • 0.1
  • 0.10
  • 0.11
  • 0.12
  • 0.13
  • 0.14
  • 0.14.1
  • 0.14.2
  • 0.15
  • 0.16
  • 0.16.1
  • 0.16.2
  • 0.16.3
  • 0.17
  • 0.18
  • 0.18.1
  • 0.18.2
  • 0.18.3
  • 0.19.0
  • 0.19.0-rc1
  • 0.19.0-rc2
  • 0.19.1
  • 0.2
  • 0.2.1
  • 0.2.2
  • 0.2.3
  • 0.2.4
  • 0.2.5
  • 0.2.6
  • 0.20.0
  • 0.20.0-rc1
  • 0.20.1
  • 0.21
  • 0.21-rc1
  • 0.21-rc2
  • 0.21.1
  • 0.21.2
  • 0.3
  • 0.3.1
  • 0.3.2
  • 0.3.3
  • 0.3.4
  • 0.3.5
  • 0.4
  • 0.5
  • 0.5.1
  • 0.5.2
  • 0.5.3
  • 0.5.4
  • 0.6
  • 0.6.1
  • 0.7
  • 0.8
  • 0.9
  • 0.9.1
  • 1.0
  • 1.0-rc1
  • 1.0.1
  • 1.1
  • 1.1-rc1
  • 1.1-rc2
  • 1.1.1
  • 1.1.2
  • 1.1.3
  • 1.1.4
  • 1.2.0
  • 1.2.0-rc1
  • 1.2.0-rc2
  • 1.2.0-testing
  • 1.2.0-testing2
  • 1.2.0-testing3
  • 1.2.0-testing4
  • 1.2.1
  • 1.2.10
  • 1.2.2
  • 1.2.3
  • 1.2.4
  • 1.2.5
  • 1.2.6
  • 1.2.6-1
  • 1.2.7
  • 1.2.8
  • 1.2.9
  • 1.3.0
  • 1.3.0-rc1
  • 1.3.0-rc2
  • 1.3.0-rc3
  • 1.3.0-rc4
  • 1.3.0-rc5
  • 1.3.0-rc6
  • 1.3.1
  • 1.3.2
  • 1.3.3
  • 1.3.4
  • 1.4.0
  • 1.4.0-rc1
  • 1.4.0-rc2
  • 1.4.1
  • 2.0.0-alpha.1
  • 2.0.0-alpha.2
190 results

Target

Select target project
  • funkwhale/funkwhale
  • Luclu7/funkwhale
  • mbothorel/funkwhale
  • EorlBruder/funkwhale
  • tcit/funkwhale
  • JocelynDelalande/funkwhale
  • eneiluj/funkwhale
  • reg/funkwhale
  • ButterflyOfFire/funkwhale
  • m4sk1n/funkwhale
  • wxcafe/funkwhale
  • andybalaam/funkwhale
  • jcgruenhage/funkwhale
  • pblayo/funkwhale
  • joshuaboniface/funkwhale
  • n3ddy/funkwhale
  • gegeweb/funkwhale
  • tohojo/funkwhale
  • emillumine/funkwhale
  • Te-k/funkwhale
  • asaintgenis/funkwhale
  • anoadragon453/funkwhale
  • Sakada/funkwhale
  • ilianaw/funkwhale
  • l4p1n/funkwhale
  • pnizet/funkwhale
  • dante383/funkwhale
  • interfect/funkwhale
  • akhardya/funkwhale
  • svfusion/funkwhale
  • noplanman/funkwhale
  • nykopol/funkwhale
  • roipoussiere/funkwhale
  • Von/funkwhale
  • aurieh/funkwhale
  • icaria36/funkwhale
  • floreal/funkwhale
  • paulwalko/funkwhale
  • comradekingu/funkwhale
  • FurryJulie/funkwhale
  • Legolars99/funkwhale
  • Vierkantor/funkwhale
  • zachhats/funkwhale
  • heyjake/funkwhale
  • sn0w/funkwhale
  • jvoisin/funkwhale
  • gordon/funkwhale
  • Alexander/funkwhale
  • bignose/funkwhale
  • qasim.ali/funkwhale
  • fakegit/funkwhale
  • Kxze/funkwhale
  • stenstad/funkwhale
  • creak/funkwhale
  • Kaze/funkwhale
  • Tixie/funkwhale
  • IISergII/funkwhale
  • lfuelling/funkwhale
  • nhaddag/funkwhale
  • yoasif/funkwhale
  • ifischer/funkwhale
  • keslerm/funkwhale
  • flupe/funkwhale
  • petitminion/funkwhale
  • ariasuni/funkwhale
  • ollie/funkwhale
  • ngaumont/funkwhale
  • techknowlogick/funkwhale
  • Shleeble/funkwhale
  • theflyingfrog/funkwhale
  • jonatron/funkwhale
  • neobrain/funkwhale
  • eorn/funkwhale
  • KokaKiwi/funkwhale
  • u1-liquid/funkwhale
  • marzzzello/funkwhale
  • sirenwatcher/funkwhale
  • newer027/funkwhale
  • codl/funkwhale
  • Zwordi/funkwhale
  • gisforgabriel/funkwhale
  • iuriatan/funkwhale
  • simon/funkwhale
  • bheesham/funkwhale
  • zeoses/funkwhale
  • accraze/funkwhale
  • meliurwen/funkwhale
  • divadsn/funkwhale
  • Etua/funkwhale
  • sdrik/funkwhale
  • Soran/funkwhale
  • kuba-orlik/funkwhale
  • cristianvogel/funkwhale
  • Forceu/funkwhale
  • jeff/funkwhale
  • der_scheibenhacker/funkwhale
  • owlnical/funkwhale
  • jovuit/funkwhale
  • SilverFox15/funkwhale
  • phw/funkwhale
  • mayhem/funkwhale
  • sridhar/funkwhale
  • stromlin/funkwhale
  • rrrnld/funkwhale
  • nitaibezerra/funkwhale
  • jaller94/funkwhale
  • pcouy/funkwhale
  • eduxstad/funkwhale
  • codingHahn/funkwhale
  • captain/funkwhale
  • polyedre/funkwhale
  • leishenailong/funkwhale
  • ccritter/funkwhale
  • lnceballosz/funkwhale
  • fpiesche/funkwhale
  • Fanyx/funkwhale
  • markusblogde/funkwhale
  • Firobe/funkwhale
  • devilcius/funkwhale
  • freaktechnik/funkwhale
  • blopware/funkwhale
  • cone/funkwhale
  • thanksd/funkwhale
  • vachan-maker/funkwhale
  • bbenti/funkwhale
  • tarator/funkwhale
  • prplecake/funkwhale
  • DMarzal/funkwhale
  • lullis/funkwhale
  • hanacgr/funkwhale
  • albjeremias/funkwhale
  • xeruf/funkwhale
  • llelite/funkwhale
  • RoiArthurB/funkwhale
  • cloo/funkwhale
  • nztvar/funkwhale
  • Keunes/funkwhale
  • petitminion/funkwhale-petitminion
  • m-idler/funkwhale
  • SkyLeite/funkwhale
140 results
Select Git revision
  • 303-json-ld
  • 629-cookie-auth
  • 735-table-truncate
  • develop
  • domain-policies
  • live-streaming
  • master
  • webdav
  • 0.1
  • 0.10
  • 0.11
  • 0.12
  • 0.13
  • 0.14
  • 0.14.1
  • 0.14.2
  • 0.15
  • 0.16
  • 0.16.1
  • 0.16.2
  • 0.16.3
  • 0.17
  • 0.18
  • 0.18.1
  • 0.18.2
  • 0.2
  • 0.2.1
  • 0.2.2
  • 0.2.3
  • 0.2.4
  • 0.2.5
  • 0.2.6
  • 0.3
  • 0.3.1
  • 0.3.2
  • 0.3.3
  • 0.3.4
  • 0.3.5
  • 0.4
  • 0.5
  • 0.5.1
  • 0.5.2
  • 0.5.3
  • 0.5.4
  • 0.6
  • 0.6.1
  • 0.7
  • 0.8
  • 0.9
  • 0.9.1
50 results
Show changes
Showing
with 1194 additions and 156 deletions
import logging
logger = logging.getLogger("funkwhale_api.cli")
from django.contrib.admin import register as initial_register, site, ModelAdmin # noqa
from django.contrib.admin import site # noqa: F401
from django.contrib.admin import ModelAdmin
from django.contrib.admin import register as initial_register
from django.db.models.fields.related import RelatedField
from . import models
from . import tasks
from . import models, tasks
def register(model):
......@@ -45,3 +46,20 @@ class MutationAdmin(ModelAdmin):
search_fields = ["created_by__preferred_username"]
list_filter = ["type", "is_approved", "is_applied"]
actions = [apply]
@register(models.Attachment)
class AttachmentAdmin(ModelAdmin):
list_display = [
"uuid",
"actor",
"url",
"file",
"size",
"mimetype",
"creation_date",
"last_fetch_date",
]
list_select_related = True
search_fields = ["actor__domain__name"]
list_filter = ["mimetype"]
from django.apps import AppConfig, apps
from django.conf import settings
from . import mutations
from config import plugins
from . import mutations, utils
class CommonConfig(AppConfig):
......@@ -11,3 +14,7 @@ class CommonConfig(AppConfig):
app_names = [app.name for app in apps.app_configs.values()]
mutations.registry.autodiscover(app_names)
utils.monkey_patch_request_build_absolute_uri()
plugins.startup.autodiscover([p + ".funkwhale_ready" for p in settings.PLUGINS])
for p in plugins._plugins.values():
p["settings"] = plugins.load_settings(p["name"], p["settings"])
from urllib.parse import parse_qs
from django.contrib.auth.models import AnonymousUser
from rest_framework import exceptions
from rest_framework_jwt.authentication import BaseJSONWebTokenAuthentication
from funkwhale_api.users.models import User
class TokenHeaderAuth(BaseJSONWebTokenAuthentication):
def get_jwt_value(self, request):
try:
qs = request.get("query_string", b"").decode("utf-8")
parsed = parse_qs(qs)
token = parsed["token"][0]
except KeyError:
raise exceptions.AuthenticationFailed("No token")
if not token:
raise exceptions.AuthenticationFailed("Empty token")
return token
class TokenAuthMiddleware:
def __init__(self, inner):
# Store the ASGI application we were passed
self.inner = inner
def __call__(self, scope):
auth = TokenHeaderAuth()
try:
user, token = auth.authenticate(scope)
except (User.DoesNotExist, exceptions.AuthenticationFailed):
user = AnonymousUser()
scope["user"] = user
return self.inner(scope)
from django.utils.encoding import smart_text
from django.utils.translation import ugettext as _
from allauth.account.models import EmailAddress
from django.core.cache import cache
from django.utils.translation import gettext as _
from oauth2_provider.contrib.rest_framework.authentication import (
OAuth2Authentication as BaseOAuth2Authentication,
)
from rest_framework import exceptions
from rest_framework_jwt import authentication
from rest_framework_jwt.settings import api_settings
from funkwhale_api.users import models as users_models
class JSONWebTokenAuthenticationQS(authentication.BaseJSONWebTokenAuthentication):
www_authenticate_realm = "api"
class UnverifiedEmail(Exception):
def __init__(self, user):
self.user = user
def get_jwt_value(self, request):
token = request.query_params.get("jwt")
if "jwt" in request.query_params and not token:
msg = _("Invalid Authorization header. No credentials provided.")
raise exceptions.AuthenticationFailed(msg)
return token
def authenticate_header(self, request):
return '{0} realm="{1}"'.format(
api_settings.JWT_AUTH_HEADER_PREFIX, self.www_authenticate_realm
)
def resend_confirmation_email(request, user):
THROTTLE_DELAY = 500
cache_key = f"auth:resent-email-confirmation:{user.pk}"
if cache.get(cache_key):
return False
# We do the sending of the conformation by hand because we don't want to pass the request down
# to the email rendering, which would cause another UnverifiedEmail Exception and restarts the sending
# again and again
email = EmailAddress.objects.get_for_user(user, user.email)
email.send_confirmation()
cache.set(cache_key, True, THROTTLE_DELAY)
return True
class BearerTokenHeaderAuth(authentication.BaseJSONWebTokenAuthentication):
"""
For backward compatibility purpose, we used Authorization: JWT <token>
but Authorization: Bearer <token> is probably better.
"""
www_authenticate_realm = "api"
class OAuth2Authentication(BaseOAuth2Authentication):
def authenticate(self, request):
try:
return super().authenticate(request)
except UnverifiedEmail as e:
request.oauth2_error = {"error": "unverified_email"}
resend_confirmation_email(request, e.user)
def get_jwt_value(self, request):
auth = authentication.get_authorization_header(request).split()
auth_header_prefix = "bearer"
if not auth:
if api_settings.JWT_AUTH_COOKIE:
return request.COOKIES.get(api_settings.JWT_AUTH_COOKIE)
return None
class ApplicationTokenAuthentication:
def authenticate(self, request):
try:
header = request.headers["Authorization"]
except KeyError:
return
if smart_text(auth[0].lower()) != auth_header_prefix:
return None
if "Bearer" not in header:
return
if len(auth) == 1:
msg = _("Invalid Authorization header. No credentials provided.")
raise exceptions.AuthenticationFailed(msg)
elif len(auth) > 2:
msg = _(
"Invalid Authorization header. Credentials string "
"should not contain spaces."
token = header.split()[-1].strip()
try:
application = users_models.Application.objects.exclude(user=None).get(
token=token
)
except users_models.Application.DoesNotExist:
return
user = users_models.User.objects.all().for_auth().get(id=application.user_id)
if not user.is_active:
msg = _("User account is disabled.")
raise exceptions.AuthenticationFailed(msg)
return auth[1]
def authenticate_header(self, request):
return '{0} realm="{1}"'.format("Bearer", self.www_authenticate_realm)
def authenticate(self, request):
auth = super().authenticate(request)
if auth:
if not auth[0].actor:
auth[0].create_actor()
return auth
class JSONWebTokenAuthentication(authentication.JSONWebTokenAuthentication):
def authenticate(self, request):
auth = super().authenticate(request)
if user.should_verify_email():
raise UnverifiedEmail(user)
if auth:
if not auth[0].actor:
auth[0].create_actor()
return auth
request.scopes = application.scope.split()
return user, None
import logging
from django_redis.client import default
logger = logging.getLogger(__name__)
class RedisClient(default.DefaultClient):
def get(self, key, default=None, version=None, client=None):
try:
return super().get(key, default=default, version=version, client=client)
except ValueError as e:
if "unsupported pickle protocol" in str(e):
# pickle deserialization error
logger.warn("Error while deserializing pickle value from cache")
return default
else:
raise
def get_many(self, *args, **kwargs):
try:
return super().get_many(*args, **kwargs)
except ValueError as e:
if "unsupported pickle protocol" in str(e):
# pickle deserialization error
logger.warn("Error while deserializing pickle value from cache")
return {}
else:
raise
......@@ -8,6 +8,7 @@ from django.core.serializers.json import DjangoJSONEncoder
logger = logging.getLogger(__name__)
channel_layer = get_channel_layer()
group_add = async_to_sync(channel_layer.group_add)
group_discard = async_to_sync(channel_layer.group_discard)
def group_send(group, event):
......
......@@ -14,7 +14,15 @@ class JsonAuthConsumer(JsonWebsocketConsumer):
def accept(self):
super().accept()
for group in self.groups:
channels.group_add(group, self.channel_name)
for group in self.scope["user"].get_channels_groups():
groups = self.scope["user"].get_channels_groups() + self.groups
for group in groups:
channels.group_add(group, self.channel_name)
def disconnect(self, close_code):
if self.scope.get("user", False) and self.scope.get("user").pk is not None:
groups = self.scope["user"].get_channels_groups() + self.groups
else:
groups = self.groups
for group in groups:
channels.group_discard(group, self.channel_name)
from django.db import transaction
from drf_spectacular.utils import OpenApiParameter, extend_schema
from rest_framework import decorators, exceptions, response, status
from rest_framework import decorators
from rest_framework import exceptions
from rest_framework import response
from rest_framework import status
from . import filters
from . import models
from . import filters, models
from . import mutations as common_mutations
from . import serializers
from . import signals
from . import tasks
from . import utils
from . import serializers, signals, tasks, utils
def action_route(serializer_class):
......@@ -87,4 +80,16 @@ def mutations_route(types):
)
return response.Response(serializer.data, status=status.HTTP_201_CREATED)
return decorators.action(methods=["get", "post"], detail=True)(mutations)
return extend_schema(
methods=["post"], responses=serializers.APIMutationSerializer()
)(
extend_schema(
methods=["get"],
responses=serializers.APIMutationSerializer(many=True),
parameters=[OpenApiParameter("id", location="query", exclude=True)],
)(
decorators.action(
methods=["get", "post"], detail=True, required_scope="edits"
)(mutations)
)
)
from dynamic_preferences import types
from dynamic_preferences.registries import global_preferences_registry
from funkwhale_api.common import preferences
common = types.Section("common")
@global_preferences_registry.register
class APIAutenticationRequired(
preferences.DefaultFromSettingMixin, types.BooleanPreference
):
class APIAutenticationRequired(types.BooleanPreference):
section = common
name = "api_authentication_required"
verbose_name = "API Requires authentication"
setting = "API_AUTHENTICATION_REQUIRED"
default = True
help_text = (
"If disabled, anonymous users will be able to query the API "
"and access music data (as well as other data exposed in the API "
......
import factory
from funkwhale_api.factories import registry, NoUpdateOnCreate
from funkwhale_api.factories import NoUpdateOnCreate, registry
from funkwhale_api.federation import factories as federation_factories
......@@ -16,10 +15,31 @@ class MutationFactory(NoUpdateOnCreate, factory.django.DjangoModelFactory):
class Meta:
model = "common.Mutation"
@factory.post_generation
def target(self, create, extracted, **kwargs):
if not create:
# Simple build, do nothing.
return
self.target = extracted
self.save()
@registry.register
class AttachmentFactory(NoUpdateOnCreate, factory.django.DjangoModelFactory):
url = factory.Faker("federation_url")
uuid = factory.Faker("uuid4")
actor = factory.SubFactory(federation_factories.ActorFactory)
file = factory.django.ImageField()
class Meta:
model = "common.Attachment"
@registry.register
class CommonFactory(NoUpdateOnCreate, factory.django.DjangoModelFactory):
text = factory.Faker("paragraph")
content_type = "text/plain"
class Meta:
model = "common.Content"
@registry.register
class PluginConfiguration(NoUpdateOnCreate, factory.django.DjangoModelFactory):
code = "test"
conf = {"foo": "bar"}
class Meta:
model = "common.PluginConfiguration"
import django_filters
from django import forms
from django.core.serializers.json import DjangoJSONEncoder
from django.db import models
from rest_framework import serializers
from . import search
......@@ -22,19 +24,38 @@ def privacy_level_query(user, lookup_field="privacy_level", user_field="user"):
if user.is_anonymous:
return models.Q(**{lookup_field: "everyone"})
return models.Q(
**{"{}__in".format(lookup_field): ["instance", "everyone"]}
) | models.Q(**{lookup_field: "me", user_field: user})
followers_query = models.Q(
**{
f"{lookup_field}": "followers",
f"{user_field}__actor__in": user.actor.get_approved_followings(),
}
)
# Federated TrackFavorite don't have an user associated with the trackfavorite.actor
# to do : if we implement the followers privacy_level this will become a problem
no_user_query = models.Q(**{f"{user_field}__isnull": True})
return (
models.Q(**{f"{lookup_field}__in": ["instance", "everyone"]})
| models.Q(**{lookup_field: "me", user_field: user})
| followers_query
| no_user_query
)
class SearchFilter(django_filters.CharFilter):
def __init__(self, *args, **kwargs):
self.search_fields = kwargs.pop("search_fields")
self.fts_search_fields = kwargs.pop("fts_search_fields", [])
super().__init__(*args, **kwargs)
def filter(self, qs, value):
if not value:
return qs
if self.fts_search_fields:
query = search.get_fts_query(
value, self.fts_search_fields, model=self.parent.Meta.model
)
else:
query = search.get_query(value, self.search_fields)
return qs.filter(query)
......@@ -52,3 +73,118 @@ class SmartSearchFilter(django_filters.CharFilter):
except forms.ValidationError:
return qs.none()
return search.apply(qs, cleaned)
def get_generic_filter_query(value, relation_name, choices):
parts = value.split(":", 1)
type = parts[0]
try:
conf = choices[type]
except KeyError:
raise forms.ValidationError("Invalid type")
related_queryset = conf["queryset"]
related_model = related_queryset.model
filter_query = models.Q(
**{
"{}_content_type__app_label".format(
relation_name
): related_model._meta.app_label,
"{}_content_type__model".format(
relation_name
): related_model._meta.model_name,
}
)
if len(parts) > 1:
id_attr = conf.get("id_attr", "id")
id_field = conf.get("id_field", serializers.IntegerField(min_value=1))
try:
id_value = parts[1]
id_value = id_field.to_internal_value(id_value)
except (TypeError, KeyError, serializers.ValidationError):
raise forms.ValidationError("Invalid id")
query_getter = conf.get(
"get_query", lambda attr, value: models.Q(**{attr: value})
)
obj_query = query_getter(id_attr, id_value)
try:
obj = related_queryset.get(obj_query)
except related_queryset.model.DoesNotExist:
raise forms.ValidationError("Invalid object")
filter_query &= models.Q(**{f"{relation_name}_id": obj.id})
return filter_query
class GenericRelationFilter(django_filters.CharFilter):
def __init__(self, relation_name, choices, *args, **kwargs):
self.relation_name = relation_name
self.choices = choices
super().__init__(*args, **kwargs)
def filter(self, qs, value):
if not value:
return qs
try:
filter_query = get_generic_filter_query(
value, relation_name=self.relation_name, choices=self.choices
)
except forms.ValidationError:
return qs.none()
return qs.filter(filter_query)
class GenericRelation(serializers.JSONField):
def __init__(self, choices, *args, **kwargs):
self.choices = choices
self.encoder = kwargs.setdefault("encoder", DjangoJSONEncoder)
super().__init__(*args, **kwargs)
def to_representation(self, value):
if not value:
return
type = None
id = None
id_attr = None
for key, choice in self.choices.items():
if isinstance(value, choice["queryset"].model):
type = key
id_attr = choice.get("id_attr", "id")
id = getattr(value, id_attr)
break
if type:
return {"type": type, id_attr: id}
def to_internal_value(self, v):
v = super().to_internal_value(v)
if not v or not isinstance(v, dict):
raise serializers.ValidationError("Invalid data")
try:
type = v["type"]
field = serializers.ChoiceField(choices=list(self.choices.keys()))
type = field.to_internal_value(type)
except (TypeError, KeyError, serializers.ValidationError):
raise serializers.ValidationError("Invalid type")
conf = self.choices[type]
id_attr = conf.get("id_attr", "id")
id_field = conf.get("id_field", serializers.IntegerField(min_value=1))
queryset = conf["queryset"]
try:
id_value = v[id_attr]
id_value = id_field.to_internal_value(id_value)
except (TypeError, KeyError, serializers.ValidationError):
raise serializers.ValidationError(f"Invalid {id_attr}")
query_getter = conf.get(
"get_query", lambda attr, value: models.Q(**{attr: value})
)
query = query_getter(id_attr, id_value)
try:
obj = queryset.get(query)
except queryset.model.DoesNotExist:
raise serializers.ValidationError("Object not found")
return obj
from django import forms
from django.db.models import Q
from django_filters import widgets
from django.db.models.functions import Lower
from django_filters import rest_framework as filters
from django_filters import widgets
from drf_spectacular.utils import extend_schema_field
from . import fields
from . import models
from . import search
from . import fields, models, search, utils
class NoneObject(object):
class NoneObject:
def __eq__(self, other):
return other.__class__ == NoneObject
NONE = NoneObject()
NULL_BOOLEAN_CHOICES = [
BOOLEAN_CHOICES = [
(True, True),
("true", True),
("True", True),
......@@ -26,6 +25,8 @@ NULL_BOOLEAN_CHOICES = [
("False", False),
("0", False),
("no", False),
]
NULL_BOOLEAN_CHOICES = BOOLEAN_CHOICES + [
("None", NONE),
("none", NONE),
("Null", NONE),
......@@ -46,9 +47,10 @@ class CoerceChoiceField(forms.ChoiceField):
try:
return [b for a, b in self.choices if v == a][0]
except IndexError:
raise forms.ValidationError("Invalid value {}".format(value))
raise forms.ValidationError(f"Invalid value {value}")
@extend_schema_field(bool)
class NullBooleanFilter(filters.ChoiceFilter):
field_class = CoerceChoiceField
......@@ -62,9 +64,7 @@ class NullBooleanFilter(filters.ChoiceFilter):
return qs
if value == NONE:
value = None
qs = self.get_method(qs)(
**{"%s__%s" % (self.field_name, self.lookup_expr): value}
)
qs = self.get_method(qs)(**{f"{self.field_name}__{self.lookup_expr}": value})
return qs.distinct() if self.distinct else qs
......@@ -76,10 +76,26 @@ def clean_null_boolean_filter(v):
return v
def clean_boolean_filter(v):
return CoerceChoiceField(choices=BOOLEAN_CHOICES).clean(v)
def get_null_boolean_filter(name):
return {"handler": lambda v: Q(**{name: clean_null_boolean_filter(v)})}
def get_boolean_filter(name):
return {"handler": lambda v: Q(**{name: clean_boolean_filter(v)})}
def get_generic_relation_filter(relation_name, choices):
return {
"handler": lambda v: fields.get_generic_filter_query(
v, relation_name=relation_name, choices=choices
)
}
class DummyTypedMultipleChoiceField(forms.TypedMultipleChoiceField):
def valid_value(self, value):
return True
......@@ -101,7 +117,30 @@ class MultipleQueryFilter(filters.TypedMultipleChoiceFilter):
def __init__(self, *args, **kwargs):
kwargs["widget"] = QueryArrayWidget()
super().__init__(*args, **kwargs)
self.lookup_expr = "in"
def filter_target(value):
config = {
"artist": ["artist", "target_id", int],
"album": ["album", "target_id", int],
"track": ["track", "target_id", int],
}
parts = value.lower().split(" ")
if parts[0].strip() not in config:
raise forms.ValidationError("Improper target")
conf = config[parts[0].strip()]
query = Q(target_content_type__model=conf[0])
if len(parts) > 1:
_, lookup_field, validator = conf
try:
lookup_value = validator(parts[1].strip())
except TypeError:
raise forms.ValidationError("Imparsable target id")
return query & Q(**{lookup_field: lookup_value})
return query
class MutationFilter(filters.FilterSet):
......@@ -116,7 +155,8 @@ class MutationFilter(filters.FilterSet):
filter_fields={
"domain": {"to": "created_by__domain__name__iexact"},
"is_approved": get_null_boolean_filter("is_approved"),
"is_applied": {"to": "is_applied"},
"target": {"handler": filter_target},
"is_applied": get_boolean_filter("is_applied"),
},
)
)
......@@ -124,3 +164,95 @@ class MutationFilter(filters.FilterSet):
class Meta:
model = models.Mutation
fields = ["is_approved", "is_applied", "type"]
class EmptyQuerySet(ValueError):
pass
class ActorScopeFilter(filters.CharFilter):
def __init__(self, *args, **kwargs):
self.actor_field = kwargs.pop("actor_field")
self.library_field = kwargs.pop("library_field", None)
super().__init__(*args, **kwargs)
def filter(self, queryset, value):
if not value:
return queryset
request = getattr(self.parent, "request", None)
if not request:
return queryset.none()
user = getattr(request, "user", None)
actor = getattr(user, "actor", None)
scopes = [v.strip().lower() for v in value.split(",")]
query = None
for scope in scopes:
try:
right_query = self.get_query(scope, user, actor)
except ValueError:
return queryset.none()
query = utils.join_queries_or(query, right_query)
return queryset.filter(query).distinct()
def get_query(self, scope, user, actor):
from funkwhale_api.federation import models as federation_models
if scope == "me":
return self.filter_me(actor)
elif scope == "all":
return Q(pk__gte=0)
elif scope == "subscribed":
if not actor or self.library_field is None:
raise EmptyQuerySet()
followed_libraries = federation_models.LibraryFollow.objects.filter(
approved=True, actor=user.actor
).values_list("target_id", flat=True)
if not self.library_field:
predicate = "pk__in"
else:
predicate = f"{self.library_field}__in"
return Q(**{predicate: followed_libraries})
elif scope.startswith("actor:"):
full_username = scope.split("actor:", 1)[1]
username, domain = full_username.split("@")
try:
actor = federation_models.Actor.objects.get(
preferred_username__iexact=username,
domain_id=domain,
)
except federation_models.Actor.DoesNotExist:
raise EmptyQuerySet()
return Q(**{self.actor_field: actor})
elif scope.startswith("domain:"):
domain = scope.split("domain:", 1)[1]
return Q(**{f"{self.actor_field}__domain_id": domain})
else:
raise EmptyQuerySet()
def filter_me(self, actor):
if not actor:
raise EmptyQuerySet()
return Q(**{self.actor_field: actor})
class CaseInsensitiveNameOrderingFilter(filters.OrderingFilter):
def filter(self, qs, value):
order_by = []
if value is None:
return qs
for param in value:
if param == "name":
order_by.append(Lower("name"))
else:
order_by.append(self.get_ordering_value(param))
return qs.order_by(*order_by)
# from https://gist.github.com/carlopires/1262033/c52ef0f7ce4f58108619508308372edd8d0bd518
ISO_639_CHOICES = [
("ab", "Abkhaz"),
("aa", "Afar"),
("af", "Afrikaans"),
("ak", "Akan"),
("sq", "Albanian"),
("am", "Amharic"),
("ar", "Arabic"),
("an", "Aragonese"),
("hy", "Armenian"),
("as", "Assamese"),
("av", "Avaric"),
("ae", "Avestan"),
("ay", "Aymara"),
("az", "Azerbaijani"),
("bm", "Bambara"),
("ba", "Bashkir"),
("eu", "Basque"),
("be", "Belarusian"),
("bn", "Bengali"),
("bh", "Bihari"),
("bi", "Bislama"),
("bs", "Bosnian"),
("br", "Breton"),
("bg", "Bulgarian"),
("my", "Burmese"),
("ca", "Catalan; Valencian"),
("ch", "Chamorro"),
("ce", "Chechen"),
("ny", "Chichewa; Chewa; Nyanja"),
("zh", "Chinese"),
("cv", "Chuvash"),
("kw", "Cornish"),
("co", "Corsican"),
("cr", "Cree"),
("hr", "Croatian"),
("cs", "Czech"),
("da", "Danish"),
("dv", "Divehi; Maldivian;"),
("nl", "Dutch"),
("dz", "Dzongkha"),
("en", "English"),
("eo", "Esperanto"),
("et", "Estonian"),
("ee", "Ewe"),
("fo", "Faroese"),
("fj", "Fijian"),
("fi", "Finnish"),
("fr", "French"),
("ff", "Fula"),
("gl", "Galician"),
("ka", "Georgian"),
("de", "German"),
("el", "Greek, Modern"),
("gn", "Guaraní"),
("gu", "Gujarati"),
("ht", "Haitian"),
("ha", "Hausa"),
("he", "Hebrew (modern)"),
("hz", "Herero"),
("hi", "Hindi"),
("ho", "Hiri Motu"),
("hu", "Hungarian"),
("ia", "Interlingua"),
("id", "Indonesian"),
("ie", "Interlingue"),
("ga", "Irish"),
("ig", "Igbo"),
("ik", "Inupiaq"),
("io", "Ido"),
("is", "Icelandic"),
("it", "Italian"),
("iu", "Inuktitut"),
("ja", "Japanese"),
("jv", "Javanese"),
("kl", "Kalaallisut"),
("kn", "Kannada"),
("kr", "Kanuri"),
("ks", "Kashmiri"),
("kk", "Kazakh"),
("km", "Khmer"),
("ki", "Kikuyu, Gikuyu"),
("rw", "Kinyarwanda"),
("ky", "Kirghiz, Kyrgyz"),
("kv", "Komi"),
("kg", "Kongo"),
("ko", "Korean"),
("ku", "Kurdish"),
("kj", "Kwanyama, Kuanyama"),
("la", "Latin"),
("lb", "Luxembourgish"),
("lg", "Luganda"),
("li", "Limburgish"),
("ln", "Lingala"),
("lo", "Lao"),
("lt", "Lithuanian"),
("lu", "Luba-Katanga"),
("lv", "Latvian"),
("gv", "Manx"),
("mk", "Macedonian"),
("mg", "Malagasy"),
("ms", "Malay"),
("ml", "Malayalam"),
("mt", "Maltese"),
("mi", "Māori"),
("mr", "Marathi (Marāṭhī)"),
("mh", "Marshallese"),
("mn", "Mongolian"),
("na", "Nauru"),
("nv", "Navajo, Navaho"),
("nb", "Norwegian Bokmål"),
("nd", "North Ndebele"),
("ne", "Nepali"),
("ng", "Ndonga"),
("nn", "Norwegian Nynorsk"),
("no", "Norwegian"),
("ii", "Nuosu"),
("nr", "South Ndebele"),
("oc", "Occitan"),
("oj", "Ojibwe, Ojibwa"),
("cu", "Old Church Slavonic"),
("om", "Oromo"),
("or", "Oriya"),
("os", "Ossetian, Ossetic"),
("pa", "Panjabi, Punjabi"),
("pi", "Pāli"),
("fa", "Persian"),
("pl", "Polish"),
("ps", "Pashto, Pushto"),
("pt", "Portuguese"),
("qu", "Quechua"),
("rm", "Romansh"),
("rn", "Kirundi"),
("ro", "Romanian, Moldavan"),
("ru", "Russian"),
("sa", "Sanskrit (Saṁskṛta)"),
("sc", "Sardinian"),
("sd", "Sindhi"),
("se", "Northern Sami"),
("sm", "Samoan"),
("sg", "Sango"),
("sr", "Serbian"),
("gd", "Scottish Gaelic"),
("sn", "Shona"),
("si", "Sinhala, Sinhalese"),
("sk", "Slovak"),
("sl", "Slovene"),
("so", "Somali"),
("st", "Southern Sotho"),
("es", "Spanish; Castilian"),
("su", "Sundanese"),
("sw", "Swahili"),
("ss", "Swati"),
("sv", "Swedish"),
("ta", "Tamil"),
("te", "Telugu"),
("tg", "Tajik"),
("th", "Thai"),
("ti", "Tigrinya"),
("bo", "Tibetan"),
("tk", "Turkmen"),
("tl", "Tagalog"),
("tn", "Tswana"),
("to", "Tonga"),
("tr", "Turkish"),
("ts", "Tsonga"),
("tt", "Tatar"),
("tw", "Twi"),
("ty", "Tahitian"),
("ug", "Uighur, Uyghur"),
("uk", "Ukrainian"),
("ur", "Urdu"),
("uz", "Uzbek"),
("ve", "Venda"),
("vi", "Vietnamese"),
("vo", "Volapük"),
("wa", "Walloon"),
("cy", "Welsh"),
("wo", "Wolof"),
("fy", "Western Frisian"),
("xh", "Xhosa"),
("yi", "Yiddish"),
("yo", "Yoruba"),
("za", "Zhuang, Chuang"),
("zu", "Zulu"),
]
ISO_639_BY_CODE = {code: name for code, name in ISO_639_CHOICES}
from django.conf import settings
from django.contrib.auth.management.commands.createsuperuser import (
Command as BaseCommand,
)
from django.core.management.base import CommandError
class Command(BaseCommand):
def handle(self, *apps_label, **options):
"""
Creating Django Superusers would bypass some of our username checks, which can lead to unexpected behaviour.
We therefore prohibit the execution of the command.
"""
force = settings.FORCE
if not force == 1:
raise CommandError(
"Running createsuperuser on your Funkwhale instance bypasses some of our checks "
"which can lead to unexpected behavior of your instance. We therefore suggest to "
"run `funkwhale-manage fw users create --superuser` instead."
)
return super().handle(*apps_label, **options)
import os
import debugpy
import uvicorn
from django.core.management import call_command
from django.core.management.commands.migrate import Command as BaseCommand
from funkwhale_api.common import preferences
from funkwhale_api.music.models import Library
from funkwhale_api.users.models import User
class Command(BaseCommand):
help = "Manage gitpod environment"
def add_arguments(self, parser):
parser.add_argument("command", nargs="?", type=str)
def handle(self, *args, **options):
command = options["command"]
if not command:
return self.show_help()
if command == "init":
return self.init()
if command == "dev":
return self.dev()
def show_help(self):
self.stdout.write("")
self.stdout.write("Available commands:")
self.stdout.write("init - Initialize gitpod workspace")
self.stdout.write("dev - Run Funkwhale in development mode with debug server")
self.stdout.write("")
def init(self):
user = User.objects.get(username="gitpod")
# Allow anonymous access
preferences.set("common__api_authentication_required", False)
# Download music catalog
os.system(
"git clone https://dev.funkwhale.audio/funkwhale/catalog.git /tmp/catalog"
)
os.system("mv -f /tmp/catalog/music /workspace/funkwhale/data")
os.system("rm -rf /tmp/catalog/music")
# Import music catalog into library
call_command(
"create_library",
"gitpod",
name="funkwhale/catalog",
privacy_level="everyone",
)
call_command(
"import_files",
Library.objects.get(actor=user.actor).uuid,
"/workspace/funkwhale/data/music/",
recursive=True,
in_place=True,
no_input=False,
)
def dev(self):
debugpy.listen(5678)
uvicorn.run(
"config.asgi:application",
host="0.0.0.0",
port=5000,
reload=True,
reload_dirs=[
"/workspace/funkwhale/api/config/",
"/workspace/funkwhale/api/funkwhale_api/",
],
)
import pathlib
from argparse import RawTextHelpFormatter
from django.core.management.base import BaseCommand
from django.db import transaction
from funkwhale_api.music import models
class Command(BaseCommand):
help = """
Update the reference for Uploads that have been imported with --in-place and are now moved to s3.
Please note: This does not move any file! Make sure you already moved the files to your s3 bucket.
Specify --source to filter the reference to update to files from a specific in-place directory. If no
--source is given, all in-place imported track references will be updated.
Specify --target to specify a subdirectory in the S3 bucket where you moved the files. If no --target is
given, the file is expected to be stored in the same path as before.
Examples:
Music File: /music/Artist/Album/track.ogg
--source: /music
--target unset
All files imported from /music will be updated and expected to be in the same folder structure in the bucket
Music File: /music/Artist/Album/track.ogg
--source: /music
--target: /in_place
The music file is expected to be stored in the bucket in the directory /in_place/Artist/Album/track.ogg
"""
def create_parser(self, *args, **kwargs):
parser = super().create_parser(*args, **kwargs)
parser.formatter_class = RawTextHelpFormatter
return parser
def add_arguments(self, parser):
parser.add_argument(
"--no-dry-run",
action="store_false",
dest="dry_run",
default=True,
help="Disable dry run mode and apply updates for real on the database",
)
parser.add_argument(
"--source",
type=pathlib.Path,
required=True,
help="Specify the path of the directory where the files originally were stored to update their reference.",
)
parser.add_argument(
"--target",
type=pathlib.Path,
help="Specify a subdirectory in the S3 bucket where you moved the files to.",
)
@transaction.atomic
def handle(self, *args, **options):
if options["dry_run"]:
self.stdout.write("Dry-run on, will not touch the database")
else:
self.stdout.write("Dry-run off, *changing the database*")
self.stdout.write("")
prefix = f"file://{options['source']}"
to_change = models.Upload.objects.filter(source__startswith=prefix)
self.stdout.write(f"Found {to_change.count()} uploads to update.")
target = options.get("target")
if target is None:
target = options["source"]
for upl in to_change:
upl.audio_file = str(upl.source).replace(str(prefix), str(target))
upl.source = None
self.stdout.write(f"Upload expected in {upl.audio_file}")
if not options["dry_run"]:
upl.save()
self.stdout.write("")
if options["dry_run"]:
self.stdout.write(
"Nothing was updated, rerun this command with --no-dry-run to apply the changes"
)
else:
self.stdout.write("Updating completed!")
self.stdout.write("")
import math
import random
from django.conf import settings
from django.core.management.base import BaseCommand
from django.db import transaction
from funkwhale_api.federation import keys
from funkwhale_api.federation import models as federation_models
from funkwhale_api.music import models as music_models
from funkwhale_api.tags import models as tags_models
from funkwhale_api.users import models as users_models
BATCH_SIZE = 500
def create_local_accounts(factories, count, dependencies):
password = factories["users.User"].build().password
users = factories["users.User"].build_batch(size=count)
for user in users:
# we set the hashed password by hand, because computing one for each user
# is CPU intensive
user.password = password
users = users_models.User.objects.bulk_create(users, batch_size=BATCH_SIZE)
actors = []
domain = federation_models.Domain.objects.get_or_create(
name=settings.FEDERATION_HOSTNAME
)[0]
users = [u for u in users if u.pk]
private, public = keys.get_key_pair()
for user in users:
if not user.pk:
continue
actor = federation_models.Actor(
private_key=private.decode("utf-8"),
public_key=public.decode("utf-8"),
**users_models.get_actor_data(user.username, domain=domain)
)
actors.append(actor)
actors = federation_models.Actor.objects.bulk_create(actors, batch_size=BATCH_SIZE)
for user, actor in zip(users, actors):
user.actor = actor
users_models.User.objects.bulk_update(users, ["actor"])
return actors
def create_taggable_items(dependency):
def inner(factories, count, dependencies):
objs = []
tagged_objects = dependencies.get(
dependency, list(CONFIG_BY_ID[dependency]["model"].objects.all().only("pk"))
)
tags = dependencies.get("tags", list(tags_models.Tag.objects.all().only("pk")))
for i in range(count):
tag = random.choice(tags)
tagged_object = random.choice(tagged_objects)
objs.append(
factories["tags.TaggedItem"].build(
content_object=tagged_object, tag=tag
)
)
return tags_models.TaggedItem.objects.bulk_create(
objs, batch_size=BATCH_SIZE, ignore_conflicts=True
)
return inner
CONFIG = [
{
"id": "artist_credit",
"model": music_models.ArtistCredit,
"factory": "music.ArtistCredit",
"factory_kwargs": {"joinphrase": ""},
"depends_on": [
{"field": "artist", "id": "artists", "default_factor": 0.5},
],
},
{
"id": "tracks",
"model": music_models.Track,
"factory": "music.Track",
"factory_kwargs": {"album": None},
"depends_on": [
{"field": "album", "id": "albums", "default_factor": 0.1},
{"field": "artist_credit", "id": "artist_credit", "default_factor": 0.05},
],
},
{
"id": "albums",
"model": music_models.Album,
"factory": "music.Album",
"factory_kwargs": {},
"depends_on": [
{"field": "artist_credit", "id": "artist_credit", "default_factor": 0.3}
],
},
{"id": "artists", "model": music_models.Artist, "factory": "music.Artist"},
{
"id": "local_accounts",
"model": federation_models.Actor,
"handler": create_local_accounts,
},
{
"id": "local_libraries",
"model": music_models.Library,
"factory": "music.Library",
"factory_kwargs": {"actor": None},
"depends_on": [{"field": "actor", "id": "local_accounts", "default_factor": 1}],
},
{
"id": "local_uploads",
"model": music_models.Upload,
"factory": "music.Upload",
"factory_kwargs": {"import_status": "finished", "library": None, "track": None},
"depends_on": [
{
"field": "library",
"id": "local_libraries",
"default_factor": 0.05,
"queryset": music_models.Library.objects.all().select_related(
"actor__user"
),
},
{"field": "track", "id": "tracks", "default_factor": 1},
],
},
{"id": "tags", "model": tags_models.Tag, "factory": "tags.Tag"},
{
"id": "track_tags",
"model": tags_models.TaggedItem,
"queryset": tags_models.TaggedItem.objects.filter(
content_type__app_label="music", content_type__model="track"
),
"handler": create_taggable_items("tracks"),
"depends_on": [
{
"field": "tag",
"id": "tags",
"default_factor": 0.1,
"queryset": tags_models.Tag.objects.all(),
"set": False,
},
{
"field": "content_object",
"id": "tracks",
"default_factor": 1,
"set": False,
},
],
},
{
"id": "album_tags",
"model": tags_models.TaggedItem,
"queryset": tags_models.TaggedItem.objects.filter(
content_type__app_label="music", content_type__model="album"
),
"handler": create_taggable_items("albums"),
"depends_on": [
{
"field": "tag",
"id": "tags",
"default_factor": 0.1,
"queryset": tags_models.Tag.objects.all(),
"set": False,
},
{
"field": "content_object",
"id": "albums",
"default_factor": 1,
"set": False,
},
],
},
{
"id": "artist_tags",
"model": tags_models.TaggedItem,
"queryset": tags_models.TaggedItem.objects.filter(
content_type__app_label="music", content_type__model="artist"
),
"handler": create_taggable_items("artists"),
"depends_on": [
{
"field": "tag",
"id": "tags",
"default_factor": 0.1,
"queryset": tags_models.Tag.objects.all(),
"set": False,
},
{
"field": "content_object",
"id": "artists",
"default_factor": 1,
"set": False,
},
],
},
]
CONFIG_BY_ID = {c["id"]: c for c in CONFIG}
class Rollback(Exception):
pass
def create_objects(row, factories, count, **factory_kwargs):
return factories[row["factory"]].build_batch(size=count, **factory_kwargs)
class Command(BaseCommand):
help = """
Inject demo data into your database. Useful for load testing, or setting up a demo instance.
Use with caution and only if you know what you are doing.
"""
def add_arguments(self, parser):
parser.add_argument(
"--no-dry-run",
action="store_false",
dest="dry_run",
help="Commit the changes to the database",
)
parser.add_argument(
"--create-dependencies", action="store_true", dest="create_dependencies"
)
for row in CONFIG:
parser.add_argument(
"--{}".format(row["id"].replace("_", "-")),
dest=row["id"],
type=int,
help="Number of {} objects to create".format(row["id"]),
)
dependencies = row.get("depends_on", [])
for dependency in dependencies:
parser.add_argument(
"--{}-{}-factor".format(row["id"], dependency["field"]),
dest="{}_{}_factor".format(row["id"], dependency["field"]),
type=float,
help="Number of {} objects to create per {} object".format(
dependency["id"], row["id"]
),
)
def handle(self, *args, **options):
from django.apps import apps
from funkwhale_api import factories
app_names = [app.name for app in apps.app_configs.values()]
factories.registry.autodiscover(app_names)
try:
return self.inner_handle(*args, **options)
except Rollback:
pass
@transaction.atomic
def inner_handle(self, *args, **options):
results = {}
for row in CONFIG:
self.create_batch(row, results, options, count=options.get(row["id"]))
self.stdout.write("\nFinal state of database:\n\n")
for row in CONFIG:
qs = row.get("queryset", row["model"].objects.all())
total = qs.count()
self.stdout.write("- {} {} objects".format(total, row["id"]))
self.stdout.write("")
if options["dry_run"]:
self.stdout.write(
"Run this command with --no-dry-run to commit the changes to the database"
)
raise Rollback()
self.stdout.write(self.style.SUCCESS("Done!"))
def create_batch(self, row, results, options, count):
from funkwhale_api import factories
if row["id"] in results:
# already generated
return results[row["id"]]
if not count:
return []
dependencies = row.get("depends_on", [])
create_dependencies = options.get("create_dependencies")
for dependency in dependencies:
dep_count = options.get(dependency["id"])
if not create_dependencies and dep_count is None:
continue
if dep_count is None:
factor = options[
"{}_{}_factor".format(row["id"], dependency["field"])
] or dependency.get("default_factor")
dep_count = math.ceil(factor * count)
results[dependency["id"]] = self.create_batch(
CONFIG_BY_ID[dependency["id"]], results, options, count=dep_count
)
self.stdout.write("Creating {} {}…".format(count, row["id"]))
handler = row.get("handler")
if handler:
objects = handler(factories.registry, count, dependencies=results)
else:
objects = create_objects(
row, factories.registry, count, **row.get("factory_kwargs", {})
)
for dependency in dependencies:
if not dependency.get("set", True):
continue
if create_dependencies:
candidates = results[dependency["id"]]
else:
# we use existing objects in the database
queryset = dependency.get(
"queryset", CONFIG_BY_ID[dependency["id"]]["model"].objects.all()
)
candidates = list(queryset.values_list("pk", flat=True))
picked_pks = [random.choice(candidates) for _ in objects]
picked_objects = {o.pk: o for o in queryset.filter(pk__in=picked_pks)}
saved_obj = []
for i, obj in enumerate(objects):
if create_dependencies:
value = random.choice(candidates)
else:
value = picked_objects[picked_pks[i]]
if dependency["field"] == "artist_credit":
obj.save()
obj.artist_credit.set([value])
saved_obj.append(obj)
else:
setattr(obj, dependency["field"], value)
if saved_obj:
return saved_obj
if not handler:
objects = row["model"].objects.bulk_create(objects, batch_size=BATCH_SIZE)
results[row["id"]] = objects
return objects
import os
from django.conf import settings
from django.core.management.base import CommandError
from django.core.management.commands.makemigrations import Command as BaseCommand
......@@ -11,8 +10,8 @@ class Command(BaseCommand):
We ensure the command is disabled, unless a specific env var is provided.
"""
force = os.environ.get("FORCE") == "1"
if not force:
force = settings.FORCE
if not force == 1:
raise CommandError(
"Running makemigrations on your Funkwhale instance can have desastrous"
" consequences. This command is disabled, and should only be run in "
......
......@@ -26,7 +26,7 @@ class Command(BaseCommand):
script = available_scripts[name]
except KeyError:
raise CommandError(
"{} is not a valid script. Run python manage.py script for a "
"{} is not a valid script. Run funkwhale-manage script for a "
"list of available scripts".format(name)
)
......@@ -43,14 +43,14 @@ class Command(BaseCommand):
def show_help(self):
self.stdout.write("")
self.stdout.write("Available scripts:")
self.stdout.write("Launch with: python manage.py <script_name>")
self.stdout.write("Launch with: funkwhale-manage script <script_name>")
available_scripts = self.get_scripts()
for name, script in sorted(available_scripts.items()):
self.stdout.write("")
self.stdout.write(self.style.SUCCESS(name))
self.stdout.write("")
for line in script["help"].splitlines():
self.stdout.write(" {}".format(line))
self.stdout.write(f" {line}")
self.stdout.write("")
def get_scripts(self):
......