Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision
  • 1.4.1-upgrade-release
  • 1121-download
  • 1218-smartplaylist_backend
  • 1373-login-form-move-reset-your-password-link
  • 1381-progress-bars
  • 1481
  • 1518-update-django-allauth
  • 1645
  • 1675-widget-improperly-configured-missing-resource-id
  • 1675-widget-improperly-configured-missing-resource-id-2
  • 1704-required-props-are-not-always-passed
  • 1716-add-frontend-tests-again
  • 1749-smtp-uri-configuration
  • 1930-first-upload-in-a-batch-always-fails
  • 1976-update-documentation-links-in-readme-files
  • 2054-player-layout
  • 2063-funkwhale-connection-interrupted-every-so-often-requires-network-reset-page-refresh
  • 2091-iii-6-improve-visuals-layout
  • 2151-refused-to-load-spa-manifest-json-2
  • 2154-add-to-playlist-pop-up-hidden-by-now-playing-screen
  • 2155-can-t-see-the-episode-list-of-a-podcast-as-an-anonymous-user-with-anonymous-access-enabled
  • 2156-add-management-command-to-change-file-ref-for-in-place-imported-files-to-s3
  • 2192-clear-queue-bug-when-random-shuffle-is-enabled
  • 2205-channel-page-pagination-link-dont-working
  • 2215-custom-logger-does-not-work-at-all-with-webkit-and-blink-based-browsers
  • 2228-troi-real-world-review
  • 2274-implement-new-upload-api
  • 2303-allow-users-to-own-tagged-items
  • 2395-far-right-filter
  • 2405-front-buttont-trigger-third-party-hook
  • 2408-troi-create-missing-tracks
  • 2416-revert-library-drop
  • 2422-trigger-libraries-follow-on-user-follow
  • 2429-fix-popover-auto-close
  • 2448-complete-tags
  • 2452-fetch-third-party-metadata
  • 2469-Fix-search-bar-in-ManageUploads
  • 2476-deep-upload-links
  • 2490-experiment-use-rstore
  • 2490-experimental-use-simple-data-store
  • 2490-fix-search-modal
  • 2490-search-modal
  • 2501-fix-compatibility-with-older-browsers
  • 2502-drop-uno-and-jquery
  • 2533-allow-followers-in-user-activiy-privacy-level
  • 2539-drop-ansible-installation-method-in-favor-of-docker
  • 2560-default-modal-width
  • 623-test
  • 653-enable-starting-embedded-player-at-a-specific-position-in-track
  • activitypub-overview
  • album-sliders
  • arne/2091-improve-visuals
  • back-option-for-edits
  • chore/2406-compose-modularity-scope
  • develop
  • develop-password-reset
  • env-file-cleanup
  • feat/2091-improve-visuals
  • feature/2481-vui-translations
  • fix-amd64-docker-build-gfortran
  • fix-front-node-version
  • fix-gitpod
  • fix-plugins-dev-setup
  • fix-rate-limit-serializer
  • fix-schema-channel-metadata-choices
  • flupsi/2803-improve-visuals
  • flupsi/2804-new-upload-process
  • funkwhale-fix_pwa_manifest
  • funkwhale-petitminion-2136-bug-fix-prune-skipped-upload
  • funkwhale-ui-buttons
  • georg/add-typescript
  • gitpod/test-1866
  • global-button-experiment
  • global-buttons
  • juniorjpdj/pkg-repo
  • manage-py-reference
  • merge-review
  • minimal-python-version
  • petitminion-develop-patch-84496
  • pin-mutagen-to-1.46
  • pipenv
  • plugins
  • plugins-v2
  • plugins-v3
  • pre-release/1.3.0
  • prune_skipped_uploads_docs
  • refactor/homepage
  • renovate/front-all-dependencies
  • renovate/front-major-all-dependencies
  • schema-updates
  • small-gitpod-improvements
  • spectacular_schema
  • stable
  • tempArne
  • ui-buttons
  • update-frontend-dependencies
  • upload-process-spec
  • user-concept-docs
  • v2-artists
  • vite-ws-ssl-compatible
  • 0.1
  • 0.10
  • 0.11
  • 0.12
  • 0.13
  • 0.14
  • 0.14.1
  • 0.14.2
  • 0.15
  • 0.16
  • 0.16.1
  • 0.16.2
  • 0.16.3
  • 0.17
  • 0.18
  • 0.18.1
  • 0.18.2
  • 0.18.3
  • 0.19.0
  • 0.19.0-rc1
  • 0.19.0-rc2
  • 0.19.1
  • 0.2
  • 0.2.1
  • 0.2.2
  • 0.2.3
  • 0.2.4
  • 0.2.5
  • 0.2.6
  • 0.20.0
  • 0.20.0-rc1
  • 0.20.1
  • 0.21
  • 0.21-rc1
  • 0.21-rc2
  • 0.21.1
  • 0.21.2
  • 0.3
  • 0.3.1
  • 0.3.2
  • 0.3.3
  • 0.3.4
  • 0.3.5
  • 0.4
  • 0.5
  • 0.5.1
  • 0.5.2
  • 0.5.3
  • 0.5.4
  • 0.6
  • 0.6.1
  • 0.7
  • 0.8
  • 0.9
  • 0.9.1
  • 1.0
  • 1.0-rc1
  • 1.0.1
  • 1.1
  • 1.1-rc1
  • 1.1-rc2
  • 1.1.1
  • 1.1.2
  • 1.1.3
  • 1.1.4
  • 1.2.0
  • 1.2.0-rc1
  • 1.2.0-rc2
  • 1.2.0-testing
  • 1.2.0-testing2
  • 1.2.0-testing3
  • 1.2.0-testing4
  • 1.2.1
  • 1.2.10
  • 1.2.2
  • 1.2.3
  • 1.2.4
  • 1.2.5
  • 1.2.6
  • 1.2.6-1
  • 1.2.7
  • 1.2.8
  • 1.2.9
  • 1.3.0
  • 1.3.0-rc1
  • 1.3.0-rc2
  • 1.3.0-rc3
  • 1.3.0-rc4
  • 1.3.0-rc5
  • 1.3.0-rc6
  • 1.3.1
  • 1.3.2
  • 1.3.3
  • 1.3.4
  • 1.4.0
  • 1.4.0-rc1
  • 1.4.0-rc2
  • 1.4.1
  • 2.0.0-alpha.1
  • 2.0.0-alpha.2
200 results

Target

Select target project
  • funkwhale/funkwhale
  • Luclu7/funkwhale
  • mbothorel/funkwhale
  • EorlBruder/funkwhale
  • tcit/funkwhale
  • JocelynDelalande/funkwhale
  • eneiluj/funkwhale
  • reg/funkwhale
  • ButterflyOfFire/funkwhale
  • m4sk1n/funkwhale
  • wxcafe/funkwhale
  • andybalaam/funkwhale
  • jcgruenhage/funkwhale
  • pblayo/funkwhale
  • joshuaboniface/funkwhale
  • n3ddy/funkwhale
  • gegeweb/funkwhale
  • tohojo/funkwhale
  • emillumine/funkwhale
  • Te-k/funkwhale
  • asaintgenis/funkwhale
  • anoadragon453/funkwhale
  • Sakada/funkwhale
  • ilianaw/funkwhale
  • l4p1n/funkwhale
  • pnizet/funkwhale
  • dante383/funkwhale
  • interfect/funkwhale
  • akhardya/funkwhale
  • svfusion/funkwhale
  • noplanman/funkwhale
  • nykopol/funkwhale
  • roipoussiere/funkwhale
  • Von/funkwhale
  • aurieh/funkwhale
  • icaria36/funkwhale
  • floreal/funkwhale
  • paulwalko/funkwhale
  • comradekingu/funkwhale
  • FurryJulie/funkwhale
  • Legolars99/funkwhale
  • Vierkantor/funkwhale
  • zachhats/funkwhale
  • heyjake/funkwhale
  • sn0w/funkwhale
  • jvoisin/funkwhale
  • gordon/funkwhale
  • Alexander/funkwhale
  • bignose/funkwhale
  • qasim.ali/funkwhale
  • fakegit/funkwhale
  • Kxze/funkwhale
  • stenstad/funkwhale
  • creak/funkwhale
  • Kaze/funkwhale
  • Tixie/funkwhale
  • IISergII/funkwhale
  • lfuelling/funkwhale
  • nhaddag/funkwhale
  • yoasif/funkwhale
  • ifischer/funkwhale
  • keslerm/funkwhale
  • flupe/funkwhale
  • petitminion/funkwhale
  • ariasuni/funkwhale
  • ollie/funkwhale
  • ngaumont/funkwhale
  • techknowlogick/funkwhale
  • Shleeble/funkwhale
  • theflyingfrog/funkwhale
  • jonatron/funkwhale
  • neobrain/funkwhale
  • eorn/funkwhale
  • KokaKiwi/funkwhale
  • u1-liquid/funkwhale
  • marzzzello/funkwhale
  • sirenwatcher/funkwhale
  • newer027/funkwhale
  • codl/funkwhale
  • Zwordi/funkwhale
  • gisforgabriel/funkwhale
  • iuriatan/funkwhale
  • simon/funkwhale
  • bheesham/funkwhale
  • zeoses/funkwhale
  • accraze/funkwhale
  • meliurwen/funkwhale
  • divadsn/funkwhale
  • Etua/funkwhale
  • sdrik/funkwhale
  • Soran/funkwhale
  • kuba-orlik/funkwhale
  • cristianvogel/funkwhale
  • Forceu/funkwhale
  • jeff/funkwhale
  • der_scheibenhacker/funkwhale
  • owlnical/funkwhale
  • jovuit/funkwhale
  • SilverFox15/funkwhale
  • phw/funkwhale
  • mayhem/funkwhale
  • sridhar/funkwhale
  • stromlin/funkwhale
  • rrrnld/funkwhale
  • nitaibezerra/funkwhale
  • jaller94/funkwhale
  • pcouy/funkwhale
  • eduxstad/funkwhale
  • codingHahn/funkwhale
  • captain/funkwhale
  • polyedre/funkwhale
  • leishenailong/funkwhale
  • ccritter/funkwhale
  • lnceballosz/funkwhale
  • fpiesche/funkwhale
  • Fanyx/funkwhale
  • markusblogde/funkwhale
  • Firobe/funkwhale
  • devilcius/funkwhale
  • freaktechnik/funkwhale
  • blopware/funkwhale
  • cone/funkwhale
  • thanksd/funkwhale
  • vachan-maker/funkwhale
  • bbenti/funkwhale
  • tarator/funkwhale
  • prplecake/funkwhale
  • DMarzal/funkwhale
  • lullis/funkwhale
  • hanacgr/funkwhale
  • albjeremias/funkwhale
  • xeruf/funkwhale
  • llelite/funkwhale
  • RoiArthurB/funkwhale
  • cloo/funkwhale
  • nztvar/funkwhale
  • Keunes/funkwhale
  • petitminion/funkwhale-petitminion
  • m-idler/funkwhale
  • SkyLeite/funkwhale
140 results
Select Git revision
  • 278-search-browse
  • 303-json-ld
  • 316-ultrasonic
  • 334-don-t-display-an-empty-page-browser
  • 463-user-libraries
  • 463-user-libraries-full
  • ButterflyOfFire/funkwhale-patch-1
  • avatar-everywhere
  • develop
  • master
  • playlist-component
  • 0.1
  • 0.10
  • 0.11
  • 0.12
  • 0.13
  • 0.14
  • 0.14.1
  • 0.14.2
  • 0.15
  • 0.16
  • 0.16.1
  • 0.16.2
  • 0.16.3
  • 0.2
  • 0.2.1
  • 0.2.2
  • 0.2.3
  • 0.2.4
  • 0.2.5
  • 0.2.6
  • 0.3
  • 0.3.1
  • 0.3.2
  • 0.3.3
  • 0.3.4
  • 0.3.5
  • 0.4
  • 0.5
  • 0.5.1
  • 0.5.2
  • 0.5.3
  • 0.5.4
  • 0.6
  • 0.6.1
  • 0.7
  • 0.8
  • 0.9
  • 0.9.1
49 results
Show changes
Showing
with 1894 additions and 6 deletions
import pathlib
from argparse import RawTextHelpFormatter
from django.core.management.base import BaseCommand
from django.db import transaction
from funkwhale_api.music import models
class Command(BaseCommand):
help = """
Update the reference for Uploads that have been imported with --in-place and are now moved to s3.
Please note: This does not move any file! Make sure you already moved the files to your s3 bucket.
Specify --source to filter the reference to update to files from a specific in-place directory. If no
--source is given, all in-place imported track references will be updated.
Specify --target to specify a subdirectory in the S3 bucket where you moved the files. If no --target is
given, the file is expected to be stored in the same path as before.
Examples:
Music File: /music/Artist/Album/track.ogg
--source: /music
--target unset
All files imported from /music will be updated and expected to be in the same folder structure in the bucket
Music File: /music/Artist/Album/track.ogg
--source: /music
--target: /in_place
The music file is expected to be stored in the bucket in the directory /in_place/Artist/Album/track.ogg
"""
def create_parser(self, *args, **kwargs):
parser = super().create_parser(*args, **kwargs)
parser.formatter_class = RawTextHelpFormatter
return parser
def add_arguments(self, parser):
parser.add_argument(
"--no-dry-run",
action="store_false",
dest="dry_run",
default=True,
help="Disable dry run mode and apply updates for real on the database",
)
parser.add_argument(
"--source",
type=pathlib.Path,
required=True,
help="Specify the path of the directory where the files originally were stored to update their reference.",
)
parser.add_argument(
"--target",
type=pathlib.Path,
help="Specify a subdirectory in the S3 bucket where you moved the files to.",
)
@transaction.atomic
def handle(self, *args, **options):
if options["dry_run"]:
self.stdout.write("Dry-run on, will not touch the database")
else:
self.stdout.write("Dry-run off, *changing the database*")
self.stdout.write("")
prefix = f"file://{options['source']}"
to_change = models.Upload.objects.filter(source__startswith=prefix)
self.stdout.write(f"Found {to_change.count()} uploads to update.")
target = options.get("target")
if target is None:
target = options["source"]
for upl in to_change:
upl.audio_file = str(upl.source).replace(str(prefix), str(target))
upl.source = None
self.stdout.write(f"Upload expected in {upl.audio_file}")
if not options["dry_run"]:
upl.save()
self.stdout.write("")
if options["dry_run"]:
self.stdout.write(
"Nothing was updated, rerun this command with --no-dry-run to apply the changes"
)
else:
self.stdout.write("Updating completed!")
self.stdout.write("")
import math
import random
from django.conf import settings
from django.core.management.base import BaseCommand
from django.db import transaction
from funkwhale_api.federation import keys
from funkwhale_api.federation import models as federation_models
from funkwhale_api.music import models as music_models
from funkwhale_api.tags import models as tags_models
from funkwhale_api.users import models as users_models
BATCH_SIZE = 500
def create_local_accounts(factories, count, dependencies):
password = factories["users.User"].build().password
users = factories["users.User"].build_batch(size=count)
for user in users:
# we set the hashed password by hand, because computing one for each user
# is CPU intensive
user.password = password
users = users_models.User.objects.bulk_create(users, batch_size=BATCH_SIZE)
actors = []
domain = federation_models.Domain.objects.get_or_create(
name=settings.FEDERATION_HOSTNAME
)[0]
users = [u for u in users if u.pk]
private, public = keys.get_key_pair()
for user in users:
if not user.pk:
continue
actor = federation_models.Actor(
private_key=private.decode("utf-8"),
public_key=public.decode("utf-8"),
**users_models.get_actor_data(user.username, domain=domain)
)
actors.append(actor)
actors = federation_models.Actor.objects.bulk_create(actors, batch_size=BATCH_SIZE)
for user, actor in zip(users, actors):
user.actor = actor
users_models.User.objects.bulk_update(users, ["actor"])
return actors
def create_taggable_items(dependency):
def inner(factories, count, dependencies):
objs = []
tagged_objects = dependencies.get(
dependency, list(CONFIG_BY_ID[dependency]["model"].objects.all().only("pk"))
)
tags = dependencies.get("tags", list(tags_models.Tag.objects.all().only("pk")))
for i in range(count):
tag = random.choice(tags)
tagged_object = random.choice(tagged_objects)
objs.append(
factories["tags.TaggedItem"].build(
content_object=tagged_object, tag=tag
)
)
return tags_models.TaggedItem.objects.bulk_create(
objs, batch_size=BATCH_SIZE, ignore_conflicts=True
)
return inner
CONFIG = [
{
"id": "artist_credit",
"model": music_models.ArtistCredit,
"factory": "music.ArtistCredit",
"factory_kwargs": {"joinphrase": ""},
"depends_on": [
{"field": "artist", "id": "artists", "default_factor": 0.5},
],
},
{
"id": "tracks",
"model": music_models.Track,
"factory": "music.Track",
"factory_kwargs": {"album": None},
"depends_on": [
{"field": "album", "id": "albums", "default_factor": 0.1},
{"field": "artist_credit", "id": "artist_credit", "default_factor": 0.05},
],
},
{
"id": "albums",
"model": music_models.Album,
"factory": "music.Album",
"factory_kwargs": {},
"depends_on": [
{"field": "artist_credit", "id": "artist_credit", "default_factor": 0.3}
],
},
{"id": "artists", "model": music_models.Artist, "factory": "music.Artist"},
{
"id": "local_accounts",
"model": federation_models.Actor,
"handler": create_local_accounts,
},
{
"id": "local_libraries",
"model": music_models.Library,
"factory": "music.Library",
"factory_kwargs": {"actor": None},
"depends_on": [{"field": "actor", "id": "local_accounts", "default_factor": 1}],
},
{
"id": "local_uploads",
"model": music_models.Upload,
"factory": "music.Upload",
"factory_kwargs": {"import_status": "finished", "library": None, "track": None},
"depends_on": [
{
"field": "library",
"id": "local_libraries",
"default_factor": 0.05,
"queryset": music_models.Library.objects.all().select_related(
"actor__user"
),
},
{"field": "track", "id": "tracks", "default_factor": 1},
],
},
{"id": "tags", "model": tags_models.Tag, "factory": "tags.Tag"},
{
"id": "track_tags",
"model": tags_models.TaggedItem,
"queryset": tags_models.TaggedItem.objects.filter(
content_type__app_label="music", content_type__model="track"
),
"handler": create_taggable_items("tracks"),
"depends_on": [
{
"field": "tag",
"id": "tags",
"default_factor": 0.1,
"queryset": tags_models.Tag.objects.all(),
"set": False,
},
{
"field": "content_object",
"id": "tracks",
"default_factor": 1,
"set": False,
},
],
},
{
"id": "album_tags",
"model": tags_models.TaggedItem,
"queryset": tags_models.TaggedItem.objects.filter(
content_type__app_label="music", content_type__model="album"
),
"handler": create_taggable_items("albums"),
"depends_on": [
{
"field": "tag",
"id": "tags",
"default_factor": 0.1,
"queryset": tags_models.Tag.objects.all(),
"set": False,
},
{
"field": "content_object",
"id": "albums",
"default_factor": 1,
"set": False,
},
],
},
{
"id": "artist_tags",
"model": tags_models.TaggedItem,
"queryset": tags_models.TaggedItem.objects.filter(
content_type__app_label="music", content_type__model="artist"
),
"handler": create_taggable_items("artists"),
"depends_on": [
{
"field": "tag",
"id": "tags",
"default_factor": 0.1,
"queryset": tags_models.Tag.objects.all(),
"set": False,
},
{
"field": "content_object",
"id": "artists",
"default_factor": 1,
"set": False,
},
],
},
]
CONFIG_BY_ID = {c["id"]: c for c in CONFIG}
class Rollback(Exception):
pass
def create_objects(row, factories, count, **factory_kwargs):
return factories[row["factory"]].build_batch(size=count, **factory_kwargs)
class Command(BaseCommand):
help = """
Inject demo data into your database. Useful for load testing, or setting up a demo instance.
Use with caution and only if you know what you are doing.
"""
def add_arguments(self, parser):
parser.add_argument(
"--no-dry-run",
action="store_false",
dest="dry_run",
help="Commit the changes to the database",
)
parser.add_argument(
"--create-dependencies", action="store_true", dest="create_dependencies"
)
for row in CONFIG:
parser.add_argument(
"--{}".format(row["id"].replace("_", "-")),
dest=row["id"],
type=int,
help="Number of {} objects to create".format(row["id"]),
)
dependencies = row.get("depends_on", [])
for dependency in dependencies:
parser.add_argument(
"--{}-{}-factor".format(row["id"], dependency["field"]),
dest="{}_{}_factor".format(row["id"], dependency["field"]),
type=float,
help="Number of {} objects to create per {} object".format(
dependency["id"], row["id"]
),
)
def handle(self, *args, **options):
from django.apps import apps
from funkwhale_api import factories
app_names = [app.name for app in apps.app_configs.values()]
factories.registry.autodiscover(app_names)
try:
return self.inner_handle(*args, **options)
except Rollback:
pass
@transaction.atomic
def inner_handle(self, *args, **options):
results = {}
for row in CONFIG:
self.create_batch(row, results, options, count=options.get(row["id"]))
self.stdout.write("\nFinal state of database:\n\n")
for row in CONFIG:
qs = row.get("queryset", row["model"].objects.all())
total = qs.count()
self.stdout.write("- {} {} objects".format(total, row["id"]))
self.stdout.write("")
if options["dry_run"]:
self.stdout.write(
"Run this command with --no-dry-run to commit the changes to the database"
)
raise Rollback()
self.stdout.write(self.style.SUCCESS("Done!"))
def create_batch(self, row, results, options, count):
from funkwhale_api import factories
if row["id"] in results:
# already generated
return results[row["id"]]
if not count:
return []
dependencies = row.get("depends_on", [])
create_dependencies = options.get("create_dependencies")
for dependency in dependencies:
dep_count = options.get(dependency["id"])
if not create_dependencies and dep_count is None:
continue
if dep_count is None:
factor = options[
"{}_{}_factor".format(row["id"], dependency["field"])
] or dependency.get("default_factor")
dep_count = math.ceil(factor * count)
results[dependency["id"]] = self.create_batch(
CONFIG_BY_ID[dependency["id"]], results, options, count=dep_count
)
self.stdout.write("Creating {} {}…".format(count, row["id"]))
handler = row.get("handler")
if handler:
objects = handler(factories.registry, count, dependencies=results)
else:
objects = create_objects(
row, factories.registry, count, **row.get("factory_kwargs", {})
)
for dependency in dependencies:
if not dependency.get("set", True):
continue
if create_dependencies:
candidates = results[dependency["id"]]
else:
# we use existing objects in the database
queryset = dependency.get(
"queryset", CONFIG_BY_ID[dependency["id"]]["model"].objects.all()
)
candidates = list(queryset.values_list("pk", flat=True))
picked_pks = [random.choice(candidates) for _ in objects]
picked_objects = {o.pk: o for o in queryset.filter(pk__in=picked_pks)}
saved_obj = []
for i, obj in enumerate(objects):
if create_dependencies:
value = random.choice(candidates)
else:
value = picked_objects[picked_pks[i]]
if dependency["field"] == "artist_credit":
obj.save()
obj.artist_credit.set([value])
saved_obj.append(obj)
else:
setattr(obj, dependency["field"], value)
if saved_obj:
return saved_obj
if not handler:
objects = row["model"].objects.bulk_create(objects, batch_size=BATCH_SIZE)
results[row["id"]] = objects
return objects
from django.conf import settings
from django.core.management.base import CommandError
from django.core.management.commands.makemigrations import Command as BaseCommand
class Command(BaseCommand):
def handle(self, *apps_label, **options):
"""
Running makemigrations in production can have desastrous consequences.
We ensure the command is disabled, unless a specific env var is provided.
"""
force = settings.FORCE
if not force == 1:
raise CommandError(
"Running makemigrations on your Funkwhale instance can have desastrous"
" consequences. This command is disabled, and should only be run in "
"development environments."
)
return super().handle(*apps_label, **options)
from django.core.management.commands.migrate import Command as BaseCommand
def patch_write(buffer):
"""
Django is trying to help us when running migrate, by checking we don't have
model changes not included in migrations. Unfortunately, running makemigrations
on production instances create unwanted migrations and corrupt the database.
So we disabled the makemigrations command, and we're patching the
write method to ensure misleading messages are never shown to the user,
because https://github.com/django/django/blob/2.1.5/django/core/management/commands/migrate.py#L186
does not leave an easy way to disable them.
"""
unpatched = buffer.write
def p(message, *args, **kwargs):
if "'manage.py makemigrations'" in message or "not yet reflected" in message:
return
return unpatched(message, *args, **kwargs)
setattr(buffer, "write", p)
class Command(BaseCommand):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
patch_write(self.stdout)
......@@ -26,7 +26,7 @@ class Command(BaseCommand):
script = available_scripts[name]
except KeyError:
raise CommandError(
"{} is not a valid script. Run python manage.py script for a "
"{} is not a valid script. Run funkwhale-manage script for a "
"list of available scripts".format(name)
)
......@@ -43,14 +43,14 @@ class Command(BaseCommand):
def show_help(self):
self.stdout.write("")
self.stdout.write("Available scripts:")
self.stdout.write("Launch with: python manage.py <script_name>")
self.stdout.write("Launch with: funkwhale-manage script <script_name>")
available_scripts = self.get_scripts()
for name, script in sorted(available_scripts.items()):
self.stdout.write("")
self.stdout.write(self.style.SUCCESS(name))
self.stdout.write("")
for line in script["help"].splitlines():
self.stdout.write(" {}".format(line))
self.stdout.write(f" {line}")
self.stdout.write("")
def get_scripts(self):
......
from django.core.management.commands.migrate import Command as BaseCommand
from funkwhale_api.federation import factories
from funkwhale_api.federation.models import Actor
class Command(BaseCommand):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.help = "Helper to generate randomized testdata"
self.type_choices = {"notifications": self.handle_notifications}
self.missing_args_message = f"Please specify one of the following sub-commands: {*self.type_choices.keys(), }"
def add_arguments(self, parser):
subparsers = parser.add_subparsers(dest="subcommand")
notification_parser = subparsers.add_parser("notifications")
notification_parser.add_argument(
"username", type=str, help="Username to send the notifications to"
)
notification_parser.add_argument(
"--count", type=int, help="Number of elements to create", default=1
)
def handle(self, *args, **options):
self.type_choices[options["subcommand"]](options)
def handle_notifications(self, options):
self.stdout.write(
f"Create {options['count']} notification(s) for {options['username']}"
)
try:
actor = Actor.objects.get(preferred_username=options["username"])
except Actor.DoesNotExist:
self.stdout.write(
"The user you want to create notifications for does not exist"
)
return
follow_activity = factories.ActivityFactory(type="Follow")
for _ in range(options["count"]):
factories.InboxItemFactory(actor=actor, activity=follow_activity)
import html
import io
import logging
import os
import re
import time
import tracemalloc
import urllib.parse
import xml.sax.saxutils
from django import http, urls
from django.conf import settings
from django.contrib import auth
from django.core.cache import caches
from django.middleware import csrf
from rest_framework import views
from funkwhale_api.federation import utils as federation_utils
from . import preferences, session, throttling, utils
EXCLUDED_PATHS = ["/api", "/federation", "/.well-known"]
logger = logging.getLogger(__name__)
def should_fallback_to_spa(path):
if path == "/":
return True
return not any([path.startswith(m) for m in EXCLUDED_PATHS])
def serve_spa(request):
html = get_spa_html(settings.FUNKWHALE_SPA_HTML_ROOT)
head, tail = html.split("</head>", 1)
if settings.FUNKWHALE_SPA_REWRITE_MANIFEST:
new_url = (
settings.FUNKWHALE_SPA_REWRITE_MANIFEST_URL
or federation_utils.full_url(urls.reverse("api:v1:instance:spa-manifest"))
)
title = preferences.get("instance__name")
if title:
head = replace_title(head, title)
head = replace_manifest_url(head, new_url)
if not preferences.get("common__api_authentication_required"):
try:
request_tags = get_request_head_tags(request) or []
except urls.exceptions.Resolver404:
# we don't have any custom tags for this route
request_tags = []
else:
# API is not open, we don't expose any custom data
request_tags = []
default_tags = get_default_head_tags(request.path)
unique_attributes = ["name", "property"]
final_tags = request_tags
skip = []
for t in final_tags:
for attr in unique_attributes:
if attr in t:
skip.append(t[attr])
for t in default_tags:
existing = False
for attr in unique_attributes:
if t.get(attr) in skip:
existing = True
break
if not existing:
final_tags.append(t)
# let's inject our meta tags in the HTML
head += "\n" + "\n".join(render_tags(final_tags)) + "\n</head>"
css = get_custom_css() or ""
if css:
# We add the style add the end of the body to ensure it has the highest
# priority (since it will come after other stylesheets)
body, tail = tail.split("</body>", 1)
css = f"<style>{css}</style>"
tail = body + "\n" + css + "\n</body>" + tail
# set a csrf token so that visitor can login / query API if needed
token = csrf.get_token(request)
response = http.HttpResponse(head + tail)
response.set_cookie("csrftoken", token, max_age=None)
return response
MANIFEST_LINK_REGEX = re.compile(r"<link [^>]*rel=(?:'|\")?manifest(?:'|\")?[^>]*>")
TITLE_REGEX = re.compile(r"<title>.*</title>")
def replace_manifest_url(head, new_url):
replacement = f'<link rel=manifest href="{new_url}">'
head = MANIFEST_LINK_REGEX.sub(replacement, head)
return head
def replace_title(head, new_title):
replacement = f"<title>{html.escape(new_title)}</title>"
head = TITLE_REGEX.sub(replacement, head)
return head
def get_spa_html(spa_url):
return get_spa_file(spa_url, "index.html")
def get_spa_file(spa_url, name):
if spa_url.startswith("/"):
# spa_url is an absolute path to index.html, on the local disk.
# However, we may want to access manifest.json or other files as well, so we
# strip the filename
path = os.path.join(os.path.dirname(spa_url), name)
# we try to open a local file
with open(path, "rb") as f:
return f.read().decode("utf-8")
cache_key = f"spa-file:{spa_url}:{name}"
cached = caches["local"].get(cache_key)
if cached:
return cached
response = session.get_session().get(
utils.join_url(spa_url, name),
)
response.raise_for_status()
response.encoding = "utf-8"
content = response.text
caches["local"].set(cache_key, content, settings.FUNKWHALE_SPA_HTML_CACHE_DURATION)
return content
def get_default_head_tags(path):
instance_name = preferences.get("instance__name")
short_description = preferences.get("instance__short_description")
app_name = settings.APP_NAME
parts = [instance_name, app_name]
return [
{"tag": "meta", "property": "og:type", "content": "website"},
{
"tag": "meta",
"property": "og:site_name",
"content": " - ".join([p for p in parts if p]),
},
{"tag": "meta", "property": "og:description", "content": short_description},
{
"tag": "meta",
"property": "og:image",
"content": utils.join_url(
settings.FUNKWHALE_URL, "/android-chrome-512x512.png"
),
},
{
"tag": "meta",
"property": "og:url",
"content": utils.join_url(settings.FUNKWHALE_URL, path),
},
]
def render_tags(tags):
"""
Given a dict like {'tag': 'meta', 'hello': 'world'}
return a html ready tag like
<meta hello="world" />
"""
for tag in tags:
yield "<{tag} {attrs} />".format(
tag=tag.pop("tag"),
attrs=" ".join(
[f'{a}="{html.escape(str(v))}"' for a, v in sorted(tag.items()) if v]
),
)
def get_request_head_tags(request):
accept_header = request.headers.get("Accept") or None
redirect_to_ap = (
False
if not accept_header
else not federation_utils.should_redirect_ap_to_html(accept_header)
)
match = urls.resolve(request.path, urlconf=settings.SPA_URLCONF)
return match.func(
request, *match.args, redirect_to_ap=redirect_to_ap, **match.kwargs
)
def get_custom_css():
css = preferences.get("ui__custom_css").strip()
if not css:
return
return xml.sax.saxutils.escape(css)
class ApiRedirect(Exception):
def __init__(self, url):
self.url = url
def get_api_response(request, url):
"""
Quite ugly but we have no choice. When Accept header is set to application/activity+json
some clients expect to get a JSON payload (instead of the HTML we return). Since
redirecting to the URL does not work (because it makes the signature verification fail),
we grab the internal view corresponding to the URL, call it and return this as the
response
"""
path = urllib.parse.urlparse(url).path
try:
match = urls.resolve(path)
except urls.exceptions.Resolver404:
return http.HttpResponseNotFound()
response = match.func(request, *match.args, **match.kwargs)
if hasattr(response, "render"):
response.render()
return response
class SPAFallbackMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
response = self.get_response(request)
if response.status_code == 404 and should_fallback_to_spa(request.path):
try:
return serve_spa(request)
except ApiRedirect as e:
return get_api_response(request, e.url)
return response
class DevHttpsMiddleware:
"""
In development, it's sometimes difficult to have django use HTTPS
when we have django behind nginx behind traefix.
We thus use a simple setting (in dev ONLY) to control that.
"""
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if settings.FORCE_HTTPS_URLS:
setattr(request.__class__, "scheme", "https")
setattr(
request,
"get_host",
lambda: request.__class__.get_host(request).replace(":80", ":443"),
)
return self.get_response(request)
def monkey_patch_rest_initialize_request():
"""
Rest framework use it's own APIRequest, meaning we can't easily
access our throttling info in the middleware. So me monkey patch the
`initialize_request` method from rest_framework to keep a link between both requests
"""
original = views.APIView.initialize_request
def replacement(self, request, *args, **kwargs):
r = original(self, request, *args, **kwargs)
setattr(request, "_api_request", r)
return r
setattr(views.APIView, "initialize_request", replacement)
monkey_patch_rest_initialize_request()
def monkey_patch_auth_get_user():
"""
We need an actor on our users for many endpoints, so we monkey patch
auth.get_user to create it if it's missing
"""
original = auth.get_user
def replacement(request):
r = original(request)
if not r.is_anonymous and not r.actor:
r.create_actor()
return r
setattr(auth, "get_user", replacement)
monkey_patch_auth_get_user()
class ThrottleStatusMiddleware:
"""
Include useful information regarding throttling in API responses to
ensure clients can adapt.
"""
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
try:
response = self.get_response(request)
except throttling.TooManyRequests:
# manual throttling in non rest_framework view, we have to return
# the proper response ourselves
response = http.HttpResponse(status=429)
request_to_check = request
try:
request_to_check = request._api_request
except AttributeError:
pass
throttle_status = getattr(request_to_check, "_throttle_status", None)
if throttle_status:
response["X-RateLimit-Limit"] = str(throttle_status["num_requests"])
response["X-RateLimit-Scope"] = str(throttle_status["scope"])
response["X-RateLimit-Remaining"] = throttle_status["num_requests"] - len(
throttle_status["history"]
)
response["X-RateLimit-Duration"] = str(throttle_status["duration"])
if throttle_status["history"]:
now = int(time.time())
# At this point, the client can send additional requests
oldtest_request = throttle_status["history"][-1]
remaining = throttle_status["duration"] - (now - int(oldtest_request))
response["Retry-After"] = str(remaining)
# At this point, all Rate Limit is reset to 0
latest_request = throttle_status["history"][0]
remaining = throttle_status["duration"] - (now - int(latest_request))
response["X-RateLimit-Reset"] = str(now + remaining)
response["X-RateLimit-ResetSeconds"] = str(remaining)
return response
class VerboseBadRequestsMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
response = self.get_response(request)
if response.status_code == 400:
logger.warning("Bad request: %s", response.content)
return response
class ProfilerMiddleware:
"""
from https://github.com/omarish/django-cprofile-middleware/blob/master/django_cprofile_middleware/middleware.py
Simple profile middleware to profile django views. To run it, add ?prof to
the URL like this:
http://localhost:8000/view/?prof
Optionally pass the following to modify the output:
?sort => Sort the output by a given metric. Default is time.
See
http://docs.python.org/2/library/profile.html#pstats.Stats.sort_stats
for all sort options.
?count => The number of rows to display. Default is 100.
?download => Download profile file suitable for visualization. For example
in snakeviz or RunSnakeRun
This is adapted from an example found here:
http://www.slideshare.net/zeeg/django-con-high-performance-django-presentation.
"""
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if "prof" not in request.GET:
return self.get_response(request)
import profile
import pstats
profiler = profile.Profile()
response = profiler.runcall(self.get_response, request)
profiler.create_stats()
if "prof-download" in request.GET:
import marshal
output = marshal.dumps(profiler.stats)
response = http.HttpResponse(
output, content_type="application/octet-stream"
)
response["Content-Disposition"] = "attachment; filename=view.prof"
response["Content-Length"] = len(output)
stream = io.StringIO()
stats = pstats.Stats(profiler, stream=stream)
stats.sort_stats(request.GET.get("prof-sort", "cumtime"))
stats.print_stats(int(request.GET.get("count", 100)))
response = http.HttpResponse("<pre>%s</pre>" % stream.getvalue())
return response
class PymallocMiddleware:
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
if tracemalloc.is_tracing():
snapshot = tracemalloc.take_snapshot()
stats = snapshot.statistics("lineno")
print("Memory trace")
for stat in stats[:25]:
print(stat)
return self.get_response(request)
......@@ -3,8 +3,20 @@ from django.db import migrations
from django.contrib.postgres.operations import UnaccentExtension
class CustomUnaccentExtension(UnaccentExtension):
def database_forwards(self, app_label, schema_editor, from_state, to_state):
check_sql = "SELECT 1 FROM pg_extension WHERE extname = 'unaccent'"
with schema_editor.connection.cursor() as cursor:
cursor.execute(check_sql)
result = cursor.fetchall()
if result:
return
return super().database_forwards(app_label, schema_editor, from_state, to_state)
class Migration(migrations.Migration):
dependencies = []
operations = [UnaccentExtension()]
operations = [CustomUnaccentExtension()]
# Generated by Django 2.1.5 on 2019-01-31 15:44
import django.contrib.postgres.fields.jsonb
from django.db import migrations, models
import django.db.models.deletion
import django.utils.timezone
import uuid
class Migration(migrations.Migration):
initial = True
dependencies = [
("federation", "0017_auto_20190130_0926"),
("contenttypes", "0002_remove_content_type_name"),
("common", "0001_initial"),
]
operations = [
migrations.CreateModel(
name="Mutation",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("fid", models.URLField(db_index=True, max_length=500, unique=True)),
(
"uuid",
models.UUIDField(db_index=True, default=uuid.uuid4, unique=True),
),
("type", models.CharField(db_index=True, max_length=100)),
("is_approved", models.BooleanField(default=None, null=True)),
("is_applied", models.BooleanField(default=None, null=True)),
(
"creation_date",
models.DateTimeField(
db_index=True, default=django.utils.timezone.now
),
),
(
"applied_date",
models.DateTimeField(blank=True, db_index=True, null=True),
),
("summary", models.TextField(max_length=2000, blank=True, null=True)),
("payload", django.contrib.postgres.fields.jsonb.JSONField()),
(
"previous_state",
django.contrib.postgres.fields.jsonb.JSONField(
null=True, default=None
),
),
("target_id", models.IntegerField(null=True)),
(
"approved_by",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="approved_mutations",
to="federation.Actor",
),
),
(
"created_by",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="created_mutations",
to="federation.Actor",
),
),
(
"target_content_type",
models.ForeignKey(
null=True,
on_delete=django.db.models.deletion.CASCADE,
related_name="targeting_mutations",
to="contenttypes.ContentType",
),
),
],
)
]
# Generated by Django 2.0.2 on 2018-02-27 18:43
from django.db import migrations
from django.contrib.postgres.operations import CITextExtension
class CustomCITExtension(CITextExtension):
def database_forwards(self, app_label, schema_editor, from_state, to_state):
check_sql = "SELECT 1 FROM pg_extension WHERE extname = 'citext'"
with schema_editor.connection.cursor() as cursor:
cursor.execute(check_sql)
result = cursor.fetchall()
if result:
return
return super().database_forwards(app_label, schema_editor, from_state, to_state)
class Migration(migrations.Migration):
dependencies = [("common", "0002_mutation")]
operations = [CustomCITExtension()]
# Generated by Django 2.2.6 on 2019-11-11 13:38
import django.contrib.postgres.fields.jsonb
import django.core.serializers.json
from django.db import migrations, models
import django.db.models.deletion
import django.utils.timezone
import funkwhale_api.common.models
import funkwhale_api.common.validators
import uuid
import versatileimagefield.fields
class Migration(migrations.Migration):
dependencies = [
('common', '0003_cit_extension'),
]
operations = [
migrations.CreateModel(
name='Attachment',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('url', models.URLField(max_length=500, unique=True, null=True)),
('uuid', models.UUIDField(db_index=True, default=uuid.uuid4, unique=True)),
('creation_date', models.DateTimeField(default=django.utils.timezone.now)),
('last_fetch_date', models.DateTimeField(blank=True, null=True)),
('size', models.IntegerField(blank=True, null=True)),
('mimetype', models.CharField(blank=True, max_length=200, null=True)),
('file', versatileimagefield.fields.VersatileImageField(max_length=255, upload_to=funkwhale_api.common.models.get_file_path, validators=[funkwhale_api.common.validators.ImageDimensionsValidator(min_height=50, min_width=50), funkwhale_api.common.validators.FileValidator(allowed_extensions=['png', 'jpg', 'jpeg'], max_size=5242880)])),
('actor', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='attachments', to='federation.Actor', null=True)),
],
),
]
# Generated by Django 2.2.7 on 2019-11-25 14:21
import django.contrib.postgres.fields.jsonb
import django.core.serializers.json
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('common', '0004_auto_20191111_1338'),
]
operations = [
migrations.AlterField(
model_name='mutation',
name='payload',
field=django.contrib.postgres.fields.jsonb.JSONField(encoder=django.core.serializers.json.DjangoJSONEncoder),
),
migrations.AlterField(
model_name='mutation',
name='previous_state',
field=django.contrib.postgres.fields.jsonb.JSONField(default=None, encoder=django.core.serializers.json.DjangoJSONEncoder, null=True),
),
migrations.CreateModel(
name='MutationAttachment',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('attachment', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='mutation_attachment', to='common.Attachment')),
('mutation', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='mutation_attachment', to='common.Mutation')),
],
options={
'unique_together': {('attachment', 'mutation')},
},
),
]
# Generated by Django 2.2.7 on 2020-01-13 10:14
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('common', '0005_auto_20191125_1421'),
]
operations = [
migrations.CreateModel(
name='Content',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('text', models.CharField(blank=True, max_length=5000, null=True)),
('content_type', models.CharField(max_length=100)),
],
),
]
# Generated by Django 2.2.9 on 2020-01-16 16:10
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('common', '0006_content'),
]
operations = [
migrations.AlterField(
model_name='attachment',
name='url',
field=models.URLField(max_length=500, null=True),
),
]
# Generated by Django 3.0.8 on 2020-07-01 13:17
from django.conf import settings
import django.contrib.postgres.fields.jsonb
from django.db import migrations, models
import django.db.models.deletion
import django.utils.timezone
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('common', '0007_auto_20200116_1610'),
]
operations = [
migrations.AlterField(
model_name='attachment',
name='url',
field=models.URLField(blank=True, max_length=500, null=True),
),
migrations.CreateModel(
name='PluginConfiguration',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('code', models.CharField(max_length=100)),
('conf', django.contrib.postgres.fields.jsonb.JSONField(blank=True, null=True)),
('enabled', models.BooleanField(default=False)),
('creation_date', models.DateTimeField(default=django.utils.timezone.now)),
('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='plugins', to=settings.AUTH_USER_MODEL)),
],
options={
'unique_together': {('user', 'code')},
},
),
]
# Generated by Django 3.2.13 on 2022-06-27 19:15
import django.core.serializers.json
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('common', '0008_auto_20200701_1317'),
]
operations = [
migrations.AlterField(
model_name='mutation',
name='is_applied',
field=models.BooleanField(default=None, null=True),
),
migrations.AlterField(
model_name='mutation',
name='is_approved',
field=models.BooleanField(default=None, null=True),
),
migrations.AlterField(
model_name='mutation',
name='payload',
field=models.JSONField(encoder=django.core.serializers.json.DjangoJSONEncoder),
),
migrations.AlterField(
model_name='mutation',
name='previous_state',
field=models.JSONField(default=None, encoder=django.core.serializers.json.DjangoJSONEncoder, null=True),
),
migrations.AlterField(
model_name='pluginconfiguration',
name='conf',
field=models.JSONField(blank=True, null=True),
),
]
from django.db.models import Q
from django.shortcuts import get_object_or_404
from rest_framework import serializers
class MultipleLookupDetailMixin:
lookup_value_regex = "[^/]+"
lookup_field = "composite"
def get_object(self):
queryset = self.filter_queryset(self.get_queryset())
relevant_lookup = None
value = None
for lookup in self.url_lookups:
field_validator = lookup["validator"]
try:
value = field_validator(self.kwargs["composite"])
except serializers.ValidationError:
continue
else:
relevant_lookup = lookup
break
get_query = relevant_lookup.get(
"get_query", lambda value: Q(**{relevant_lookup["lookup_field"]: value})
)
query = get_query(value)
obj = get_object_or_404(queryset, query)
# May raise a permission denied
self.check_object_permissions(self.request, obj)
return obj
import mimetypes
import uuid
import magic
from django.conf import settings
from django.contrib.contenttypes.fields import GenericForeignKey
from django.contrib.contenttypes.models import ContentType
from django.core.serializers.json import DjangoJSONEncoder
from django.db import connections, models, transaction
from django.db.models import JSONField, Lookup
from django.db.models.fields import Field
from django.db.models.sql.compiler import SQLCompiler
from django.dispatch import receiver
from django.urls import reverse
from django.utils import timezone
from versatileimagefield.fields import VersatileImageField
from versatileimagefield.image_warmer import VersatileImageFieldWarmer
from funkwhale_api.federation import utils as federation_utils
from . import utils, validators
CONTENT_TEXT_MAX_LENGTH = 5000
CONTENT_TEXT_SUPPORTED_TYPES = [
"text/html",
"text/markdown",
"text/plain",
]
@Field.register_lookup
class NotEqual(Lookup):
lookup_name = "ne"
def as_sql(self, compiler, connection):
lhs, lhs_params = self.process_lhs(compiler, connection)
rhs, rhs_params = self.process_rhs(compiler, connection)
params = lhs_params + rhs_params
return f"{lhs} <> {rhs}", params
class NullsLastSQLCompiler(SQLCompiler):
def get_order_by(self):
result = super().get_order_by()
if result and self.connection.vendor == "postgresql":
return [
(
expr,
(
sql + " NULLS LAST" if not sql.endswith(" NULLS LAST") else sql,
params,
is_ref,
),
)
for (expr, (sql, params, is_ref)) in result
]
return result
class NullsLastQuery(models.sql.query.Query):
"""Use a custom compiler to inject 'NULLS LAST' (for PostgreSQL)."""
def get_compiler(self, using=None, connection=None, elide_empty=True):
if using is None and connection is None:
raise ValueError("Need either using or connection")
if using:
connection = connections[using]
return NullsLastSQLCompiler(self, connection, using, elide_empty)
class NullsLastQuerySet(models.QuerySet):
def __init__(self, model=None, query=None, using=None, hints=None):
super().__init__(model, query, using, hints)
self.query = query or NullsLastQuery(self.model)
class LocalFromFidQuerySet:
def local(self, include=True):
host = settings.FEDERATION_HOSTNAME
query = models.Q(fid__startswith=f"http://{host}/") | models.Q(
fid__startswith=f"https://{host}/"
)
if include:
return self.filter(query)
else:
return self.filter(~query)
class GenericTargetQuerySet(models.QuerySet):
def get_for_target(self, target):
content_type = ContentType.objects.get_for_model(target)
return self.filter(target_content_type=content_type, target_id=target.pk)
class Mutation(models.Model):
fid = models.URLField(unique=True, max_length=500, db_index=True)
uuid = models.UUIDField(unique=True, db_index=True, default=uuid.uuid4)
created_by = models.ForeignKey(
"federation.Actor",
related_name="created_mutations",
on_delete=models.SET_NULL,
null=True,
blank=True,
)
approved_by = models.ForeignKey(
"federation.Actor",
related_name="approved_mutations",
on_delete=models.SET_NULL,
null=True,
blank=True,
)
type = models.CharField(max_length=100, db_index=True)
# None = no choice, True = approved, False = refused
is_approved = models.BooleanField(default=None, null=True)
# None = not applied, True = applied, False = failed
is_applied = models.BooleanField(default=None, null=True)
creation_date = models.DateTimeField(default=timezone.now, db_index=True)
applied_date = models.DateTimeField(null=True, blank=True, db_index=True)
summary = models.TextField(max_length=2000, null=True, blank=True)
payload = JSONField(encoder=DjangoJSONEncoder)
previous_state = JSONField(null=True, default=None, encoder=DjangoJSONEncoder)
target_id = models.IntegerField(null=True)
target_content_type = models.ForeignKey(
ContentType,
null=True,
on_delete=models.CASCADE,
related_name="targeting_mutations",
)
target = GenericForeignKey("target_content_type", "target_id")
objects = GenericTargetQuerySet.as_manager()
def get_federation_id(self):
if self.fid:
return self.fid
return federation_utils.full_url(
reverse("federation:edits-detail", kwargs={"uuid": self.uuid})
)
def save(self, **kwargs):
if not self.pk and not self.fid:
self.fid = self.get_federation_id()
return super().save(**kwargs)
@transaction.atomic
def apply(self):
from . import mutations
if self.is_applied:
raise ValueError("Mutation was already applied")
previous_state = mutations.registry.apply(
type=self.type, obj=self.target, payload=self.payload
)
self.previous_state = previous_state
self.is_applied = True
self.applied_date = timezone.now()
self.save(update_fields=["is_applied", "applied_date", "previous_state"])
return previous_state
def get_file_path(instance, filename):
return utils.ChunkedPath("attachments")(instance, filename)
class AttachmentQuerySet(models.QuerySet):
def attached(self, include=True):
related_fields = [
"covered_album",
"mutation_attachment",
"covered_track",
"covered_artist",
"iconed_actor",
]
query = None
for field in related_fields:
field_query = ~models.Q(**{field: None})
query = query | field_query if query else field_query
if not include:
query = ~query
return self.filter(query)
def local(self, include=True):
if include:
return self.filter(actor__domain_id=settings.FEDERATION_HOSTNAME)
else:
return self.exclude(actor__domain_id=settings.FEDERATION_HOSTNAME)
class Attachment(models.Model):
# Remote URL where the attachment can be fetched
url = models.URLField(max_length=500, null=True, blank=True)
uuid = models.UUIDField(unique=True, db_index=True, default=uuid.uuid4)
# Actor associated with the attachment
actor = models.ForeignKey(
"federation.Actor",
related_name="attachments",
on_delete=models.CASCADE,
null=True,
)
creation_date = models.DateTimeField(default=timezone.now)
last_fetch_date = models.DateTimeField(null=True, blank=True)
# File size
size = models.IntegerField(null=True, blank=True)
mimetype = models.CharField(null=True, blank=True, max_length=200)
file = VersatileImageField(
upload_to=get_file_path,
max_length=255,
validators=[
validators.ImageDimensionsValidator(min_width=50, min_height=50),
validators.FileValidator(
allowed_extensions=["png", "jpg", "jpeg"],
max_size=1024 * 1024 * 5,
),
],
)
objects = AttachmentQuerySet.as_manager()
def save(self, **kwargs):
if self.file and not self.size:
self.size = self.file.size
if self.file and not self.mimetype:
self.mimetype = self.guess_mimetype()
return super().save()
@property
def is_local(self):
return federation_utils.is_local(self.fid)
def guess_mimetype(self):
f = self.file
b = min(1000000, f.size)
t = magic.from_buffer(f.read(b), mime=True)
if not t.startswith("image/"):
# failure, we try guessing by extension
mt, _ = mimetypes.guess_type(f.name)
if mt:
t = mt
return t
@property
def download_url_original(self):
if self.file:
return utils.media_url(self.file.url)
proxy_url = reverse("api:v1:attachments-proxy", kwargs={"uuid": self.uuid})
return federation_utils.full_url(proxy_url + "?next=original")
@property
def download_url_small_square_crop(self):
if self.file:
return utils.media_url(self.file.crop["50x50"].url)
proxy_url = reverse("api:v1:attachments-proxy", kwargs={"uuid": self.uuid})
return federation_utils.full_url(proxy_url + "?next=small_square_crop")
@property
def download_url_medium_square_crop(self):
if self.file:
return utils.media_url(self.file.crop["200x200"].url)
proxy_url = reverse("api:v1:attachments-proxy", kwargs={"uuid": self.uuid})
return federation_utils.full_url(proxy_url + "?next=medium_square_crop")
@property
def download_url_large_square_crop(self):
if self.file:
return utils.media_url(self.file.crop["600x600"].url)
proxy_url = reverse("api:v1:attachments-proxy", kwargs={"uuid": self.uuid})
return federation_utils.full_url(proxy_url + "?next=large_square_crop")
class MutationAttachment(models.Model):
"""
When using attachments in mutations, we need to keep a reference to
the attachment to ensure it is not pruned by common/tasks.py.
This is what this model does.
"""
attachment = models.OneToOneField(
Attachment, related_name="mutation_attachment", on_delete=models.CASCADE
)
mutation = models.OneToOneField(
Mutation, related_name="mutation_attachment", on_delete=models.CASCADE
)
class Meta:
unique_together = ("attachment", "mutation")
class Content(models.Model):
"""
A text content that can be associated to other models, like a description, a summary, etc.
"""
text = models.CharField(max_length=CONTENT_TEXT_MAX_LENGTH, blank=True, null=True)
content_type = models.CharField(max_length=100)
@property
def rendered(self):
from . import utils
return utils.render_html(self.text, self.content_type)
@property
def as_plain_text(self):
from . import utils
return utils.render_plain_text(self.rendered)
def truncate(self, length):
text = self.as_plain_text
truncated = text[:length]
if len(truncated) < len(text):
truncated += ""
return truncated
@receiver(models.signals.post_save, sender=Attachment)
def warm_attachment_thumbnails(sender, instance, **kwargs):
if not instance.file or not settings.CREATE_IMAGE_THUMBNAILS:
return
warmer = VersatileImageFieldWarmer(
instance_or_queryset=instance,
rendition_key_set="attachment_square",
image_attr="file",
)
num_created, failed_to_create = warmer.warm()
@receiver(models.signals.post_save, sender=Mutation)
def trigger_mutation_post_init(sender, instance, created, **kwargs):
if not created:
return
from . import mutations
try:
conf = mutations.registry.get_conf(instance.type, instance.target)
except mutations.ConfNotFound:
return
serializer = conf["serializer_class"]()
try:
handler = serializer.mutation_post_init
except AttributeError:
return
handler(instance)
CONTENT_FKS = {
"music.Track": ["description"],
"music.Album": ["description"],
"music.Artist": ["description"],
}
@receiver(models.signals.post_delete, sender=None)
def remove_attached_content(sender, instance, **kwargs):
fk_fields = CONTENT_FKS.get(instance._meta.label, [])
for field in fk_fields:
if getattr(instance, f"{field}_id"):
try:
getattr(instance, field).delete()
except Content.DoesNotExist:
pass
class PluginConfiguration(models.Model):
"""
Store plugin configuration in DB
"""
code = models.CharField(max_length=100)
user = models.ForeignKey(
"users.User",
related_name="plugins",
on_delete=models.CASCADE,
null=True,
blank=True,
)
conf = JSONField(null=True, blank=True)
enabled = models.BooleanField(default=False)
creation_date = models.DateTimeField(default=timezone.now)
class Meta:
unique_together = ("user", "code")
import persisting_theory
from django.db import models, transaction
from rest_framework import serializers
class ConfNotFound(KeyError):
pass
class Registry(persisting_theory.Registry):
look_into = "mutations"
def connect(self, type, klass, perm_checkers=None):
def decorator(serializer_class):
t = self.setdefault(type, {})
t[klass] = {
"serializer_class": serializer_class,
"perm_checkers": perm_checkers or {},
}
return serializer_class
return decorator
@transaction.atomic
def apply(self, type, obj, payload):
conf = self.get_conf(type, obj)
serializer = conf["serializer_class"](obj, data=payload)
serializer.is_valid(raise_exception=True)
previous_state = serializer.get_previous_state(obj, serializer.validated_data)
serializer.apply(obj, serializer.validated_data)
return previous_state
def is_valid(self, type, obj, payload):
conf = self.get_conf(type, obj)
serializer = conf["serializer_class"](obj, data=payload)
return serializer.is_valid(raise_exception=True)
def get_validated_payload(self, type, obj, payload):
conf = self.get_conf(type, obj)
serializer = conf["serializer_class"](obj, data=payload)
serializer.is_valid(raise_exception=True)
return serializer.payload_serialize(serializer.validated_data)
def has_perm(self, perm, type, obj, actor):
if perm not in ["approve", "suggest"]:
raise ValueError(f"Invalid permission {perm}")
conf = self.get_conf(type, obj)
checker = conf["perm_checkers"].get(perm)
if not checker:
return False
return checker(obj=obj, actor=actor)
def get_conf(self, type, obj):
try:
type_conf = self[type]
except KeyError:
raise ConfNotFound(f"{type} is not a registered mutation")
try:
conf = type_conf[obj.__class__]
except KeyError:
try:
conf = type_conf[None]
except KeyError:
raise ConfNotFound(
f"No mutation configuration found for {obj.__class__}"
)
return conf
class MutationSerializer(serializers.Serializer):
def apply(self, obj, validated_data):
raise NotImplementedError()
def post_apply(self, obj, validated_data):
pass
def get_previous_state(self, obj, validated_data):
return
def payload_serialize(self, data):
return data
class UpdateMutationSerializer(serializers.ModelSerializer, MutationSerializer):
def __init__(self, *args, **kwargs):
# we force partial mode, because update mutations are partial
kwargs.setdefault("partial", True)
super().__init__(*args, **kwargs)
@transaction.atomic
def apply(self, obj, validated_data):
r = self.update(obj, validated_data)
self.post_apply(r, validated_data)
return r
def validate(self, validated_data):
if not validated_data:
raise serializers.ValidationError("You must update at least one field")
return super().validate(validated_data)
def db_serialize(self, validated_data):
serialized_relations = self.get_serialized_relations()
data = {}
# ensure model fields are serialized properly
for key, value in list(validated_data.items()):
if not isinstance(value, models.Model):
data[key] = value
continue
field = serialized_relations[key]
data[key] = getattr(value, field)
return data
def payload_serialize(self, data):
data = super().payload_serialize(data)
# we use our serialized_relations configuration
# to ensure we store ids instead of model instances in our json
# payload
for field, attr in self.get_serialized_relations().items():
try:
obj = data[field]
except KeyError:
continue
if obj is None:
data[field] = None
else:
data[field] = getattr(obj, attr)
return data
def create(self, validated_data):
validated_data = self.db_serialize(validated_data)
return super().create(validated_data)
def get_previous_state(self, obj, validated_data):
return get_update_previous_state(
obj,
*list(validated_data.keys()),
serialized_relations=self.get_serialized_relations(),
handlers=self.get_previous_state_handlers(),
)
def get_serialized_relations(self):
return {}
def get_previous_state_handlers(self):
return {}
def get_update_previous_state(obj, *fields, serialized_relations={}, handlers={}):
if not fields:
raise ValueError("You need to provide at least one field")
state = {}
for field in fields:
if field in handlers:
state[field] = handlers[field](obj)
continue
value = getattr(obj, field)
if isinstance(value, models.Model):
# we store the related object id and repr for better UX
id_field = serialized_relations[field]
related_value = getattr(value, id_field)
state[field] = {"value": related_value, "repr": str(value)}
else:
state[field] = {"value": value}
return state
registry = Registry()
from rest_framework.pagination import PageNumberPagination
from rest_framework.pagination import PageNumberPagination, _positive_int
class FunkwhalePagination(PageNumberPagination):
page_size_query_param = "page_size"
max_page_size = 50
default_max_page_size = 50
default_page_size = None
view = None
def paginate_queryset(self, queryset, request, view=None):
self.view = view
return super().paginate_queryset(queryset, request, view)
def get_page_size(self, request):
max_page_size = (
getattr(self.view, "max_page_size", 0) or self.default_max_page_size
)
page_size = getattr(self.view, "default_page_size", 0) or max_page_size
if self.page_size_query_param:
try:
return _positive_int(
request.query_params[self.page_size_query_param],
strict=True,
cutoff=max_page_size,
)
except (KeyError, ValueError):
pass
return page_size