Skip to content
GitLab
Projects
Groups
Snippets
/
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in / Register
Toggle navigation
Menu
Open sidebar
Auri
funkwhale
Commits
d44b8627
Verified
Commit
d44b8627
authored
Apr 17, 2018
by
Eliot Berriot
Browse files
Merge branch 'release/0.9'
parents
78d0de0e
dd97a9b4
Changes
142
Hide whitespace changes
Inline
Side-by-side
.env.dev
View file @
d44b8627
API_AUTHENTICATION_REQUIRED=True
RAVEN_ENABLED=false
RAVEN_DSN=https://44332e9fdd3d42879c7d35bf8562c6a4:0062dc16a22b41679cd5765e5342f716@sentry.eliotberriot.com/5
DJANGO_ALLOWED_HOSTS=
localhost,nginx
DJANGO_ALLOWED_HOSTS=
.funkwhale.test,localhost,nginx,0.0.0.0,127.0.0.1
DJANGO_SETTINGS_MODULE=config.settings.local
DJANGO_SECRET_KEY=dev
C_FORCE_ROOT=true
FUNKWHALE_URL=http://localhost
FUNKWHALE_HOSTNAME=localhost
FUNKWHALE_PROTOCOL=http
PYTHONDONTWRITEBYTECODE=true
WEBPACK_DEVSERVER_PORT=8080
.gitignore
View file @
d44b8627
...
...
@@ -35,7 +35,6 @@ htmlcov
# Translations
*.mo
*.pot
# Pycharm
.idea
...
...
@@ -75,6 +74,7 @@ api/static
api/.pytest_cache
# Front
front/static/translations
front/node_modules/
front/dist/
front/npm-debug.log*
...
...
@@ -87,3 +87,5 @@ docs/_build
data/
.env
po/*.po
.gitlab-ci.yml
View file @
d44b8627
...
...
@@ -68,6 +68,8 @@ build_front:
script
:
-
yarn install
-
yarn run i18n-extract
-
yarn run i18n-compile
-
yarn run build
cache
:
key
:
"
$CI_PROJECT_ID__front_dependencies"
...
...
CHANGELOG
View file @
d44b8627
...
...
@@ -3,6 +3,79 @@ Changelog
.. towncrier
0.9 (2018-04-17)
----------------
Features:
- Add internationalization support (#5)
- Can now follow and import music from remote libraries (#136, #137)
Enhancements:
- Added a i18n-extract yarn script to extract strings to PO files (#162)
- User admin now includes signup and last login dates (#148)
- We now use a proper user agent including instance version and url during
outgoing requests
Federation is here!
^^^^^^^^^^^^^^^^^^^
This is for real this time, and includes:
- Following other Funkwhale libraries
- Importing tracks from remote libraries (tracks are hotlinked, and only cached for a short amount of time)
- Searching accross federated catalogs
Note that by default, federation is opt-in, on a per-instance basis:
instances will request access to your catalog, and you can accept or refuse
those requests. You can also revoke the access at any time.
Documentation was updated with relevant instructions to use and benefit
from this new feature: https://docs.funkwhale.audio/federation.html
Preparing internationalization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Funkwhale's front-end as always been english-only, and this is a barrier
to new users. The work make Funkwhale's interface translatable was started
in this release by Baptiste. Although nothing is translated yet,
this release includes behind the stage changes that will make it possible in
the near future.
Many thank to Baptiste for the hard work and for figuring out a proper solution
to this difficult problem.
Upgrade path
^^^^^^^^^^^^
In addition to the usual instructions from
https://docs.funkwhale.audio/upgrading.html, non-docker users will have
to setup an additional systemd unit file for recurrent tasks.
This was forgotten in the deployment documentation, but recurrent tasks,
managed by the celery beat process, will be needed more and more in subsequent
releases. Right now, we'll be using to clear the cache for federated music files
and keep disk usage to a minimum.
In the future, they will also be needed to refetch music metadata or federated
information periodically.
Celery beat can be enabled easily::
curl -L -o "/etc/systemd/system/funkwhale-beat.service" "https://code.eliotberriot.com/funkwhale/funkwhale/raw/develop/deploy/funkwhale-beat.service"
# Also edit /etc/systemd/system/funkwhale.target
# and ensure the Wants= line contains the following:
# Wants=funkwhale-server.service funkwhale-worker.service funkwhale-beat.service
nano /etc/systemd/system/funkwhale.target
# reload configuration
systemctl daemon-reload
Docker users already have celerybeat enabled.
0.8 (2018-04-02)
----------------
...
...
@@ -71,27 +144,16 @@ and add the following snippets::
This will ensure federation endpoints will be reachable in the future.
You can of course skip this part if you know you will not federate your instance.
A new ``FEDERATION_ENABLED`` env var have also been added to control wether
A new ``FEDERATION_ENABLED`` env var have also been added to control w
h
ether
federation is enabled or not on the application side. This settings defaults
to True, which should have no consequenc
i
es at the moment, since actual
to True, which should have no consequences at the moment, since actual
federation is not implemented and the only available endpoints are for
testing purposes.
Add ``FEDERATION_ENABLED=false`` to your .env file to disable federation
on the application side.
The last step involves generating RSA private and public keys for signing
your instance requests on the federation. This can be done via::
# on docker setups
docker-compose run --rm api python manage.py generate_keys --no-input
# on non-docker setups
source /srv/funkwhale/virtualenv/bin/activate
source /srv/funkwhale/load_env
python manage.py generate_keys --no-input
To test and troobleshoot federation, we've added a bot account. This bot is available at @test@yourinstancedomain,
To test and troubleshoot federation, we've added a bot account. This bot is available at @test@yourinstancedomain,
and sending it "/ping", for example, via Mastodon, should trigger
a response.
...
...
README.rst
View file @
d44b8627
...
...
@@ -206,3 +206,91 @@ Typical workflow for a merge request
6. Push your branch
7. Create your merge request
8. Take a step back and enjoy, we're really grateful you did all of this and took the time to contribute!
Internationalization
--------------------
When working on the front-end, any end-user string should be translated
using either ``<i18next path="yourstring">`` or the ``$t('yourstring')``
function.
Extraction is done by calling ``yarn run i18n-extract``, which
will pull all the strings from source files and put them in a PO file.
Working with federation locally
-------------------------------
To achieve that, you'll need:
1. to update your dns resolver to resolve all your .dev hostnames locally
2. a reverse proxy (such as traefik) to catch those .dev requests and
and with https certificate
3. two instances (or more) running locally, following the regular dev setup
Resolve .dev names locally
^^^^^^^^^^^^^^^^^^^^^^^^^^
If you use dnsmasq, this is as simple as doing::
echo "address=/test/172.17.0.1" | sudo tee /etc/dnsmasq.d/test.conf
sudo systemctl restart dnsmasq
If you use NetworkManager with dnsmasq integration, use this instead::
echo "address=/test/172.17.0.1" | sudo tee /etc/NetworkManager/dnsmasq.d/test.conf
sudo systemctl restart NetworkManager
Add wildcard certificate to the trusted certificates
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Simply copy bundled certificates::
sudo cp docker/ssl/test.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
This certificate is a wildcard for ``*.funkwhale.test``
Run a reverse proxy for your instances
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Create docker network
^^^^^^^^^^^^^^^^^^^^
Create the federation network::
docker network create federation
Launch everything
^^^^^^^^^^^^^^^^^
Launch the traefik proxy::
docker-compose -f docker/traefik.yml up -d
Then, in separate terminals, you can setup as many different instances as you
need::
export COMPOSE_PROJECT_NAME=node2
docker-compose -f dev.yml run --rm api python manage.py migrate
docker-compose -f dev.yml run --rm api python manage.py createsuperuser
docker-compose -f dev.yml up nginx api front nginx api celeryworker
Note that by default, if you don't export the COMPOSE_PROJECT_NAME,
we will default to node1 as the name of your instance.
Assuming your project name is ``node1``, your server will be reachable
at ``https://node1.funkwhale.test/``. Not that you'll have to trust
the SSL Certificate as it's self signed.
When working on federation with traefik, ensure you have this in your ``env``::
# This will ensure we don't bind any port on the host, and thus enable
# multiple instances of funkwhale to be spawned concurrently.
WEBPACK_DEVSERVER_PORT_BINDING=
# This disable certificate verification
EXTERNAL_REQUESTS_VERIFY_SSL=false
# this ensure you don't have incorrect urls pointing to http resources
FUNKWHALE_PROTOCOL=https
api/config/api_urls.py
View file @
d44b8627
...
...
@@ -32,6 +32,10 @@ v1_patterns += [
include
(
(
'funkwhale_api.instance.urls'
,
'instance'
),
namespace
=
'instance'
)),
url
(
r
'^federation/'
,
include
(
(
'funkwhale_api.federation.api_urls'
,
'federation'
),
namespace
=
'federation'
)),
url
(
r
'^providers/'
,
include
(
(
'funkwhale_api.providers.urls'
,
'providers'
),
...
...
api/config/settings/common.py
View file @
d44b8627
...
...
@@ -13,6 +13,8 @@ from __future__ import absolute_import, unicode_literals
from
urllib.parse
import
urlsplit
import
os
import
environ
from
celery.schedules
import
crontab
from
funkwhale_api
import
__version__
ROOT_DIR
=
environ
.
Path
(
__file__
)
-
3
# (/a/b/myfile.py - 3 = /)
...
...
@@ -25,12 +27,35 @@ try:
except
FileNotFoundError
:
pass
FUNKWHALE_URL
=
env
(
'FUNKWHALE_URL'
)
FUNKWHALE_HOSTNAME
=
urlsplit
(
FUNKWHALE_URL
).
netloc
FUNKWHALE_HOSTNAME
=
None
FUNKWHALE_HOSTNAME_SUFFIX
=
env
(
'FUNKWHALE_HOSTNAME_SUFFIX'
,
default
=
None
)
FUNKWHALE_HOSTNAME_PREFIX
=
env
(
'FUNKWHALE_HOSTNAME_PREFIX'
,
default
=
None
)
if
FUNKWHALE_HOSTNAME_PREFIX
and
FUNKWHALE_HOSTNAME_SUFFIX
:
# We're in traefik case, in development
FUNKWHALE_HOSTNAME
=
'{}.{}'
.
format
(
FUNKWHALE_HOSTNAME_PREFIX
,
FUNKWHALE_HOSTNAME_SUFFIX
)
FUNKWHALE_PROTOCOL
=
env
(
'FUNKWHALE_PROTOCOL'
,
default
=
'https'
)
else
:
try
:
FUNKWHALE_HOSTNAME
=
env
(
'FUNKWHALE_HOSTNAME'
)
FUNKWHALE_PROTOCOL
=
env
(
'FUNKWHALE_PROTOCOL'
,
default
=
'https'
)
except
Exception
:
FUNKWHALE_URL
=
env
(
'FUNKWHALE_URL'
)
_parsed
=
urlsplit
(
FUNKWHALE_URL
)
FUNKWHALE_HOSTNAME
=
_parsed
.
netloc
FUNKWHALE_PROTOCOL
=
_parsed
.
scheme
FUNKWHALE_URL
=
'{}://{}'
.
format
(
FUNKWHALE_PROTOCOL
,
FUNKWHALE_HOSTNAME
)
FEDERATION_ENABLED
=
env
.
bool
(
'FEDERATION_ENABLED'
,
default
=
True
)
FEDERATION_HOSTNAME
=
env
(
'FEDERATION_HOSTNAME'
,
default
=
FUNKWHALE_HOSTNAME
)
FEDERATION_COLLECTION_PAGE_SIZE
=
env
.
int
(
'FEDERATION_COLLECTION_PAGE_SIZE'
,
default
=
50
)
FEDERATION_MUSIC_NEEDS_APPROVAL
=
env
.
bool
(
'FEDERATION_MUSIC_NEEDS_APPROVAL'
,
default
=
True
)
ALLOWED_HOSTS
=
env
.
list
(
'DJANGO_ALLOWED_HOSTS'
)
# APP CONFIGURATION
...
...
@@ -144,16 +169,6 @@ FIXTURE_DIRS = (
# ------------------------------------------------------------------------------
EMAIL_BACKEND
=
env
(
'DJANGO_EMAIL_BACKEND'
,
default
=
'django.core.mail.backends.smtp.EmailBackend'
)
# MANAGER CONFIGURATION
# ------------------------------------------------------------------------------
# See: https://docs.djangoproject.com/en/dev/ref/settings/#admins
ADMINS
=
(
(
"""Eliot Berriot"""
,
'contact@eliotberriot.om'
),
)
# See: https://docs.djangoproject.com/en/dev/ref/settings/#managers
MANAGERS
=
ADMINS
# DATABASE CONFIGURATION
# ------------------------------------------------------------------------------
# See: https://docs.djangoproject.com/en/dev/ref/settings/#databases
...
...
@@ -321,6 +336,16 @@ CELERY_BROKER_URL = env(
# Your common stuff: Below this line define 3rd party library settings
CELERY_TASK_DEFAULT_RATE_LIMIT
=
1
CELERY_TASK_TIME_LIMIT
=
300
CELERYBEAT_SCHEDULE
=
{
'federation.clean_music_cache'
:
{
'task'
:
'funkwhale_api.federation.tasks.clean_music_cache'
,
'schedule'
:
crontab
(
hour
=
'*/2'
),
'options'
:
{
'expires'
:
60
*
2
,
},
}
}
import
datetime
JWT_AUTH
=
{
'JWT_ALLOW_REFRESH'
:
True
,
...
...
@@ -411,3 +436,8 @@ ACCOUNT_USERNAME_BLACKLIST = [
'staff'
,
'service'
,
]
+
env
.
list
(
'ACCOUNT_USERNAME_BLACKLIST'
,
default
=
[])
EXTERNAL_REQUESTS_VERIFY_SSL
=
env
.
bool
(
'EXTERNAL_REQUESTS_VERIFY_SSL'
,
default
=
True
)
api/funkwhale_api/__init__.py
View file @
d44b8627
# -*- coding: utf-8 -*-
__version__
=
'0.
7
'
__version__
=
'0.
9
'
__version_info__
=
tuple
([
int
(
num
)
if
num
.
isdigit
()
else
num
for
num
in
__version__
.
replace
(
'-'
,
'.'
,
1
).
split
(
'.'
)])
api/funkwhale_api/common/fields.py
View file @
d44b8627
import
django_filters
from
django.db
import
models
from
funkwhale_api.music
import
utils
PRIVACY_LEVEL_CHOICES
=
[
(
'me'
,
'Only me'
),
...
...
@@ -25,3 +29,15 @@ def privacy_level_query(user, lookup_field='privacy_level'):
'followers'
,
'instance'
,
'everyone'
]
})
class
SearchFilter
(
django_filters
.
CharFilter
):
def
__init__
(
self
,
*
args
,
**
kwargs
):
self
.
search_fields
=
kwargs
.
pop
(
'search_fields'
)
super
().
__init__
(
*
args
,
**
kwargs
)
def
filter
(
self
,
qs
,
value
):
if
not
value
:
return
qs
query
=
utils
.
get_query
(
value
,
self
.
search_fields
)
return
qs
.
filter
(
query
)
api/funkwhale_api/common/session.py
0 → 100644
View file @
d44b8627
import
requests
from
django.conf
import
settings
import
funkwhale_api
def
get_user_agent
():
return
'python-requests (funkwhale/{}; +{})'
.
format
(
funkwhale_api
.
__version__
,
settings
.
FUNKWHALE_URL
)
def
get_session
():
s
=
requests
.
Session
()
s
.
headers
[
'User-Agent'
]
=
get_user_agent
()
return
s
api/funkwhale_api/common/utils.py
View file @
d44b8627
from
urllib.parse
import
urlencode
,
parse_qs
,
urlsplit
,
urlunsplit
import
os
import
shutil
...
...
@@ -25,3 +26,20 @@ def on_commit(f, *args, **kwargs):
return
transaction
.
on_commit
(
lambda
:
f
(
*
args
,
**
kwargs
)
)
def
set_query_parameter
(
url
,
**
kwargs
):
"""Given a URL, set or replace a query parameter and return the
modified URL.
>>> set_query_parameter('http://example.com?foo=bar&biz=baz', 'foo', 'stuff')
'http://example.com?foo=stuff&biz=baz'
"""
scheme
,
netloc
,
path
,
query_string
,
fragment
=
urlsplit
(
url
)
query_params
=
parse_qs
(
query_string
)
for
param_name
,
param_value
in
kwargs
.
items
():
query_params
[
param_name
]
=
[
param_value
]
new_query_string
=
urlencode
(
query_params
,
doseq
=
True
)
return
urlunsplit
((
scheme
,
netloc
,
path
,
new_query_string
,
fragment
))
api/funkwhale_api/downloader/downloader.py
View file @
d44b8627
import
os
import
requests
import
json
from
urllib.parse
import
quote_plus
import
youtube_dl
...
...
api/funkwhale_api/federation/activity.py
View file @
d44b8627
import
logging
import
json
import
requests
import
requests_http_signature
from
.
import
signing
logger
=
logging
.
getLogger
(
__name__
)
from
.
import
serializers
from
.
import
tasks
ACTIVITY_TYPES
=
[
'Accept'
,
...
...
@@ -42,44 +36,32 @@ ACTIVITY_TYPES = [
OBJECT_TYPES
=
[
'Article'
,
'Audio'
,
'Collection'
,
'Document'
,
'Event'
,
'Image'
,
'Note'
,
'OrderedCollection'
,
'Page'
,
'Place'
,
'Profile'
,
'Relationship'
,
'Tombstone'
,
'Video'
,
]
]
+
ACTIVITY_TYPES
def
deliver
(
activity
,
on_behalf_of
,
to
=
[]):
from
.
import
actors
logger
.
info
(
'Preparing activity delivery to %s'
,
to
)
auth
=
requests_http_signature
.
HTTPSignatureAuth
(
use_auth_header
=
False
,
headers
=
[
'(request-target)'
,
'user-agent'
,
'host'
,
'date'
,
'content-type'
,],
algorithm
=
'rsa-sha256'
,
key
=
on_behalf_of
.
private_key
.
encode
(
'utf-8'
),
key_id
=
on_behalf_of
.
private_key_id
,
return
tasks
.
send
.
delay
(
activity
=
activity
,
actor_id
=
on_behalf_of
.
pk
,
to
=
to
)
for
url
in
to
:
recipient_actor
=
actors
.
get_actor
(
url
)
logger
.
debug
(
'delivering to %s'
,
recipient_actor
.
inbox_url
)
logger
.
debug
(
'activity content: %s'
,
json
.
dumps
(
activity
))
response
=
requests
.
post
(
auth
=
auth
,
json
=
activity
,
url
=
recipient_actor
.
inbox_url
,
headers
=
{
'Content-Type'
:
'application/activity+json'
}
)
response
.
raise_for_status
()
logger
.
debug
(
'Remote answered with %s'
,
response
.
status_code
)
def
accept_follow
(
follow
):
serializer
=
serializers
.
AcceptFollowSerializer
(
follow
)
return
deliver
(
serializer
.
data
,
to
=
[
follow
.
actor
.
url
],
on_behalf_of
=
follow
.
target
)
api/funkwhale_api/federation/actors.py
View file @
d44b8627
import
logging
import
requests
import
uuid
import
xml
from
django.conf
import
settings
from
django.db
import
transaction
from
django.urls
import
reverse
from
django.utils
import
timezone
...
...
@@ -10,9 +11,16 @@ from rest_framework.exceptions import PermissionDenied
from
dynamic_preferences.registries
import
global_preferences_registry
from
funkwhale_api.common
import
session
from
funkwhale_api.common
import
utils
as
funkwhale_utils
from
funkwhale_api.music
import
models
as
music_models
from
funkwhale_api.music
import
tasks
as
music_tasks
from
.
import
activity
from
.
import
keys
from
.
import
models
from
.
import
serializers
from
.
import
signing
from
.
import
utils
logger
=
logging
.
getLogger
(
__name__
)
...
...
@@ -24,8 +32,10 @@ def remove_tags(text):
def
get_actor_data
(
actor_url
):
response
=
requests
.
get
(
response
=
session
.
get_session
()
.
get
(
actor_url
,
timeout
=
5
,
verify
=
settings
.
EXTERNAL_REQUESTS_VERIFY_SSL
,
headers
=
{
'Accept'
:
'application/activity+json'
,
}
...
...
@@ -37,6 +47,7 @@ def get_actor_data(actor_url):
raise
ValueError
(
'Invalid actor payload: {}'
.
format
(
response
.
text
))
def
get_actor
(
actor_url
):
data
=
get_actor_data
(
actor_url
)
serializer
=
serializers
.
ActorSerializer
(
data
=
data
)
...
...
@@ -47,31 +58,48 @@ def get_actor(actor_url):
class
SystemActor
(
object
):
additional_attributes
=
{}
manually_approves_followers
=
False
def
get_request_auth
(
self
):
actor
=
self
.
get_actor_instance
()
return
signing
.
get_auth
(
actor
.
private_key
,
actor
.
private_key_id
)
def
serialize
(
self
):
actor
=
self
.
get_actor_instance
()
serializer
=
serializers
.
ActorSerializer
(
actor
)
return
serializer
.
data
def
get_actor_instance
(
self
):
a
=
models
.
Actor
(
**
self
.
get_instance_argument
(
self
.
id
,
name
=
self
.
name
,
summary
=
self
.
summary
,
**
self
.
additional_attributes
)
try
:
return
models
.
Actor
.
objects
.
get
(
url
=
self
.
get_actor_url
())
except
models
.
Actor
.
DoesNotExist
:
pass
private
,
public
=
keys
.
get_key_pair
()
args
=
self
.
get_instance_argument
(
self
.
id
,
name
=
self
.
name
,
summary
=
self
.
summary
,
**
self
.
additional_attributes
)
a
.
pk
=
self
.
id
return
a
args
[
'private_key'
]
=
private
.
decode
(
'utf-8'
)
args
[
'public_key'
]
=
public
.
decode
(
'utf-8'
)
return
models
.
Actor
.
objects
.
create
(
**
args
)
def
get_actor_url
(
self
):
return
utils
.
full_url
(
reverse
(
'federation:instance-actors-detail'
,
kwargs
=
{
'actor'
:
self
.
id
}))
def
get_instance_argument
(
self
,
id
,
name
,
summary
,
**
kwargs
):
preferences
=
global_preferences_registry
.
manager
()
p
=
{
'preferred_username'
:
id
,
'domain'
:
settings
.
FEDERATION_HOSTNAME
,
'type'
:
'Person'
,
'name'
:
name
.
format
(
host
=
settings
.
FEDERATION_HOSTNAME
),
'manually_approves_followers'
:
True
,
'url'
:
utils
.
full_url
(
reverse
(
'federation:instance-actors-detail'
,
kwargs
=
{
'actor'
:
id
})),
'url'
:
self
.
get_actor_url
(),
'shared_inbox_url'
:
utils
.
full_url
(
reverse
(
'federation:instance-actors-inbox'
,
...
...
@@ -84,8 +112,6 @@ class SystemActor(object):
reverse
(
'federation:instance-actors-outbox'
,
kwargs
=
{
'actor'
:
id
})),
'public_key'
:
preferences
[
'federation__public_key'
],
'private_key'
:
preferences
[
'federation__private_key'
],
'summary'
:
summary
.
format
(
host
=
settings
.
FEDERATION_HOSTNAME
)
}
p
.
update
(
kwargs
)
...
...
@@ -95,7 +121,7 @@ class SystemActor(object):
raise
NotImplementedError
def
post_inbox
(
self
,
data
,
actor
=
None
):
r
aise
NotImplementedErr
or