Documentation update for the new in-place import

CHANGELOG

@@ -3,6 +3,52 @@ Changelog
 
 .. towncrier
 
+0.10 (Unreleased)
+-----------------
+
+
+In-place import
+^^^^^^^^^^^^^^^
+
+This release includes in-place imports for the CLI import. This means you can
+load gigabytes of music into funkwhale without worrying about about Funkwhale
+copying those music files in its internal storage and eating your disk space.
+
+This new feature is documented <here> and require additional configuration
+to ensure funkwhale and your webserver can serve those files properly.
+
+**Non-docker users:**
+
+Assuming your music is stored in ``/srv/funkwhale/data/music``, add the following
+block to your nginx configuration::
+
+    location /_protected/music {
+        internal;
+        alias   /srv/funkwhale/data/music;
+    }
+
+And the following to your .env file::
+
+    MUSIC_DIRECTORY_PATH=/srv/funkwhale/data/music
+
+**Docker users:**
+
+Assuming your music is stored in ``/srv/funkwhale/data/music``, add the following
+block to your nginx configuration::
+
+    location /_protected/music {
+        internal;
+        alias   /srv/funkwhale/data/music;
+    }
+
+Assuming you have the following volume directive in your ``docker-compose.yml``
+(it's the default): ``/srv/funkwhale/data/music:/music:ro``, then add
+the following to your .env file::
+
+    MUSIC_DIRECTORY_PATH=/music
+    MUSIC_DIRECTORY_SERVE_PATH=/srv/funkwhale/data/music
+
+
 0.9.1 (2018-04-17)
 ------------------
 

deploy/env.prod.sample

+# If you have any doubts about what a setting does,
+# check https://docs.funkwhale.audio/configuration.html#configuration-reference
+
 # If you're tweaking this file from the template, ensure you edit at least the
 # following variables:
 # - DJANGO_SECRET_KEY
 # - DJANGO_ALLOWED_HOSTS
 # - FUNKWHALE_URL
-
-# Additionaly, on non-docker setup **only**, you'll also have to tweak/uncomment those variables:
+# On non-docker setup **only**, you'll also have to tweak/uncomment those variables:
 # - DATABASE_URL
 # - CACHE_URL
 # - STATIC_ROOT
 # - MEDIA_ROOT
 #
 # You **don't** need to update those variables on pure docker setups.
-
+#
+# Additional options you may want to check:
+# - MUSIC_DIRECTORY_PATH and MUSIC_DIRECTORY_SERVE_PATH if you plan to use
+#   in-place import
 # Docker only
 # -----------
 
@@ -19,7 +24,9 @@
 # (it will be interpolated in docker-compose file)
 # You can comment or ignore this if you're not using docker
 FUNKWHALE_VERSION=latest
+MUSIC_DIRECTORY_PATH=/music
 
+# End of Docker-only configuration
 
 # General configuration
 # ---------------------
@@ -34,6 +41,7 @@ FUNKWHALE_API_PORT=5000
 # your instance
 FUNKWHALE_URL=https://yourdomain.funwhale
 
+
 # API/Django configuration
 
 # Database configuration
@@ -94,3 +102,9 @@ FEDERATION_ENABLED=True
 # means anyone can subscribe to your library and import your file,
 # use with caution.
 FEDERATION_MUSIC_NEEDS_APPROVAL=True
+
+# In-place import settings
+# You can safely leave those settings uncommented if you don't plan to use
+# in place imports.
+# MUSIC_DIRECTORY_PATH=
+# MUSIC_DIRECTORY_SERVE_PATH=

deploy/nginx.conf

@@ -84,6 +84,14 @@ server {
         alias   /srv/funkwhale/data/media;
     }
 
+    location /_protected/music {
+        # this is an internal location that is used to serve
+        # audio files once correct permission / authentication
+        # has been checked on API side
+        internal;
+        alias   /srv/funkwhale/data/music;
+    }
+
     # Transcoding logic and caching
     location = /transcode-auth {
         include /etc/nginx/funkwhale_proxy.conf;

docker/nginx/conf.dev

@@ -42,6 +42,10 @@ http {
             internal;
             alias   /protected/media;
         }
+        location /_protected/music {
+            internal;
+            alias   /music;
+        }
         location = /transcode-auth {
             # needed so we can authenticate transcode requests, but still
             # cache the result

docs/configuration.rst

@@ -33,3 +33,44 @@ The URL should be ``/api/admin/dynamic_preferences/globalpreferencemodel/`` (pre
 If you plan to use acoustid and external imports
 (e.g. with the youtube backends), you should edit the corresponding
 settings in this interface.
+
+Configuration reference
+-----------------------
+
+.. _setting-MUSIC_DIRECTORY_PATH:
+
+``MUSIC_DIRECTORY_PATH``
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Default: ``None``
+
+The path on your server where Funwkhale can import files using :ref:`in-place import
+<in-place-import>`. It must be readable by the webserver and funkwhale
+api and worker processes.
+
+On docker installations, we recommend you use the default of ``/music``
+for this value. For non-docker installation, you can use any absolute path.
+``/srv/funkwhale/data/music`` is a safe choice if you don't know what to use.
+
+.. note:: This path should not include any trailing slash
+
+.. _setting-MUSIC_DIRECTORY_SERVE_PATH:
+
+``MUSIC_DIRECTORY_SERVE_PATH``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Default: :ref:`setting-MUSIC_DIRECTORY_PATH`
+
+When using Docker, the value of :ref:`MUSIC_DIRECTORY_PATH` in your containers
+may differ from the real path on your host. Assuming you have the following directive
+in your :file:`docker-compose.yml` file::
+
+    volumes:
+      - /srv/funkwhale/data/music:/music:ro
+
+Then, the value of :ref:`setting-MUSIC_DIRECTORY_SERVE_PATH` should be
+``/srv/funkwhale/data``. This must be readable by the webserver.
+
+On non-docker setup, you don't need to configure this setting.
+
+.. note:: This path should not include any trailing slash

docs/importing-music.rst

@@ -22,8 +22,15 @@ to the ``/music`` directory on the container:
 
     docker-compose run --rm api python manage.py import_files "/music/**/*.ogg" --recursive --noinput
 
-For the best results, we recommand tagging your music collection through
-`Picard <http://picard.musicbrainz.org/>`_ in order to have the best quality metadata.
+The import command supports several options, and you can check the help to
+get details::
+
+    docker-compose run --rm api python manage.py import_files --help
+
+.. note::
+
+    For the best results, we recommand tagging your music collection through
+    `Picard <http://picard.musicbrainz.org/>`_ in order to have the best quality metadata.
 
 .. note::
 
@@ -39,18 +46,39 @@ For the best results, we recommand tagging your music collection through
 
     At the moment, only OGG/Vorbis and MP3 files with ID3 tags are supported
 
-.. note::
 
-    The --recursive flag will work only on Python 3.5+, which is the default
-    version When using Docker or Debian 9. If you use an older version of Python,
-    remove the --recursive flag and use more explicit import patterns instead::
+.. _in-place-import:
+
+In-place import
+^^^^^^^^^^^^^^^
+
+By default, the CLI-importer will copy imported files to Funkwhale's internal
+storage. This means importing a 1Gb library will result in the same amount
+of space being used by Funkwhale.
+
+While this behaviour has some benefits (easier backups and configuration),
+it's not always the best choice, especially if you have a huge library
+to import and don't want to double your disk usage.
+
+The CLI importer supports an additional ``--in-place`` option that triggers the
+following behaviour during import:
+
+1. Imported files are not store in funkwhale anymore
+2. Instead, Funkwhale will store the file path and use it to serve the music
+
+Because those files are not managed by Funkwhale, we offer additional
+configuration options to ensure the webserver can serve them properly:
+
+- :ref:`setting-MUSIC_DIRECTORY_PATH`
+- :ref:`setting-MUSIC_DIRECTORY_SERVE_PATH`
 
-        # this will only import ogg files at the second level
-        "/srv/funkwhale/data/music/*/*.ogg"
-        # this will only import ogg files in the fiven directory
-        "/srv/funkwhale/data/music/System-of-a-down/*.ogg"
+.. warning::
 
+    While in-place import is faster and less disk-space-hungry, it's also
+    more fragile: if, for some reason, you move or rename the source files,
+    Funkwhale will not be able to serve those files anymore.
 
+    Thus, be especially careful when you manipulate the source files.
 
 Getting demo tracks
 ^^^^^^^^^^^^^^^^^^^