Fix #256: Added troubleshotting and technical overview documentation

.gitlab-ci.yml

@@ -63,12 +63,15 @@ review_docs:
     BUILD_PATH: "../public"
   before_script:
     - cd docs
+    - apt-get update
+    - apt-get install -y graphviz
+    - pip install sphinx
+
   cache:
     key: "$CI_PROJECT_ID__sphinx"
     paths:
       - "$PIP_CACHE_DIR"
   script:
-    - pip install sphinx
     - ./build_docs.sh
     - mkdir -p /static/docs/$CI_BUILD_REF_SLUG
     - cp -r $CI_PROJECT_DIR/public/* /static/docs/$CI_BUILD_REF_SLUG
@@ -205,8 +208,10 @@ pages:
     BUILD_PATH: "../public"
   before_script:
     - cd docs
-  script:
+    - apt-get update
+    - apt-get install -y graphviz
     - pip install sphinx
+  script:
     - ./build_docs.sh
   cache:
     key: "$CI_PROJECT_ID__sphinx"

CONTRIBUTING

 Contribute to Funkwhale development
-==================================
+===================================
 
 First of all, thank you for your interest in the project! We really
 appreciate the fact that you're about to take some time to read this
@@ -111,7 +111,7 @@ Create it like this::
 
 
 Create docker network
-^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^
 
 Create the federation network::
 

changes/changelog.d/256.doc

+Added troubleshotting and technical overview documentation (#256)

docs/Dockerfile

-FROM python:3.6-alpine
+FROM python:3.6
 
+RUN apt-get update && apt-get install -y graphviz
 RUN pip install sphinx livereload
 WORKDIR /app/docs

docs/architecture.rst

+Architecture
+============
+
+Funkwhale is made of several components, each of them fulfilling a specific mission:
+
+.. graphviz::
+
+    digraph {
+        node [shape=record];
+        rankdir=TB
+        concentrate=true
+        user [group="frontend" label="User" fontsize="9"]
+        webui [group="frontend" label="Web interface (VueJS SPA)" fontsize="9"]
+        subapps [group="frontend" label="Subsonic-compatible apps (DSub, Clementine)" fontsize="9"]
+        proxy [label="Reverse proxy (Nginx/Apache)" fontsize="9"]
+        api [label="API Server (Django)" fontsize="9"]
+        db [label="Database (PostgreSQL)" fontsize="9"]
+        cache [label="Cache and message queue (Redis)" fontsize="9"]
+        worker [label="Worker (Celery)" fontsize="9"]
+        scheduler [label="Task scheduler (Celery Beat)" fontsize="9"]
+
+        user -> subapps -> proxy
+        user -> webui -> proxy
+        cache -> worker
+        proxy -> api
+        api -> cache
+        api -> db
+        scheduler -> cache
+        worker -> cache
+        worker -> db
+    }
+
+This graph may looks a bit scary, so we'll detail the role of each component below.
+
+The user
+--------
+
+Funkwhale users can interact with your instance using:
+
+- The official web interface
+- Third-party apps
+
+The web interface
+-----------------
+
+This refers to Funkwhale's built-in web interface, which is a Single Page application
+written in Vue JS. This application will interact with Funkwhale's API to retrieve
+or persist data.
+
+Third-party apps
+----------------
+
+Since Funkwhale implements a subset of the Subsonic API, it's compatible with existing apps such
+as DSub, Ultrasonic or Clementine that support this API. Those apps can be used as a replacement
+or in conjunction of the web interface, but the underlying data is the same.
+
+The reverse proxy
+-----------------
+
+Funkwhale's API server should never be exposed directly to the internet, as we require
+a reverse proxy (Apache or Nginx) for performance and security reasons. The reverse proxy
+will receive client HTTP requests, and:
+
+- Proxy them to the API server
+- Serve requested static files (Audio files, stylesheets, javascript, fonts...)
+
+The API server
+--------------
+
+Funkwhale's API server is the central piece of the project. This component is responsible
+for answering and processing user requests, manipulate data from the database, send long-running
+tasks to workers, etc.
+
+It's a Python/Django application.
+
+The database
+------------
+
+Most of the data such as user accounts, favorites, music metadata or playlist is stored
+in a PostgreSQL database.
+
+The cache/message queue
+-----------------------
+
+Fetching data from the database is sometimes slow or resource hungry. To reduce
+the load, Redis act as a cache for data that is considerably faster than a database.
+
+It is also a message queue that will deliver tasks to the worker.
+
+The worker
+----------
+
+Some operations are too long to live in the HTTP request/response cycle. Typically,
+importing a bunch of uploaded tracks could take a minute or two.
+
+To keep the API response time as fast as possible, we offload long-running tasks
+to a background process, also known as the Celery worker.
+
+Typical tasks include:
+
+- Handling music imports
+- Handling federation/ActivityPub messages
+- Scanning other instances libraries
+
+This worker is also able to retry failed tasks, or spawn automatically
+more process when the number of received tasks increase.
+
+The scheduler
+-------------
+
+Some long-running tasks are not triggered by user or external input, but on a recurring
+basis instead. The scheduler is responsible for triggering those tasks and put the corresponding
+messages in the message queue so the worker can process them.
+
+Recurring tasks include:
+
+- Cache cleaning
+- Music metadata refreshing

docs/conf.py

@@ -20,7 +20,7 @@
 import os
 import sys
 
-sys.path.insert(0, os.path.abspath('../api'))
+sys.path.insert(0, os.path.abspath("../api"))
 
 import funkwhale_api  # NOQA
 
@@ -33,24 +33,24 @@ import funkwhale_api  # NOQA
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
+extensions = ["sphinx.ext.graphviz"]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
 # source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
+source_suffix = ".rst"
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = "index"
 
 # General information about the project.
-project = 'funkwhale'
-copyright = '2017, Eliot Berriot'
-author = 'Eliot Berriot'
+project = "funkwhale"
+copyright = "2017, Eliot Berriot"
+author = "Eliot Berriot"
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -71,10 +71,10 @@ language = None
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This patterns also effect to html_static_path and html_extra_path
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+pygments_style = "sphinx"
 
 # If true, `todo` and `todoList` produce output, else they produce nothing.
 todo_include_todos = False
@@ -85,7 +85,7 @@ todo_include_todos = False
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'alabaster'
+html_theme = "alabaster"
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
@@ -96,13 +96,13 @@ html_theme = 'alabaster'
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = ["_static"]
 
 
 # -- Options for HTMLHelp output ------------------------------------------
 
 # Output file base name for HTML help builder.
-htmlhelp_basename = 'funkwhaledoc'
+htmlhelp_basename = "funkwhaledoc"
 
 
 # -- Options for LaTeX output ---------------------------------------------
@@ -111,15 +111,12 @@ latex_elements = {
     # The paper size ('letterpaper' or 'a4paper').
     #
     # 'papersize': 'letterpaper',
-
     # The font size ('10pt', '11pt' or '12pt').
     #
     # 'pointsize': '10pt',
-
     # Additional stuff for the LaTeX preamble.
     #
     # 'preamble': '',
-
     # Latex figure (float) alignment
     #
     # 'figure_align': 'htbp',
@@ -129,8 +126,7 @@ latex_elements = {
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
-    (master_doc, 'funkwhale.tex', 'funkwhale Documentation',
-     'Eliot Berriot', 'manual'),
+    (master_doc, "funkwhale.tex", "funkwhale Documentation", "Eliot Berriot", "manual")
 ]
 
 
@@ -138,10 +134,7 @@ latex_documents = [
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [
-    (master_doc, 'funkwhale', 'funkwhale Documentation',
-     [author], 1)
-]
+man_pages = [(master_doc, "funkwhale", "funkwhale Documentation", [author], 1)]
 
 
 # -- Options for Texinfo output -------------------------------------------
@@ -150,7 +143,13 @@ man_pages = [
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
 texinfo_documents = [
-    (master_doc, 'funkwhale', 'funkwhale Documentation',
-     author, 'funkwhale', 'One line description of project.',
-     'Miscellaneous'),
+    (
+        master_doc,
+        "funkwhale",
+        "funkwhale Documentation",
+        author,
+        "funkwhale",
+        "One line description of project.",
+        "Miscellaneous",
+    )
 ]

docs/configuration.rst

@@ -124,6 +124,16 @@ On non-docker setup, you don't need to configure this setting.
 
 .. note:: This path should not include any trailing slash
 
+.. _setting-REVERSE_PROXY_TYPE:
+
+``REVERSE_PROXY_TYPE``
+^^^^^^^^^^^^^^^^^^^^^^
+
+Default: ``nginx``
+
+The type of reverse-proxy behind which Funkwhale is served. Either ``apache2``
+or ``nginx``. This is only used if you are using in-place import.
+
 User permissions
 ----------------
 

docs/index.rst

@@ -13,9 +13,11 @@ Funkwhale is a self-hosted, modern free and open-source music server, heavily in
 
    users/index
    features
+   architecture
    installation/index
    upgrading
    configuration
+   troubleshooting
    importing-music
    federation
    api

docs/upgrading.rst

@@ -72,7 +72,7 @@ Upgrading the API
 ^^^^^^^^^^^^^^^^^
 
 On non-docker, upgrade involves a few more commands. We assume your setup
-match what is described in :doc:`debian`:
+match what is described in :doc:`/installation/debian`:
 
 .. parsed-literal::