Skip to content

WIP: Resolve "Per-user libraries"

Agate requested to merge 463-user-libraries into develop

Closes #463 (closed), #160 (closed), #147 (closed), #133 (closed)

Relates to: #252 (closed), #453, #467 (closed), #472

Makes obsoletes: #441 (closed), #442 (closed), #394 (closed) #349 (closed), #269 (closed), #203 (closed), #92 (closed), #78 (closed), #221 (closed), #250 (closed),

This PR is rather huge, and aims at solving quite a lot of issues of the platform in its current state. It's quite opiniated, and I feel like it's important to discuss the how and why here.

A few goals of Funkwhale

Among other things, I started Funkwhale to satisfy these main use cases:

  1. Have my personal music library available on the web
  2. Interact with other people around music, in a "social network"-y way (discover new artists, access music I would not have listened by myself, etc.)

I consider 1. to be basically fulfilled in the current state of the system. As an instance owner you can boot an instance, import your music and you're good to go. Of course, there are bugs and improvements and features to add around that, but the core itself is here. However, it's only fulfilled for users who have the upload permission on an instance otherwise, you are only in a consuming position, you cannot add new content.

For 2., well, it's not really the case yet. Most of the features you would expect for social discovery, such as following people, access other people activity to discover stuff, direct interaction between users, all of that is either WIP or not implemented at all.

Other the past few months, a third, equally important (at least for me) use case emerged: make Funkwhale a creator-friendly platform to broadcast music (and possibly other types of audio) and build a community around that, in a way similar to SoundCloud.

While it's theoretically possible to have open instances where creators can upload their work, the way Funkwhale is currently designed makes it harder, difficult or dangerous for instance owners to satisfy this use case. The same thing goes for use case 2., and we'll discuss why in the next part.

To sum it up, this is how Funkwhale is used or should be usable:

  1. As a user, to stream music you like
  2. As a user, to discover interesting content and interact around it
  3. As a creator, to make your work available to other people

The copyrighted federation paradox

Funkwhale was designed, from start, with copyrighted content in mind, and in an instance-centric fashion:

  • There is one library per-instance
  • All users of an instance have access to the library

Because content may be copyrighted, everything was very (too much) cautious:

  • Instances are closed by default (both for signup and music streaming)
  • Uploading to the instance library require special permissions

This worked in the context of small, familial or friend-based, instances but it became obvious we would have problems when we implemented federation of library contents between instances, in April 2018: this simply does not scale.

Because everything is instance-centric, as an instance owner, your are basically responsible of your instance library and it's content: who has the permission to manage it, what instances can access it. On the contrary, as a user, you have no control on anything at all: you cannot access another library if it's not federated with your instance. You cannot upload your own music if you don't have the permission.

All of this lead to the current situation:

  1. Instance owners have no incentive to open their instance to new users (as Mastodon, Peertube or Pleroma instance owners could do), because it's dangerous for them if they end up streaming copyrighted content to anyone
  2. Federating instances is a dangerous, all-or-nothing act of faith during which you put your trust on other instances not to be too big
  3. As a result of 1. and 2., it's impossible for new users to find instances they can join, and for creators to find instances where they can upload content without hassle

This has been a huge source of frustration and incomprehension about how the project work, slowing the growth of the network, and I think it is the time to tackle these issues.

A solution?

What if we shifted the responsibilities and power from instance owners to users, and moved (at least partially) from an instance-centric design to a user-centric design?

My proposal is quite simple, but have a lot of impact on a lot of aspects of the project (both technically and functionally). To sum it up: let's kill the instance library, and split it in users libraries.

As a user:

  1. You have an upload quota, e.g. 500MB. You can upload files on your instance up to this quota
  2. Uploaded files are stored in one of your libraries. You can create one or many libraries, that are bound to your account, and you have full control of them. During upload, you choose in which library you want to upload your files.
  3. Libraries have a visibility setting:
  • Private: your library is private and nobody except you can access it (metadata + streaming)
  • Instance: only people on your instance can access the library content.
  • Public: everybody can access the library contents, including people from other instances
  1. In addition to this visibility setting, you can explicitly share a library with other people (both on your instance and other instances).

How does it help with the mentioned issues?

User managing their own libraries has a lot of positive repercussions and unlock new use cases.

As an instance owner:

  • you spend less time managing permissions and worrying about streaming copyrighted content, because you're not managing a library at all. Of course, you are still responsible to answer to take down requests if one or your users is sharing copyrighted content publicly, but this is expected when you're hosting this kind of services.

This makes it safer and easier for instance admins to open their instance completely.

As a user:

  • you can upload your own music library to your instance and enjoy it without having to ask for specific permissions
  • you can share your music library with friends/family or even have public libraries of open content

As a creator:

  • you can upload your own work and share it publicly on the network without having to ask for specific permissions

What are the possible issues with this proposal?

  • Performance issues: introducing user libraries will necessarily have impact on the application, as we'll have more work to do to check if a user can listen to a track or not
  • User experience: the current workflow is easy: every track you see is playable. This would not be the case anymore because there could be activity (listenings, favorites, comments, etc.) on tracks that are not in any library you have access to.
  • Storage issues: it is not easy to deduplicate upload content in this context, and could lead to increased storage costs. This is at least partially mitigated by the quota system though.
  • Feature removal/possible breakages:
    • YouTube import: this feature is quite problematic because of the whole "piracy" connotation stuff. I think it is safer to remove it completely, as expose it to each and every user would only increase its visibility
    • The federation, in its current state: it's not possible to offer a clean migration path for the federation between instance libraries and user libraries. This is not such a big issue since the federation is still rather small, but we have to keep that in mind.
    • Music requests: since everybody can upload and nobody is responsible for a single library, there is not much sense to keep those

Things to think about if you're still here

  • Are there any current use case that will be impossible to fulfill if this proposal is implemented?
  • Is there any possible issue we overlooked?
  • Anything else?


Once all of those are checked, the work is complete:

  • User libraries:
    • Library management / deletion
    • File management / deletion
  • Improved imports:
    • Per user quotas
    • Default quota for new users
    • Upload to a user library
    • Advanced error management during import: what wen wrong, how can I fix it
    • Handle file replacement
    • Skip track (or inform) that a track is present in another library
    • Rework CLI with new system
  • Permission logic:
    • Library access data available in API for artists, tracks, etc.
    • Interface (play buttons, playlists, filters, etc.)
    • Handle library access in radios
  • Library sharing:
    • Request access to another library
    • Grant access to an owned library
  • Scan / import of remote libraries:
    • Scanning reporting
    • Scanning management (pause/cancel/restart)
    • Automatic scanning
  • Federation:
    • More generic API logic (routers, etc.)
    • Use AS duration field on audio
    • Follow/Unfollow/AP Notifications
    • Broadcast libraries update (deleted/visibility changed) to followers
    • Broadcast files update (added/deleted) to followers
    • Ensure cache is cleaning properly
    • Ensure federation preferences are honored for outbox / inbox
  • Admin tools:
    • Modify user quota
    • Disable
  • Cleanup:
    • Youtube Import
    • System actors, old/unused AP code
    • Import requests
  • Documentation:
    • Federation
    • User libraries
    • Import
    • Changelog
  • Migration script to convert instance library to per-user libraries (and populate missing federation data such as followers url for actors, federation ids for all music entities, etc.)
  • Final check that things work: subsonic, federation, migration script, playlists not playable, etc.
Edited by Agate

Merge request reports