S3 "in-place" import and "watch" functionality
What is the problem you are facing?
For my own use cases, I've determined that I'd like to store my library on S3 but am missing out on features of library management that I would be able to utilize if my files were stored on-server, like the import_files --watch
functionality. I researched options, which primarily led me to looking into S3FS, but after some conversations in the Matrix servers and my own reading up on how that library would work in this condition, it doesn't seem feasible.
So, I'm proposing a couple things:
- API endpoint which can be notified of S3 events.
- Idempotent management command to import files already stored on the bucket.
The former would allow one to freely operate on the objects in their S3 bucket and keep their library up to date automatically, and the latter is to allow ~equivalent usage of the import_files
management command.
What are the possible drawbacks or issues with the requested changes?
- Further reliance on S3 standards and features
- Additional hosting costs for admins that enable the feature (would need to support a lambda, as well as some slight overhead when Funkwhale needs to pull remote updated files to check for tags/checksums).
- I haven't quite worked out the end-user experience here -- presumably I would think it's similar to the in-place import functionality where the files could only be imported into a given library, but considering the entire pod would be sharing space in the bucket then the management API could keep all users' file info up to date.
- The management API would need to be privileged, either by an admin's OAuth or some API key so that people can't spoof requests to it.
- Compared to the Watchdog that has events moved/created/modified/deleted, S3 only denotes creates and deletes. For example, a move or a file rename are composed of a COPY (the file to the new name/location) and a DELETE (the old reference). This makes the logic around how to handle events a little bit trickier, but not impossible.
Context
I have already begun working on this feature, but I figured I'd make an Issue just to have something on the books. Also feel free to chime in with requests/additional functionality if desired. I'll have a MR sometime soon I hope, that I'd love feedback on. If this feature request is desired by others, feel free to add me as the assignee on this issue.
Also, Koel features similar functionality, documentation seen here.