CLI importer improvements
For context, someone is making some attempts at importing a 30TB library in Funkwhale via CLI. This MR is a results of direct discussion with this person to ensure the importer works and doesn't crash on such large libraries.
This mainly improves the filesystem crawler,
-
Handle import in batches, to avoid OOM issues -
Optimization of the check_inplace_files
command to avoid OOM issues -
Better UX with CLI importer, by removing the need to use glob patterns