For context, someone is making some attempts at importing a 30TB library in Funkwhale via CLI. This MR is a results of direct discussion with this person to ensure the importer works and doesn't crash on such large libraries.
This mainly improves the filesystem crawler,
Handle import in batches, to avoid OOM issues
Optimization of the
check_inplace_filescommand to avoid OOM issues
Better UX with CLI importer, by removing the need to use glob patterns