Skip to content

CLI importer improvements

Agate requested to merge import-files-optimization into develop

For context, someone is making some attempts at importing a 30TB library in Funkwhale via CLI. This MR is a results of direct discussion with this person to ensure the importer works and doesn't crash on such large libraries.

This mainly improves the filesystem crawler,

  • Handle import in batches, to avoid OOM issues
  • Optimization of the check_inplace_files command to avoid OOM issues
  • Better UX with CLI importer, by removing the need to use glob patterns

Merge request reports