funkwhale-beat server constantly crashing and restarting
Steps to reproduce
- Install Funkwhale using official install script (https://get.funkwhale.audio/) on Debian Buster
What happens?
funkwhale-beat.service constantly fails (and restarts) with a _dbm.error
. I'm not at all familiar with Celery, but this seems weird because I have it set up to use redis.
Full log posted below.
What is expected?
No failure.
Context
Funkwhale version(s) affected: 0.21
Debian version
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
Full log
funkwhale-beat[4555]: 2020-06-02 17:55:58,860 funkwhale_api.config INFO Loaded env file at /srv/funkwhale/config/.env
funkwhale-beat[4555]: 2020-06-02 17:55:58,860 funkwhale_api.config INFO Running with no plugins
funkwhale-beat[4555]: [2020-06-02 15:56:00,756: INFO/MainProcess] beat: Starting...
funkwhale-beat[4555]: [2020-06-02 15:56:00,759: CRITICAL/MainProcess] beat raised exception <class '_dbm.error'>: error('cannot add item to database')
funkwhale-beat[4555]: Traceback (most recent call last):
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
funkwhale-beat[4555]: return obj.__dict__[self.__name__]
funkwhale-beat[4555]: KeyError: 'scheduler'
funkwhale-beat[4555]: During handling of the above exception, another exception occurred:
funkwhale-beat[4555]: Traceback (most recent call last):
funkwhale-beat[4555]: File "/usr/lib/python3.7/shelve.py", line 111, in __getitem__
funkwhale-beat[4555]: value = self.cache[key]
funkwhale-beat[4555]: KeyError: 'entries'
funkwhale-beat[4555]: During handling of the above exception, another exception occurred:
funkwhale-beat[4555]: Traceback (most recent call last):
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 524, in _create_schedule
funkwhale-beat[4555]: self._store[str('entries')]
funkwhale-beat[4555]: File "/usr/lib/python3.7/shelve.py", line 113, in __getitem__
funkwhale-beat[4555]: f = BytesIO(self.dict[key.encode(self.keyencoding)])
funkwhale-beat[4555]: KeyError: b'entries'
funkwhale-beat[4555]: During handling of the above exception, another exception occurred:
funkwhale-beat[4555]: Traceback (most recent call last):
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/apps/beat.py", line 109, in start_scheduler
funkwhale-beat[4555]: service.start()
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 588, in start
funkwhale-beat[4555]: humanize_seconds(self.scheduler.max_interval))
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
funkwhale-beat[4555]: value = obj.__dict__[self.__name__] = self.__get(obj)
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 632, in scheduler
funkwhale-beat[4555]: return self.get_scheduler()
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 627, in get_scheduler
funkwhale-beat[4555]: lazy=lazy,
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 467, in __init__
funkwhale-beat[4555]: Scheduler.__init__(self, *args, **kwargs)
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 226, in __init__
funkwhale-beat[4555]: self.setup_schedule()
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 495, in setup_schedule
funkwhale-beat[4555]: self._create_schedule()
funkwhale-beat[4555]: File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 528, in _create_schedule
funkwhale-beat[4555]: self._store[str('entries')] = {}
funkwhale-beat[4555]: File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__
funkwhale-beat[4555]: self.dict[key.encode(self.keyencoding)] = f.getvalue()
funkwhale-beat[4555]: _dbm.error: cannot add item to database
funkwhale-beat[4555]: [2020-06-02 15:56:00,760: WARNING/MainProcess] Traceback (most recent call last):
funkwhale-beat[4555]: [2020-06-02 15:56:00,760: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] return obj.__dict__[self.__name__]
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] KeyError
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] :
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] 'scheduler'
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] During handling of the above exception, another exception occurred:
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] Traceback (most recent call last):
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] File "/usr/lib/python3.7/shelve.py", line 111, in __getitem__
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] value = self.cache[key]
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] KeyError
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] :
funkwhale-beat[4555]: [2020-06-02 15:56:00,761: WARNING/MainProcess] 'entries'
funkwhale-beat[4555]: [2020-06-02 15:56:00,762: WARNING/MainProcess] During handling of the above exception, another exception occurred:
funkwhale-beat[4555]: [2020-06-02 15:56:00,762: WARNING/MainProcess] Traceback (most recent call last):
funkwhale-beat[4555]: [2020-06-02 15:56:00,762: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 524, in _create_schedule
funkwhale-beat[4555]: [2020-06-02 15:56:00,762: WARNING/MainProcess] self._store[str('entries')]
funkwhale-beat[4555]: [2020-06-02 15:56:00,762: WARNING/MainProcess] File "/usr/lib/python3.7/shelve.py", line 113, in __getitem__
funkwhale-beat[4555]: [2020-06-02 15:56:00,762: WARNING/MainProcess] f = BytesIO(self.dict[key.encode(self.keyencoding)])
funkwhale-beat[4555]: [2020-06-02 15:56:00,762: WARNING/MainProcess] KeyError
funkwhale-beat[4555]: [2020-06-02 15:56:00,762: WARNING/MainProcess] :
funkwhale-beat[4555]: [2020-06-02 15:56:00,762: WARNING/MainProcess] b'entries'
funkwhale-beat[4555]: [2020-06-02 15:56:00,763: WARNING/MainProcess] During handling of the above exception, another exception occurred:
funkwhale-beat[4555]: [2020-06-02 15:56:00,763: WARNING/MainProcess] Traceback (most recent call last):
funkwhale-beat[4555]: [2020-06-02 15:56:00,763: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/bin/celery", line 10, in <module>
funkwhale-beat[4555]: [2020-06-02 15:56:00,763: WARNING/MainProcess] sys.exit(main())
funkwhale-beat[4555]: [2020-06-02 15:56:00,763: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/__main__.py", line 16, in main
funkwhale-beat[4555]: [2020-06-02 15:56:00,763: WARNING/MainProcess] _main()
funkwhale-beat[4555]: [2020-06-02 15:56:00,763: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/bin/celery.py", line 322, in main
funkwhale-beat[4555]: [2020-06-02 15:56:00,763: WARNING/MainProcess] cmd.execute_from_commandline(argv)
funkwhale-beat[4555]: [2020-06-02 15:56:00,763: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline
funkwhale-beat[4555]: [2020-06-02 15:56:00,764: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))
funkwhale-beat[4555]: [2020-06-02 15:56:00,764: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/bin/base.py", line 298, in execute_from_commandline
funkwhale-beat[4555]: [2020-06-02 15:56:00,764: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])
funkwhale-beat[4555]: [2020-06-02 15:56:00,764: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/bin/celery.py", line 488, in handle_argv
funkwhale-beat[4555]: [2020-06-02 15:56:00,765: WARNING/MainProcess] return self.execute(command, argv)
funkwhale-beat[4555]: [2020-06-02 15:56:00,765: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/bin/celery.py", line 420, in execute
funkwhale-beat[4555]: [2020-06-02 15:56:00,765: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
funkwhale-beat[4555]: [2020-06-02 15:56:00,765: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/bin/base.py", line 302, in run_from_argv
funkwhale-beat[4555]: [2020-06-02 15:56:00,765: WARNING/MainProcess] sys.argv if argv is None else argv, command)
funkwhale-beat[4555]: [2020-06-02 15:56:00,765: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/bin/base.py", line 386, in handle_argv
funkwhale-beat[4555]: [2020-06-02 15:56:00,766: WARNING/MainProcess] return self(*args, **options)
funkwhale-beat[4555]: [2020-06-02 15:56:00,766: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/bin/base.py", line 252, in __call__
funkwhale-beat[4555]: [2020-06-02 15:56:00,766: WARNING/MainProcess] ret = self.run(*args, **kwargs)
funkwhale-beat[4555]: [2020-06-02 15:56:00,766: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/bin/beat.py", line 109, in run
funkwhale-beat[4555]: [2020-06-02 15:56:00,766: WARNING/MainProcess] return beat().run()
funkwhale-beat[4555]: [2020-06-02 15:56:00,766: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/apps/beat.py", line 81, in run
funkwhale-beat[4555]: [2020-06-02 15:56:00,766: WARNING/MainProcess] self.start_scheduler()
funkwhale-beat[4555]: [2020-06-02 15:56:00,766: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/apps/beat.py", line 109, in start_scheduler
funkwhale-beat[4555]: [2020-06-02 15:56:00,767: WARNING/MainProcess] service.start()
funkwhale-beat[4555]: [2020-06-02 15:56:00,767: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 588, in start
funkwhale-beat[4555]: [2020-06-02 15:56:00,767: WARNING/MainProcess] humanize_seconds(self.scheduler.max_interval))
funkwhale-beat[4555]: [2020-06-02 15:56:00,767: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
funkwhale-beat[4555]: [2020-06-02 15:56:00,767: WARNING/MainProcess] value = obj.__dict__[self.__name__] = self.__get(obj)
funkwhale-beat[4555]: [2020-06-02 15:56:00,767: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 632, in scheduler
funkwhale-beat[4555]: [2020-06-02 15:56:00,768: WARNING/MainProcess] return self.get_scheduler()
funkwhale-beat[4555]: [2020-06-02 15:56:00,768: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 627, in get_scheduler
funkwhale-beat[4555]: [2020-06-02 15:56:00,768: WARNING/MainProcess] lazy=lazy,
funkwhale-beat[4555]: [2020-06-02 15:56:00,768: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 467, in __init__
funkwhale-beat[4555]: [2020-06-02 15:56:00,768: WARNING/MainProcess] Scheduler.__init__(self, *args, **kwargs)
funkwhale-beat[4555]: [2020-06-02 15:56:00,768: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 226, in __init__
funkwhale-beat[4555]: [2020-06-02 15:56:00,769: WARNING/MainProcess] self.setup_schedule()
funkwhale-beat[4555]: [2020-06-02 15:56:00,769: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 495, in setup_schedule
funkwhale-beat[4555]: [2020-06-02 15:56:00,769: WARNING/MainProcess] self._create_schedule()
funkwhale-beat[4555]: [2020-06-02 15:56:00,769: WARNING/MainProcess] File "/srv/funkwhale/virtualenv/lib/python3.7/site-packages/celery/beat.py", line 528, in _create_schedule
funkwhale-beat[4555]: [2020-06-02 15:56:00,769: WARNING/MainProcess] self._store[str('entries')] = {}
funkwhale-beat[4555]: [2020-06-02 15:56:00,769: WARNING/MainProcess] File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__
funkwhale-beat[4555]: [2020-06-02 15:56:00,770: WARNING/MainProcess] self.dict[key.encode(self.keyencoding)] = f.getvalue()
funkwhale-beat[4555]: [2020-06-02 15:56:00,770: WARNING/MainProcess] _dbm
funkwhale-beat[4555]: [2020-06-02 15:56:00,770: WARNING/MainProcess] .
funkwhale-beat[4555]: [2020-06-02 15:56:00,770: WARNING/MainProcess] error
funkwhale-beat[4555]: [2020-06-02 15:56:00,770: WARNING/MainProcess] :
funkwhale-beat[4555]: [2020-06-02 15:56:00,770: WARNING/MainProcess] cannot add item to database
funkwhale-beat[4555]: [2020-06-02 15:56:00,860: WARNING/MainProcess] Exception ignored in:
funkwhale-beat[4555]: [2020-06-02 15:56:00,860: WARNING/MainProcess] <function Shelf.__del__ at 0x7f4e0feb42f0>
funkwhale-beat[4555]: [2020-06-02 15:56:00,860: WARNING/MainProcess] Traceback (most recent call last):
funkwhale-beat[4555]: [2020-06-02 15:56:00,860: WARNING/MainProcess] File "/usr/lib/python3.7/shelve.py", line 162, in __del__
funkwhale-beat[4555]: [2020-06-02 15:56:00,861: WARNING/MainProcess] self.close()
funkwhale-beat[4555]: [2020-06-02 15:56:00,861: WARNING/MainProcess] File "/usr/lib/python3.7/shelve.py", line 144, in close
funkwhale-beat[4555]: [2020-06-02 15:56:00,861: WARNING/MainProcess] self.sync()
funkwhale-beat[4555]: [2020-06-02 15:56:00,861: WARNING/MainProcess] File "/usr/lib/python3.7/shelve.py", line 168, in sync
funkwhale-beat[4555]: [2020-06-02 15:56:00,862: WARNING/MainProcess] self[key] = entry
funkwhale-beat[4555]: [2020-06-02 15:56:00,862: WARNING/MainProcess] File "/usr/lib/python3.7/shelve.py", line 125, in __setitem__
funkwhale-beat[4555]: [2020-06-02 15:56:00,862: WARNING/MainProcess] self.dict[key.encode(self.keyencoding)] = f.getvalue()
funkwhale-beat[4555]: [2020-06-02 15:56:00,862: WARNING/MainProcess] _dbm
funkwhale-beat[4555]: [2020-06-02 15:56:00,862: WARNING/MainProcess] .
funkwhale-beat[4555]: [2020-06-02 15:56:00,862: WARNING/MainProcess] error
funkwhale-beat[4555]: [2020-06-02 15:56:00,862: WARNING/MainProcess] :
funkwhale-beat[4555]: [2020-06-02 15:56:00,862: WARNING/MainProcess] cannot add item to database
funkwhale-beat[4555]: celery beat v4.3.0 (rhubarb) is starting.
funkwhale-beat[4555]: __ - ... __ - _
funkwhale-beat[4555]: LocalTime -> 2020-06-02 15:56:00
funkwhale-beat[4555]: Configuration ->
funkwhale-beat[4555]: . broker -> redis://127.0.0.1:6379/1
funkwhale-beat[4555]: . loader -> celery.loaders.app.AppLoader
funkwhale-beat[4555]: . scheduler -> celery.beat.PersistentScheduler
funkwhale-beat[4555]: . db -> celerybeat-schedule
funkwhale-beat[4555]: . logfile -> [stderr]@%INFO
funkwhale-beat[4555]: . maxinterval -> 5.00 minutes (300s)