Sidekiq is the gold standard when it comes to background processing. I have been using since I started working with rails. Sidekiq took care of lot of the complexity of the background processing that allowed developers to concentrate on just there code/business logic.
One of the beautiful features of sidekiq was how it handled exit (SIGTERM and SIGKILL).
When sidekiq received the sigterm commands it stops accept new jobs, and just focus on finishing the job it is doing now and incase it can’t finish, it will requeue the job to another available worker or keep it in the queue for another worker (or the same worker) to pick it up when available. This was working so well for years on rails app I was working on, that I didn’t even care about it. The mina command would trigger a sidekiq restart allowing sidekiq to gracefully exit and then respawn with the new code.
The above app was running on two linux servers everything manually configured, but recently to keep up with the growth of the app we moved to amazon ECS. The app was containerized to a docker image, and made to run on about 12 Fargate Instances with ability to auto scale to 36 based on CPU and RAM usage. We grouped instances for various purposes 2 for heavy jobs, 2 for faster (more memory jobs), 2 for file processing, 4 for web, 2 for API.