I’ve maintained the gem sidekiq-unique-jobs for a long time. Since July 26, 2012. A lot of work has gone into the gem. It has been quite a journey and I appreciate the challenge.
Version 6 was unfortunately an arrogant move by me. I had the best of intentions but to be completely frank; I produced a version that wasn’t production ready.
Let’s talk about some of the challenges with building a Redis based locking system.
- Sidekiq has two processes. A client process that enqueues jobs to be processed and a server process that picks jobs off of that queue to process them (run your code). Locking or uniqueness doesn’t just involve when your code is running. We need to take into consideration the Sidekiq retry queue and scheduled set.
- Worker restarts. When a worker restarts it is difficult to handle the lock. In some situations the lock should stay and in some the lock should be removed.
- Redis is in-memory. The nature of Redis is to be fast, blazing fast… That means it will not immediately persist your data to disk. Too stay fast, it will store it in memory and periodically write its contents to disk.
In version five I discovered that uniqueness was actually bypassed in many cases, so with version six I wanted to make absolutely sure that duplicates were truly prevented.
While duplicates were indeed prevented people have been running into issues with locks never being released. This caused a lot of frustration and even though it was possible to find and delete orphaned locks manually it wasn’t a smooth enough experience.
Version seven takes the good ideas from version six and makes them reliably awesome.
These are a few of the new features that I’ve been working on: