Sidekiq Pro

Create a set of jobs which can be tracked as a group, programmatically or within the Sidekiq Web interface. You can see success and failure counts along with error information for failed jobs within the batch.

We use batches at The Clymb when uploading product catalogs to our database. Each product becomes a job, uploading 1000 products makes 1000 jobs, which are quickly processed by our swarm of Sidekiq workers. Once finished, the catalog team is notified of the results.

Are you a human? Receive an email or campfire notification when a batch has completed processing.

Are you a robot? Receive a webhook or in-process callback when the batch is complete. Use Redis pub/sub to receive real-time updates as your batch processes.

Want support for other notification mediums? Contact me and let me know.

If a Sidekiq process crashes while processing a job, that job is lost.

Sidekiq Pro uses Redis's RPOPLPUSH command to ensure that jobs will not be lost if the process crashes or gets a KILL signal.

If the Sidekiq client gets a networking error while pushing a job to Redis, an exception is raised.

The Sidekiq Pro client can withstand transient Redis outages. It will enqueue jobs locally upon error and attempt to deliver those jobs once connectivity is restored.

Sidekiq Pro can send processing time and success/failure metrics to Statsd for visualization within tools like Graphite, Librato Metrics, etc. Here's a simple graph showing processing time for two different types of workers over time.