Are you happy with your logging solution? Would you help us out by taking a 30-second survey? Click here

sneakers

A fast background processing framework for Ruby and RabbitMQ

Subscribe to updates I use sneakers


Statistics on sneakers

Number of watchers on Github 1620
Number of open issues 41
Average time to close an issue 2 months
Main language Ruby
Average time to merge a PR 12 days
Open pull requests 31+
Closed pull requests 17+
Last commit over 1 year ago
Repo Created almost 6 years ago
Repo Last Updated over 1 year ago
Size 1.27 MB
Homepage http://sneakers.io
Organization / Authorjondot
Latest Releasev2.6.0
Contributors39
Page Updated
Do you use sneakers? Leave a review!
View open issues (41)
View sneakers activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating sneakers for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Sneakers

Build Status

      __
  ,--'  >  
  `=====   

A high-performance RabbitMQ background processing framework for Ruby.

Sneakers is being used in production for both I/O and CPU intensive workloads, and have achieved the goals of high-performance and 0-maintenance, as designed.

Visit the wiki for documentation and GitHub releases for release notes.

Build Status

Installation

Add this line to your application's Gemfile:

gem 'sneakers'

And then execute:

$ bundle

Or install it yourself as:

$ gem install sneakers

Quick start

Set up a Gemfile

source 'https://rubygems.org'
gem 'sneakers'
gem 'json'
gem 'redis'

How do we add a worker? Firstly create a file and name it as boot.rb then create a worker named as Processor.

touch boot.rb

require 'sneakers'
require 'redis'
require 'json'

$redis = Redis.new

class Processor
  include Sneakers::Worker
  from_queue :logs


  def work(msg)
    err = JSON.parse(msg)
    if err["type"] == "error"
      $redis.incr "processor:#{err["error"]}"
    end

    ack!
  end
end

Let's test it out quickly from the command line:

$ sneakers work Processor --require boot.rb

We just told Sneakers to spawn a worker named Processor, but first --require a file that we dedicate to setting up environment, including workers and what-not.

If you go to your RabbitMQ admin now, you'll see a new queue named logs was created. Push a couple messages like below:

{
   "type": "error",
   "message": "HALP!",
   "error": "CODE001"
}

And this is the output you should see at your terminal.

2013-10-11T19:26:36Z p-4718 t-ovqgyb31o DEBUG: [worker-logs:1:213mmy][#<Thread:0x007fae6b05cc58>][logs][{:prefetch=>10, :durable=>true, :ack=>true, :heartbeat_interval=>2, :exchange=>"sneakers"}] Working off: log log
2013-10-11T19:26:36Z p-4718 t-ovqgyrxu4 INFO: log log
2013-10-11T19:26:40Z p-4719 t-ovqgyb364 DEBUG: [worker-logs:1:h23iit][#<Thread:0x007fae6b05cd98>][logs][{:prefetch=>10, :durable=>true, :ack=>true, :heartbeat_interval=>2, :exchange=>"sneakers"}] Working off: log log
2013-10-11T19:26:40Z p-4719 t-ovqgyrx8g INFO: log log

We'll count errors and error types with Redis.

$ redis-cli monitor
1381520329.888581 [0 127.0.0.1:49182] "incr" "processor:CODE001"

We're basically done with the ceremonies and all is left is to do some real work.

Looking at metrics

Let's use the logging_metrics provider just for the sake of fun of seeing the metrics as they happen.

# boot.rb
require 'sneakers'
require 'redis'
require 'json'
require 'sneakers/metrics/logging_metrics'
Sneakers.configure :metrics => Sneakers::Metrics::LoggingMetrics.new

# ... rest of code

Now push a message again and you'll see:

2013-10-11T19:44:37Z p-9219 t-oxh8owywg INFO: INC: work.Processor.started
2013-10-11T19:44:37Z p-9219 t-oxh8owywg INFO: TIME: work.Processor.time 0.00242
2013-10-11T19:44:37Z p-9219 t-oxh8owywg INFO: INC: work.Processor.handled.ack

Which increments started and handled.ack, and times the work unit.

From here, you can continue over to the Wiki

Docker

If you use Docker, there's some benefits to be had and you can use both docker and docker-compose with this project, in order to run tests, integration tests or a sample worker without setting up RabbitMQ or the environment needed locally on your development box.

  • To build a container run docker build .
  • To run non-integration tests within a docker container, run docker run --rm sneakers_sneakers:latest
  • To run full integration tests within a docker topology including RabbitMQ, Redis (for integration worker) run scripts/local_integration, which will use docker-compose to orchestrate the topology and the sneakers Docker image to run the tests
  • To run a sample worker within Docker, try the TitleScraper example by running script/local_worker. This will use docker-compose as well. It will also help you get a feeling for how to run Sneakers in a Docker based production environment
  • Use Dockerfile.slim instead of Dockerfile for production docker builds. It generates a more compact image, while the regular Dockerfile generates a fatter image - yet faster to iterate when developing

Compatibility

  • Sneakers 1.1.x and up (using the new generation Bunny 2.x) - Ruby 2.x.x
  • Sneakers 1.x.x and down - Ruby 1.9.x, 2.x.x

Contributing

Fork, implement, add tests, pull request, get my everlasting thanks and a respectable place here :).

Thanks:

To all Sneakers Contributors - you make this happen, thanks!

Copyright

Copyright (c) 2015 Dotan Nahum @jondot. See LICENSE for further details.

sneakers open issues Ask a question     (View All Issues)
  • almost 3 years deploy worker on heroku rake sneakers:run exits status 0
  • almost 3 years Receiving "PG::DuplicatePstatement: ERROR: prepared statement a11 already exists" when workers execute
  • almost 3 years Signal for restart without killing sneaker process
  • almost 3 years Warn of deprecated config options
  • almost 3 years Changelog?
  • almost 3 years Local configuration occurred error
  • almost 3 years Any way to log sneakers logs remotely?
  • about 3 years Worker instance gets re-used across different threads
  • about 3 years Using sneakers - Rails 5 - Active Job
  • about 3 years Skips records to enqueue from result set
  • about 3 years How to have child processes catch a kill signal?
  • about 3 years bunny gem reference is out of date
  • about 3 years Code execution 8x slower using sneakers
  • about 3 years TLS verify_peer flag ignored
  • about 3 years Incorrect concurrent use of Bunny channel in handler/workers?
  • about 3 years Possible to Pry?
  • about 3 years It looks like rails is locking the sneaker log
  • about 3 years Publisher doesn't declare/bind queue which could lead to lost messages
  • about 3 years Always timeout when i use active-record
  • over 3 years Mongoid failed with error nil
  • over 3 years Worker restart race condition can leave orphaned processes
  • over 3 years publisher and mandatory option
  • over 3 years Road to v3.0
  • over 3 years Worker breaks down with execution expired
  • over 3 years serial message processing
  • over 3 years Comma-separated backtrace is hard to read
  • over 3 years Problems with "How to: running a stand alone worker" example
  • over 3 years Unexpected error Trying to send frame through a closed connection
  • over 3 years Sneakers::Publisher leaves channels open while still creating new ones
  • almost 4 years Worker generator
sneakers open pull requests (View All Pulls)
  • Adding attr_reader of channel into queue class so that it maybe accessible via queue inside worker
  • Update copyright notice to 2016 [ci skip]
  • Maxretry more flexible
  • use #with_channel rather than maintaining a persistent channel and exchange
  • Add Sneakers::ErrorReporter
  • Support for fatal (non-retriable) errors
  • Update the gemspec to the newest version of `thread`.
  • Bunny 2.1.0
  • Content type (de)serialization
  • Cast exception class to String in maxretry handler
  • Redacting the password out of the logging
  • Now one can customize pid path for each work group. see #98
  • Hooks after message is processed
  • allow classes to be an array
  • Add another maxretry handler which does not create additional exchanges
  • Handle passing serverengine config in from runner
  • Increased default AMQP heartbeat
  • Bump Bunny to 2.3.x
  • Allow to pass Proc as worker classes
  • Add log and pid_path as options for the CLI
  • include headers in Maxretry Handler error messages, plus a rake task to retry failed messages
  • [WIP] [ISSUE-171] add default logger
  • Ensure queue exists before publishing to avoid lost messages
  • Remove Dead code
  • Replace gem thread with concurrent-ruby
  • Exp backoff
  • Fix shutdown
  • added shared connection option for worker group
  • Bumped bunny version
  • Worker group ignores connection option
  • Set hook to run before and after the Worker processes the message
sneakers questions on Stackoverflow (View All Questions)
  • RabbitMq bunny gem vs sneakers
  • How to correctly set timeout on sneakers worker
  • ActionMailer in Sneakers Rake Task
sneakers list of languages used
sneakers latest release notes
v2.6.0 v2.6.0
  • Added licence type to gemspec #274
  • Serialization content type configuration #152
  • Add NewRelic category to example #273
  • Pass additional global Bunny session configuration #276
  • Sneaker.spawn exits with exit_code 1 when no configuration is provided #280
  • Added opt-in option for sharing bunny connections for worker group #266
  • Bumped bunny version from 2.6.* to 2.7.*. #289
  • Relaced thread gem with concurrent-ruby gem #279
  • Fixed stack level to deep error when trying to serialize the exception object in maxretry handler #308
v2.5.0
Other projects in Ruby