onyx

Distributed, masterless, high performance, fault tolerant data processing

Subscribe to updates I use onyx


Statistics on onyx

Number of watchers on Github 1700
Number of open issues 89
Average time to close an issue 14 days
Main language Clojure
Average time to merge a PR 2 days
Open pull requests 17+
Closed pull requests 27+
Last commit 3 months ago
Repo Created over 4 years ago
Repo Last Updated 3 months ago
Size 16 MB
Homepage http://www.onyxpl...
Organization / Authoronyx-platform
Contributors25
Page Updated
Do you use onyx? Leave a review!
View open issues (89)
View onyx activity
View on github
Latest Open Source Launches
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating onyx for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Logo Onyx

Join the chat at https://gitter.im/onyx-platform/onyx

What is it?

  • a masterless, cloud scale, fault tolerant, high performance distributed computation system
  • batch and stream hybrid processing model
  • exposes an information model for the description and construction of distributed workflows
  • Competes against Storm, Flink, Cascading, Cascalog, Spark, Map/Reduce, Sqoop, etc
  • written in pure Clojure

What would I use this for?

  • Realtime event stream processing
  • CQRS
  • Continuous computation
  • Extract, transform, load
  • Data transformation la map-reduce
  • Data ingestion and storage medium transfer
  • Data cleaning

Installation

Available on Clojars:

[org.onyxplatform/onyx "0.12.7"]

Changelog

Changelog can be found at changes.md.

Quick Lookup Doc

A searchable set of documentation for the Onyx data model is available.

Plugins and Libraries

Plugin Use

To use the supported plugins, please use version coordinates such as [org.onyxplatform/onyx-amazon-sqs "0.12.7.0"], and read the READMEs on the 0.12.x branches linked above.

Build Status

Component release unstable
onyx core Circle CI Circle CI
onyx-local-rt Circle CI Circle CI
onyx-kafka Circle CI Circle CI
onyx-kafka-0.8 Circle CI Circle CI
onyx-datomic Circle CI Circle CI
onyx-redis Circle CI Circle CI
onyx-sql Circle CI Circle CI
onyx-bookkeeper Circle CI Circle CI
onyx-durable-queue Circle CI Circle CI
onyx-elasticsearch Circle CI Circle CI
onyx-amazon-sqs Circle CI Circle CI
onyx-amazon-s3 Circle CI Circle CI
onyx-http Circle CI Circle CI
learn-onyx Circle CI -
onyx-examples Circle CI Circle CI
onyx-dashboard Circle CI Circle CI
onyx-peer-http-query Circle CI Circle CI
lib-onyx Circle CI Circle CI
  • release: stable, released content
  • unstable: unreleased content

Unsupported plugins

Some plugins are currently unsupported in onyx 0.12.x. These are:

Companies Running Onyx in Production

LockedOn                                                                                               

Quick Start Guide

Feeling impatient? Hit the ground running ASAP with the onyx-starter repo and walkthrough. You can also boot into preloaded a Leiningen application template.

User Guide 0.12.7

Developer's Guide 0.12.7

API Docs 0.12.7

Code level API documentation can be found here.

Official plugin listing

Official plugins are vetted by Michael Drogalis. Ensure in your project that plugin versions directly correspond to the same Onyx version (e.g. onyx-kafka version 0.12.7.0-SNAPSHOT goes with onyx version 0.12.7). Fixes to plugins can be applied using a 4th versioning identifier (e.g. 0.12.7.1-SNAPSHOT).

Generate plugin templates through Leiningen with onyx-plugin.

3rd Party plugin listing

Unofficial plugins have not been vetted.

Offical Dashboard and Metrics

You can run a dashboard to monitor Onyx cluster activity, found here. Further, you can collect metrics and send them to the dashboard, or anywhere, by using the onyx-metrics plugin.

Need help?

Check out the Onyx Google Group.

Want the logo?

Feel free to use it anywhere. You can find a few different versions here.

Running the tests

A simple lein test will run the full suite for Onyx core.

Contributor list

Acknowledgements

Some code has been incorporated from the following projects:

License

Copyright 2017 Michael Drogalis

Distributed under the Eclipse Public License, the same as Clojure.

onyx open issues Ask a question     (View All Issues)
  • over 1 year core.async/size isn't used
  • over 1 year Batch function kills job on exceptions
  • over 1 year [ABS] Coordinator should probably only allow one snapshot to be occurring at a time
  • over 1 year [ABS] implement flow condition retry
  • over 1 year [ABS] Re-add handle-exception
  • over 1 year [ABS] implement closed Subscription / Publication handling
  • over 1 year [ABS] implement stream id allocation
  • over 1 year [ABS] implement network protocol buffers
  • over 1 year [ABS] S3 implementation of checkpointing
  • over 1 year [ABS] Coordinator should monitor publication positions of barriers
  • over 1 year Potential peer join ordering issue
  • over 1 year Generate plugin READMEs from their info models
  • over 1 year Upgrade rocksdb to support Windows OS
  • over 1 year [ABS] Log subscriber users have to maintain version information
  • over 1 year Link to companies in production via the "in prod" images
  • over 1 year Add test for trigger/fire-all-extents?
  • over 1 year Hanging/recurring methods
  • almost 2 years [ABS] thoughts on using aggregations as inputs for further processing steps
  • almost 2 years Share a single resource between many virtual peers
  • almost 2 years Copy button for all code examples in documentation
  • almost 2 years [ABS] Make window more usable
  • almost 2 years Allow job data to be kept in alternate storage engines
  • almost 2 years Move monitoring argument into peer-config
  • almost 2 years Breaking changes list
  • about 2 years Switch from RocksDB to LMDB
  • about 2 years Write up how the job submission process works on the peers
  • about 2 years Greedy job scheduler should allow more than one job to run
  • about 2 years Document recommended JVM settings
  • about 2 years Support parameterization of flow/transform
  • about 2 years Consider passing flow condition map into :flow/predicate and :flow/transform
onyx open pull requests (View All Pulls)
  • Improve state management and triggers #529 #530 #519 #520 #510 #507
  • Adds metadata to job submissions.
  • Renames :onyx/id to :onyx/tenancy-id
  • Validations offer suggestions
  • Better error message for lifecycle return type errors.
  • Update to aeron 0.9.5
  • Multiplex zk conn
  • Allow for multi-key :onyx/uniqueness-key if expressed as a vector
  • Bk cookie fix
  • Move monitoring config into peer config
  • First-cut of User Guide updates
  • WIP: filter using lmdb
  • WIP: Improve ZK connection state handling
  • Non blocking recovery process
  • Support custom checkpoint storage implementations
  • async checkpoint recovery.
  • minor copy-editing
onyx questions on Stackoverflow (View All Questions)
  • How to get battery level of Onyx ibeacons
  • Onyx crashed right after editing a VM's settings
  • Nonin Onyx II 9560 data
  • Best practice for parent/child-viewmodel-relationships in MVVM using Onyx?
onyx list of languages used
Other projects in Clojure