Technology moves fast! ⚡ Don't get left behind.🚶 Subscribe to our mailing list to keep up with latest and greatest in open source projects! 🏆


Subscribe to our mailing list

suro

Netflix's distributed Data Pipeline

Subscribe to updates I use suro


Statistics on suro

Number of watchers on Github 663
Number of open issues 32
Average time to close an issue 9 days
Main language Java
Average time to merge a PR about 9 hours
Open pull requests 5+
Closed pull requests 4+
Last commit over 3 years ago
Repo Created over 6 years ago
Repo Last Updated over 1 year ago
Size 4.28 MB
Organization / Authornetflix
Contributors12
Page Updated
Do you use suro? Leave a review!
View open issues (32)
View suro activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating suro for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Suro: Netflix's Data Pipeline

Suro is a data pipeline service for collecting, aggregating, and dispatching large volume of application events including log data. It has the following features:

  • It is distributed and can be horizontally scaled.
  • It supports streaming data flow, large number of connections, and high throughput.
  • It allows dynamically dispatching events to different locations with flexible dispatching rules.
  • It has a simple and flexible architecture to allow users to add additional data destinations.
  • It fits well into NetflixOSS ecosystem
  • It is a best-effort data pipeline with support of flexible retries and store-and-forward to minimize message loss

Learn more about Suro on the Suro Wiki and the Netflix TechBlog post where Suro was introduced.

Master Build Status

Pull Request Build Status

Build

NetflixGraph is built via Gradle (www.gradle.org). To build from the command line:

./gradlew build

See the build.gradle file for other gradle targets, like distTar, distZip, installApp, and runServer.

Running the server

You can run the server locally by just running ./gradlew runServer.

More more advanced usage you may wish to run ./gradlew installApp and then:

cd suro-server
java -cp "build/install/suro-server/lib/*" com.netflix.suro.SuroServer -m conf/routingmap.json -s conf/sink.json -i conf/input.json

To enable basic logging you can downloaded slf4j-simple-1.7.7.jar and copy it into suro-server then run:

cd suro-server
java -cp "build/install/suro-server/lib/*:slf4j-simple-1.7.7.jar" com.netflix.suro.SuroServer -m conf/routingmap.json -s conf/sink.json -i conf/input.json

Support

We will use the Google Group, Suro Users, to discuss issues: https://groups.google.com/forum/#!forum/suro-users

suro open issues Ask a question     (View All Issues)
  • about 3 years Suro with S3 sink
  • over 3 years Zookeeper dependencies are missing in suro-server install
  • over 4 years Make ElasticSearchSink IndexInfoBuilder more extendible
  • over 4 years routing map parser fails silently on malformed json
  • over 4 years Potential scalability problem with default RemotePrefixFormatter for S3 Sink
  • over 4 years Make AsyncSuroClient Rx
  • over 4 years Make QueuedSink Rx
  • almost 5 years Does Suro have replay functionality?
  • almost 5 years How can I add graphite with suro for monitor ?
  • almost 5 years the upload path is not reasonable when reconnect to hdfs/s3
  • almost 5 years the `rotationPeriod` not enough to full fill hourly/daily rotation
  • almost 5 years SocketException occurs when client send ping to sure server
  • almost 5 years does suro keep the order of message send ?
  • about 5 years log4 appender and eureka lb server issue
  • about 5 years Simple data conversion through groovy script
  • about 5 years Implement a Python client
  • about 5 years ConnectionPool can throw ConcurrentModificationException
  • about 5 years Artifact 'com.google.code.findbugs:jsr305:1.3.9@jar' not found.
  • over 5 years ./gradlew clean build fails on Amazon Linux
  • over 5 years TagKey contains metrics name
  • almost 6 years JavaRx message router
  • almost 6 years EventBus filter
  • almost 6 years ForkJoin MessageQueue
  • almost 6 years Disruptor for MessageQueue
  • almost 6 years MessageQueue
  • almost 6 years Injected message producer
  • almost 6 years Basic producer interface
suro open pull requests (View All Pulls)
  • Added support for and AvroFileWriter
  • Created input stats endpoint
  • use suro file's lastModify/dateCreated time for build upload path
  • support truncate suro file hourly/daily
  • Feature/marathon support
suro questions on Stackoverflow (View All Questions)
  • Netflix Suro key features
suro list of languages used
Other projects in Java