Are you happy with your logging solution? Would you help us out by taking a 30-second survey? Click here

ambrose

A platform for visualization and real-time monitoring of data workflows

Subscribe to updates I use ambrose


Statistics on ambrose

Number of watchers on Github 1176
Number of open issues 39
Average time to close an issue about 2 months
Main language Java
Average time to merge a PR 8 days
Open pull requests 5+
Closed pull requests 5+
Last commit over 3 years ago
Repo Created over 7 years ago
Repo Last Updated over 1 year ago
Size 13.8 MB
Homepage http://twitter.co...
Organization / Authortwitter
Contributors18
Page Updated
Do you use ambrose? Leave a review!
View open issues (39)
View ambrose activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating ambrose for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Ambrose Build Status

Twitter Ambrose is a platform for visualization and real-time monitoring of MapReduce data workflows. It presents a global view of all the map-reduce jobs derived from your workflow after planning and optimization. As jobs are submitted for execution on your Hadoop cluster, Ambrose updates its visualization to reflect the latest job status.

Ambrose provides the following in a web UI:

  • A workflow progress bar depicting percent completion of the entire workflow
  • A table view of all workflow jobs, along with their current state
  • A graph diagram which depicts job dependencies and metrics
    • Visual weighting of jobs based on resource consumption
    • Visual weighting of job dependencies based on data volume
  • Script view with line highlighting (Pig only)

Ambrose is built using the following front-end technologies:

Ambrose is designed to support any workflow runtime. See the following section for supported runtimes.

Follow @Ambrose on Twitter to stay in touch!

Supported runtimes

Examples

Below is a screenshot of the Ambrose workflow UI. The interface presents multiple responsive views of a single workflow. Just beneath the toolbar at the top of the window is a workflow progress bar that tracks overall completion of the workflow. Below the progress bar is a graph diagrams which depicts the workflow's jobs and their dependencies. Below the graph diagram is a table of workflow jobs.

All views react to mouseover and click events on a job, regardless of the view on which the event is triggered; Moving your mouse over the first row of the table will highlight that job's table row along with the job's node in the graph diagram. Clicking on a job in any view will select it, updating the highlighting of that job in all views. Clicking again on the same job will deselect it.

Ambrose workflow screenshot

Quickstart

To get started with Ambrose, first clone the Ambrose Github repository:

git clone https://github.com/twitter/ambrose.git
cd ambrose

Next, you can try running the Ambrose demo on your local machine. The demo starts a local web server which serves the front-end client resources and sample data. Start the demo with the following command and then browse to http://localhost:8080/workflow.html?localdata=large:

./bin/ambrose-demo

To run Ambrose with a Pig script, you'll need to build the Ambrose Pig distribution:

mvn package

You can then run the following commands to execute script.pig with an embedded web server which hosts the Ambrose web application:

cd pig/target/ambrose-pig-$VERSION-bin/ambrose-pig-$VERSION
AMBROSE_PORT=8080 ./bin/pig-ambrose -f script.pig

Note that the pig-ambrose script calls the pig script present in your local installation of Pig, so make sure $PIG_HOME/bin is in your path. Now, browse to http://localhost:8080/web/workflow.html to see the progress of your script with the Ambrose workflow UI.

Maven repository

Ambrose releases can be found in the Maven Central Repository within package com.twitter.ambrose.

How to contribute

Bug fixes, features, and documentation improvements are welcome! Please fork the project and send us a pull request on Github. You can submit issues on Github as well.

Here are some high-level goals we'd love to see contributions for:

  • Improve the front-end client
  • Add other visualization options
  • Create a new back-end for a different runtime environment
  • Create a standalone Ambrose server that's not embedded in the workflow client

Versioning

For transparency and insight into our release cycle, releases will be numbered with the follow format:

<major>.<minor>.<patch>

And constructed with the following guidelines:

  • Breaking backwards compatibility bumps the major
  • New additions without breaking backwards compatibility bumps the minor
  • Bug fixes and misc changes bump the patch

For more information on semantic versioning, please visit http://semver.org/.

Authors

License

Copyright 2013 Twitter, Inc.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

ambrose open issues Ask a question     (View All Issues)
  • over 3 years Issue while writing JSON to the file using ambrose-hive
  • over 3 years Using Ambrose with Hive 0.14
  • over 4 years When running the demo.pig script by pig works fine but when running it with ./pig-ambrose -f demo.pig I keep getting this error. I am using pig 0.12.0.
  • over 4 years Ambrose fails to work for Hive demo query
  • about 5 years Ambrose doesn't work with Pig 0.13/0.14
  • about 5 years NoClassDefFoundError on JSONUtil
  • over 5 years Failed to retrieve clusters from backend - Hortonworks HDP2
  • over 5 years Ambrose DAG data model
  • over 5 years ambrose-hive: bump up to Hive 0.13
  • over 5 years mvn eclipse:eclipse does not create projects for ambrose properly
  • over 5 years Hosted dashboard requires pig and cascading jars in the classpath
  • over 5 years AmbrosePPNL should have another interface in middle to compute workflowId based on jobConf
  • over 5 years Documentation of JSON-Files
  • over 5 years AmbrosePPNL should not poll stats from jobtracker
  • over 5 years Cascading stats are not reported correctly (with hadoop 1)
  • over 6 years Adds job physical plans to workflow ui
  • over 6 years Add physical plans to job confs for pig jobs
  • over 6 years Selection of job metric tracked by magnitude circles in graph view
  • over 6 years Deal with ctrl-c and job / workflow state transition
  • over 6 years Some completed jobs never reach 100% progress
  • over 6 years Some job events may have bad metrics (e.g. totalMappers)
  • over 6 years Event replay in the client UI should be optional
  • over 6 years Could not resolve com.twitter.ambrose.pig.EmbeddedAmbrosePigProgressNotificationListener
  • over 6 years Failed to execute goal on project ambrose-common: Could not resolve dependencies for project com.twitter.ambrose:ambrose-common:jar:0.1.1-SNAPSHOT: The following artifacts could not be resolved: log4j:log4j:jar:1.2.16, hsqldb:hsqldb:jar:1.8.0.10: Could n
  • over 6 years New feature : Adding MapReduce plan for each job.
  • over 6 years pig-ambrose does not pick up conf from PIG_CLASSPATH
  • over 6 years further modernization, modularization of javascript
  • over 6 years potential guava version conflict
  • about 7 years Failed to load dag data
  • over 7 years Loading UI before initial job submission stalls UI
ambrose open pull requests (View All Pulls)
  • Cleaning of code by fixing warnings
  • Add shutdown method to StatsWriteService
  • Utility method to wrap PPNL in dynamic proxy that ignores all the exceptions
  • New feature : Adding subgraph (MapReduce plan) for each job
  • Add cascading3 support
ambrose list of languages used
Other projects in Java