Want to take your software engineering career to the next level? Join the mailing list for career tips & advice Click here


Go library for writing standalone Map/Reduce jobs or for use with Hadoop's streaming protocol

Subscribe to updates I use dmrgo

Statistics on dmrgo

Number of watchers on Github 95
Number of open issues 0
Main language Go
Open pull requests 0+
Closed pull requests 1+
Last commit about 6 years ago
Repo Created over 8 years ago
Repo Last Updated about 2 years ago
Size 176 KB
Organization / Authordgryski
Page Updated
Do you use dmrgo? Leave a review!
View dmrgo activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Software engineers: It's time to get promoted. Starting NOW! Subscribe to my mailing list and I will equip you with tools, tips and actionable advice to grow in your career.
Evaluating dmrgo for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

dmrgo is a Go library for writing map/reduce jobs.

It can be used with Hadoop's streaming protocol, but also includes a standalone map/reduce implementation (including partitioner) for 'small' jobs (~5G-10G).

It is partially based on ideas from Yelp's MrJob package for Python, but since the Go is statically typed I've tried to make the API match more closely with Hadoop's Java API.

The traditional word count example is in the examples directory.

This code is licensed under the GPLv3, or at your option any later version.

Further reading:

MrJob: http://packages.python.org/mrjob/ https://github.com/Yelp/mrjob

Hadoop map/reduce tutorial: http://hadoop.apache.org/common/docs/current/mapred_tutorial.html

Hadoop streaming protocol: http://hadoop.apache.org/common/docs/current/streaming.html

dmrgo list of languages used
Other projects in Go