Want to take your software engineering career to the next level? Join the mailing list for career tips & advice Click here

dataset-examples

Samples for users of the Yelp Academic Dataset

Subscribe to updates I use dataset-examples


Statistics on dataset-examples

Number of watchers on Github 640
Number of open issues 14
Average time to close an issue 7 months
Main language Python
Average time to merge a PR 4 days
Open pull requests 1+
Closed pull requests 8+
Last commit almost 6 years ago
Repo Created almost 9 years ago
Repo Last Updated over 2 years ago
Size 56.6 MB
Homepage http://www.yelp.c...
Organization / Authoryelp
Contributors3
Page Updated
Do you use dataset-examples? Leave a review!
View open issues (14)
View dataset-examples activity
View on github
Book a Mock Interview With Me (Silicon Valley Engineering Leader, 100s of interviews conducted)
Software engineers: It's time to get promoted. Starting NOW! Subscribe to my mailing list and I will equip you with tools, tips and actionable advice to grow in your career.
Evaluating dataset-examples for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Build Status

Yelp's Academic Dataset Examples

We're providing three examples for use with the datasets available at http://www.yelp.com/dataset_challenge and http://www.yelp.com/academic_dataset. They all depend on mrjob and python 2.6 or later.

To install all dependencies: $ pip install -e .

To test: $ tox

Samples

json_to_csv_converter: Convert the dataset from json format to csv format.

$ python json_to_csv_converter.py yelp_academic_dataset.json # Creates yelp_academic_dataset.csv

category_predictor: Given some text, predict likely categories. For example:

$ python category_predictor/category_predictor.py yelp_academic_dataset.json > category_predictor.json
$ python category_predictor/predict.py category_predictor.json "bacon donut"
Category: "Food" - 82.66% chance
Category: "Restaurants" - 16.99% chance
Category: "Donuts" - 0.12% chance
Category: "Basque" - 0.02% chance
Category: "Spanish" - 0.02% chance

review_autopilot: Use a markov chain to finish a review. For example:

$ python review_autopilot/generate.py Food 'They have the best'
They have the best coffee is good food was delicious cookies and
a few friends i think they make this

positive_category_words: See the Yelp engineering blog for details about this example. In short, it generates positivity scores for words either globally or per-category.

Basic set-up

You can use any of mjrob's runner with these examples, but we'll focus on the local and emr runner (if you have access to your own hadoop cluster, check out the mrjob docs for instructions on how to set this up).

Local mode couldn't be easier:

# this step will take a VERY long time
python review_autopilot/autopilot.py yelp_academic_dataset.json > autopilot.json

# this should be instant
python review_autopilot/generate.py Food 'They have the best'
> hot dogs ever

Waiting a long time is kind of lame, no? Let's try the same thing using EMR.

First off, you'll need an aws_access_key and an aws_secret_access_key. You can get these from the AWS console (you'll need to sign up for an AWS developer account and enable s3 / emr usage, if you haven't already).

Create a simple mrjob.conf file, like this:

runners:
  emr:
    aws_access_key_id: YOUR_ACCESS_KEY
    aws_secret_access_key: YOUR_SECRET_KEY

Now that that's done, you can run the autopilot script on EMR.

# WARNING: this will cost you roughly $2 and take 10-20 minutes
python review_autopilot/autopilot.py --num-ec2-instances 10 --ec2-instance-type c1.medium -v --runner emr yelp_academic_dataset.json

You can save money (and time) by re-using jobflows and uploading the dataset to a personal, private s3 bucket - check out the mrjob docs for instructions on doing this.

dataset-examples open issues Ask a question     (View All Issues)
  • about 4 years Issue with JSON to CSV converter
  • over 4 years Quick question about data
  • over 4 years JSON_to_CSV_Converter
  • over 4 years Can not import the dataset into python
  • over 4 years simplejson module not found
  • over 5 years City has incorrect value
  • over 5 years Some columns are blank
dataset-examples open pull requests (View All Pulls)
  • changed import json statement to be compatible with Python 2.6+
dataset-examples list of languages used
Other projects in Python