Technology moves fast! ⚡ Don't get left behind.🚶 Subscribe to our mailing list to keep up with latest and greatest in open source projects! 🏆


Subscribe to our mailing list

researchlei

An Academic Papers Management and Discovery System

Subscribe to updates I use researchlei


Statistics on researchlei

Number of watchers on Github 134
Number of open issues 3
Main language Python
Average time to merge a PR about 2 hours
Open pull requests 1+
Closed pull requests 0+
Last commit about 6 years ago
Repo Created about 6 years ago
Repo Last Updated over 1 year ago
Size 215 KB
Organization / Authorkarpathy
Contributors1
Page Updated
Do you use researchlei? Leave a review!
View open issues (3)
View researchlei activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating researchlei for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Research Lei

This page contains code for Research Lei: http://cs.stanford.edu/people/karpathy/researchlei/

Google Group: https://groups.google.com/forum/?fromgroups#!forum/research-lei

Feel free to contact me for questions, suggestions on Twitter: https://twitter.com/karpathy or via email: karpathy@cs.stanford.edu

Installation

  1. Clone this repository git clone https://github.com/karpathy/researchlei.git

  2. (Optional) Make sure you have ImageMagick installed on your system if you'd like to extract image thumbnails from downloaded papers. In Ubuntu, this is available as sudo apt-get install imagemagick

  3. (Optional) Install pdftotext. This is included by default in many Linux distributions. This tool is used to extract all the words from a paper and find the top 100. Later, this can be used for other fancy processing, such as topic models, tfidf similarity rankings, etc.

  4. You will need Python, preferrably Python 2.7.

  5. Obtain Microsoft Academic Search API APP ID key and place it into a file appid.txt. Since App ID's are rate limited to 200 queries per minute, I would strongly encourage you to obtain your own key from Microsoft (the request involves a single email and they reply fast). However, if you'd only like to check it out first for a bit, fill out this form.

Usage

  1. You start with an empty database in the beginning. To add a paper, run, for example: python addpaper.py name building rome in a day. This gets the addpaper.py script to search Microsoft Academic Search by name for a paper with a title that contains the query words building, rome, in, a, day. The script will then guide you through downloading its citations, reference, and the actual .pdf of the paper. (Sadly, you may find that Microsoft's Academic Search is sparser than Google Scholar, especially with more recent work :( I contacted them about this and they said they are working on an update to their index. Unfortunately, Google Scholar does not provide convenient API, makes scraping difficult, and does not provide information that is as complete.)

  2. The main Python script addpaper.py creates a JSON file that the client/index.html renders for the UI. Open it to see your library (remember to refresh it too every time you run addpaper.py)! Some browsers like Safari and Chrome will not (by default) allow you to do an AJAX call to read the local JSON file. This can be fixed by starting Chrome with a special flag (--allow-file-access-from-files). In Ubuntu, you can drag the Chrome icon to desktop, right click -> properties and append it to Command. Alternatively, just use Firefox!

Usage Example TLDR

  1. add a paper to library: python addpaper.py name building rome in a day
  2. view library: open client/index.html
  3. follow instructions in interface to download any specific paper of interest, or goto 0. to add a custom new paper

FAQ

Q: Microsoft Academic Search is sparser than Google Scholar, especially on very recent work. Could you use Scholar instead? A: No. Google Scholar does not provide API that allows you to easily download their structured data. It is possible to scrape the HTML manually, but they aggressively throttle your requests. Even if you could, they don't provide as much data. For example, they don't provide references, abstracts, etc. Or if there are more than a few authors on a paper, they simply write ... and refer you to a publisher's page for the paper. I wish Google Scholar was as cool as Microsoft Academic Search. However, I've written MAS about this issue and they told me that they are actively working to index more papers and that we should all stay tuned. Lets hope for the best.

Licence

BSD licence

researchlei open issues Ask a question     (View All Issues)
  • almost 6 years Microsoft Academic Search is being retired
  • about 6 years Integrate with something like CiteULike?
researchlei open pull requests (View All Pulls)
  • fixed spelling and added a missing letter/word
researchlei list of languages used
Other projects in Python