Want to take your software engineering career to the next level? Join the mailing list for career tips & advice Click here


Web interface for browsing, search and filtering recent arxiv submissions

Subscribe to updates I use arxiv-sanity-preserver

Statistics on arxiv-sanity-preserver

Number of watchers on Github 3853
Number of open issues 73
Average time to close an issue about 1 month
Main language Python
Average time to merge a PR 11 days
Open pull requests 23+
Closed pull requests 7+
Last commit over 1 year ago
Repo Created almost 5 years ago
Repo Last Updated 3 months ago
Size 977 KB
Homepage http://www.arxiv-...
Organization / Authorkarpathy
Page Updated
Do you use arxiv-sanity-preserver? Leave a review!
View open issues (73)
View arxiv-sanity-preserver activity
View on github
Book a Mock Interview With Me (Silicon Valley Engineering Leader, 100s of interviews conducted)
Software engineers: It's time to get promoted. Starting NOW! Subscribe to my mailing list and I will equip you with tools, tips and actionable advice to grow in your career.
Evaluating arxiv-sanity-preserver for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

arxiv sanity preserver

This project is a web interface that attempts to tame the overwhelming flood of papers on Arxiv. It allows researchers to keep track of recent papers, search for papers, sort papers by similarity to any paper, see recent popular papers, to add papers to a personal library, and to get personalized recommendations of (new or old) Arxiv papers. This code is currently running live at www.arxiv-sanity.com/, where it's serving 25,000+ Arxiv papers from Machine Learning (cs.[CV|AI|CL|LG|NE]/stat.ML) over the last ~3 years. With this code base you could replicate the website to any of your favorite subsets of Arxiv by simply changing the categories in fetch_papers.py.

user interface

Code layout

There are two large parts of the code:

Indexing code. Uses Arxiv API to download the most recent papers in any categories you like, and then downloads all papers, extracts all text, creates tfidf vectors based on the content of each paper. This code is therefore concerned with the backend scraping and computation: building up a database of arxiv papers, calculating content vectors, creating thumbnails, computing SVMs for people, etc.

User interface. Then there is a web server (based on Flask/Tornado/sqlite) that allows searching through the database and filtering papers by similarity, etc.


Several: You will need numpy, feedparser (to process xml files), scikit learn (for tfidf vectorizer, training of SVM), flask (for serving the results), flask_limiter, and tornado (if you want to run the flask server in production). Also dateutil, and scipy. And sqlite3 for database (accounts, library support, etc.). Most of these are easy to get through pip, e.g.:

$ virtualenv env                # optional: use virtualenv
$ source env/bin/activate       # optional: use virtualenv
$ pip install -r requirements.txt

You will also need ImageMagick and pdftotext, which you can install on Ubuntu as sudo apt-get install imagemagick poppler-utils. Bleh, that's a lot of dependencies isn't it.

Processing pipeline

The processing pipeline requires you to run a series of scripts, and at this stage I really encourage you to manually inspect each script, as they may contain various inline settings you might want to change. In order, the processing pipeline is:

  1. Run fetch_papers.py to query arxiv API and create a file db.p that contains all information for each paper. This script is where you would modify the query, indicating which parts of arxiv you'd like to use. Note that if you're trying to pull too many papers arxiv will start to rate limit you. You may have to run the script multiple times, and I recommend using the arg --start-index to restart where you left off when you were last interrupted by arxiv.
  2. Run download_pdfs.py, which iterates over all papers in parsed pickle and downloads the papers into folder pdf
  3. Run parse_pdf_to_text.py to export all text from pdfs to files in txt
  4. Run thumb_pdf.py to export thumbnails of all pdfs to thumb
  5. Run analyze.py to compute tfidf vectors for all documents based on bigrams. Saves a tfidf.p, tfidf_meta.p and sim_dict.p pickle files.
  6. Run buildsvm.py to train SVMs for all users (if any), exports a pickle user_sim.p
  7. Run make_cache.py for various preprocessing so that server starts faster (and make sure to run sqlite3 as.db < schema.sql if this is the very first time ever you're starting arxiv-sanity, which initializes an empty database).
  8. Start the mongodb daemon in the background. Mongodb can be installed by following the instructions here - https://docs.mongodb.com/tutorials/install-mongodb-on-ubuntu/.
    • Start the mongodb server with - sudo service mongod start.
    • Verify if the server is running in the background : The last line of /var/log/mongodb/mongod.log file must be - [initandlisten] waiting for connections on port <port>
  9. Run the flask server with serve.py. Visit localhost:5000 and enjoy sane viewing of papers!

Optionally you can also run the twitter_daemon.py in a screen session, which uses your Twitter API credentials (stored in twitter.txt) to query Twitter periodically looking for mentions of papers in the database, and writes the results to the pickle file twitter.p.

I have a simple shell script that runs these commands one by one, and every day I run this script to fetch new papers, incorporate them into the database, and recompute all tfidf vectors/classifiers. More details on this process below.

protip: numpy/BLAS: The script analyze.py does quite a lot of heavy lifting with numpy. I recommend that you carefully set up your numpy to use BLAS (e.g. OpenBLAS), otherwise the computations will take a long time. With ~25,000 papers and ~5000 users the script runs in several hours on my current machine with a BLAS-linked numpy.

Running online

If you'd like to run the flask server online (e.g. AWS) run it as python serve.py --prod.

You also want to create a secret_key.txt file and fill it with random text (see top of serve.py).

Current workflow

Running the site live is not currently set up for a fully automatic plug and play operation. Instead it's a bit of a manual process and I thought I should document how I'm keeping this code alive right now. I have a script that performs the following update early morning after arxiv papers come out (~midnight PST):

python fetch_papers.py
python download_pdfs.py
python parse_pdf_to_text.py
python thumb_pdf.py
python analyze.py
python buildsvm.py
python make_cache.py

I run the server in a screen session, so screen -S serve to create it (or -r to reattach to it) and run:

python serve.py --prod --port 80

The server will load the new files and begin hosting the site. Note that on some systems you can't use port 80 without sudo. Your two options are to use iptables to reroute ports or you can use setcap to elavate the permissions of your python interpreter that runs serve.py. In this case I'd recommend careful permissions and maybe virtualenv, etc.

arxiv-sanity-preserver open issues Ask a question     (View All Issues)
  • about 4 years Convert does not work on Mac OS
  • about 4 years library not showing my saved papers.
  • about 4 years Use file list instead of database in `analyze.py`
  • over 4 years add support for http://www.jmlr.org/
  • over 4 years cs.AI
  • over 4 years Add a LICENSE file, please.
  • over 4 years Analyzing uses too much memory
  • over 4 years Hosting "fork" for physics categories
  • over 4 years Would it make sense to add LSA/LDA vec to TF IDF representation?
  • over 4 years Feature Requests/Suggestions
  • over 4 years arxiv paper ios app
  • over 4 years Would it be useful to have an extension to this project where you can see the ancestors and predecessors of a research paper?
  • almost 5 years Any plan for including other fields? (and some suggestion about social features)
  • almost 5 years Stemming, ELK
  • almost 5 years mention poppler dependency in readme
  • almost 5 years Any plans for adding feature of commenting on papers?
arxiv-sanity-preserver open pull requests (View All Pulls)
  • Better text extraction quality by PDFLib Text and Image Extraction Toolkit(TET)
  • paper titles link to single view
  • Two features I implemented locally
  • Multiprocessing for thumbnails.
  • Use latest pickle data stream version
  • Add Dockerfile and supporting script
  • Fix #49 by adding `./` to relative path
  • Adding License File;
  • Make codebase python3 compatible;
  • Improve ios layout
  • Added pymongo dependency for serve and twitter_daemon
  • SEC,PRF: templates/main.html: build w/ DOM nodes
  • py3
  • Add arXiv vanity links to the papers
  • Variable file is not defined -> Rename to f
  • add search to pages: top recent, top hype, friends, recommended, library
  • Allows searching by arXiv ID.
  • RSS feed that will publish all papers whose updated datetime is newer than yesterday.
  • Fixed Issue #102 Use "sys.exit()" without "import sys" in thumb_pdf.py line 16
  • Updated readme instructions
  • accept paper IDs with or without a dot
  • fixed results_per_iteration miscount
  • Fix link in readme
arxiv-sanity-preserver list of languages used
Other projects in Python
Powered by Autocode - Instant Webhooks, Scripts and APIs
Autocode logo wordmark