Want to take your software engineering career to the next level? Join the mailing list for career tips & advice Click here


Distributed Neural Networks for Spark

Subscribe to updates I use SparkNet

Statistics on SparkNet

Number of watchers on Github 601
Number of open issues 28
Average time to close an issue 7 days
Main language C++
Average time to merge a PR about 16 hours
Open pull requests 6+
Closed pull requests 7+
Last commit over 4 years ago
Repo Created over 4 years ago
Repo Last Updated 17 days ago
Size 7.49 MB
Organization / Authoramplab
Page Updated
Do you use SparkNet? Leave a review!
View open issues (28)
View SparkNet activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Software engineers: It's time to get promoted. Starting NOW! Subscribe to my mailing list and I will equip you with tools, tips and actionable advice to grow in your career.
Evaluating SparkNet for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)


Distributed Neural Networks for Spark. Details are available in the paper. Ask questions on the sparknet-users mailing list!

Quick Start

Start a Spark cluster using our AMI

  1. Create an AWS secret key and access key. Instructions here.
  2. Run export AWS_SECRET_ACCESS_KEY= and export AWS_ACCESS_KEY_ID= with the relevant values.
  3. Clone our repository locally.
  4. Start a 5-worker Spark cluster on EC2 by running

    SparkNet/ec2/spark-ec2 --key-pair=key \
                           --identity-file=key.pem \
                           --region=eu-west-1 \
                           --zone=eu-west-1c \
                           --instance-type=g2.8xlarge \
                           --ami=ami-d0833da3 \
                           --copy-aws-credentials \
                           --spark-version=1.5.0 \
                           --spot-price=1.5 \
                           --no-ganglia \
                           --user-data SparkNet/ec2/cloud-config.txt \
                           --slaves=5 \
                           launch sparknet

    You will probably have to change several fields in this command. For example, the flags --key-pair and --identity-file specify the key pair you will use to connect to the cluster. The flag --slaves specifies the number of Spark workers.

Train Cifar using SparkNet

  1. SSH to the Spark master as root.
  2. Run bash /root/SparkNet/data/cifar10/get_cifar10.sh to get the Cifar data
  3. Train Cifar on 5 workers using

    /root/spark/bin/spark-submit --class apps.CifarApp /root/SparkNet/target/scala-2.10/sparknet-assembly-0.1-SNAPSHOT.jar 5
  4. That's all! Information is logged on the master in /root/SparkNet/training_log*.txt.

Train ImageNet using SparkNet

  1. Obtain the ImageNet data by following the instructions here with

    wget http://.../ILSVRC2012_img_train.tar
    wget http://.../ILSVRC2012_img_val.tar

    This involves creating an account and submitting a request.

  2. On the Spark master, create ~/.aws/credentials with the following content:


    and fill in the two fields.

  3. Copy this to the workers with ~/spark-ec2/copy-dir ~/.aws (copy this command exactly because it is somewhat sensitive to the trailing backslashes and that kind of thing).

  4. Create an Amazon S3 bucket with name S3_BUCKET.

  5. Upload the ImageNet data in the appropriate format to S3 with the command

    python $SPARKNET_HOME/scripts/put_imagenet_on_s3.py $S3_BUCKET \
        --train_tar_file=/path/to/ILSVRC2012_img_train.tar \
        --val_tar_file=/path/to/ILSVRC2012_img_val.tar \
        --new_width=256 \

    This command resizes the images to 256x256, shuffles the training data, and tars the validation files into chunks.

  6. Train ImageNet on 5 workers using

    /root/spark/bin/spark-submit --class apps.ImageNetApp /root/SparkNet/target/scala-2.10/sparknet-assembly-0.1-SNAPSHOT.jar 5 $S3_BUCKET

Installing SparkNet on an existing Spark cluster

The specific instructions might depend on your cluster configurations, if you run into problems, make sure to share your experience on the mailing list.

  1. If you are going to use GPUs, make sure that CUDA-7.0 is installed on all the nodes.

  2. Depending on your configuration, you might have to add the following to your ~/.bashrc, and run source ~/.bashrc.

    export LD_LIBRARY_PATH=/usr/local/cuda-7.0/targets/x86_64-linux/lib/
    export _JAVA_OPTIONS=-Xmx8g
    export SPARKNET_HOME=/root/SparkNet/

    Keep in mind to substitute in the right directories (the first one should contain the file libcudart.so.7.0).

  3. Clone the SparkNet repository git clone https://github.com/amplab/SparkNet.git in your home directory.

  4. Copy the SparkNet directory on all the nodes using

    ~/spark-ec2/copy-dir ~/SparkNet
  5. Build SparkNet with

    cd ~/SparkNet
    git pull
    sbt assembly
  6. Now you can for example run the CIFAR App as shown above.

Building your own AMI

  1. Start an EC2 instance with Ubuntu 14.04 and a GPU instance type (e.g., g2.8xlarge). Suppose it has IP address xxx.xx.xx.xxx.
  2. Connect to the node as ubuntu:

    ssh -i ~/.ssh/key.pem ubuntu@xxx.xx.xx.xxx
  3. Install an editor

    sudo apt-get update
    sudo apt-get install emacs
  4. Open the file

    sudo emacs /root/.ssh/authorized_keys

    and delete everything before ssh-rsa ... so that you can connect to the node as root.

  5. Close the connection with exit.

  6. Connect to the node as root:

    ssh -i ~/.ssh/key.pem root@xxx.xx.xx.xxx
  7. Install CUDA-7.0.

    wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_7.0-28_amd64.deb
    dpkg -i cuda-repo-ubuntu1404_7.0-28_amd64.deb
    apt-get update
    apt-get upgrade -y
    apt-get install -y linux-image-extra-`uname -r` linux-headers-`uname -r` linux-image-`uname -r`
    apt-get install cuda-7-0 -y
  8. Install sbt. Instructions here.

  9. apt-get update

  10. apt-get install awscli s3cmd

  11. Install Java apt-get install openjdk-7-jdk.

  12. Clone the SparkNet repository git clone https://github.com/amplab/SparkNet.git in your home directory.

  13. Add the following to your ~/.bashrc, and run source ~/.bashrc.

    export LD_LIBRARY_PATH=/usr/local/cuda-7.0/targets/x86_64-linux/lib/
    export _JAVA_OPTIONS=-Xmx8g
    export SPARKNET_HOME=/root/SparkNet/

    Some of these paths may need to be adapted, but the LD_LIBRARY_PATH directory should contain libcudart.so.7.0 (this file can be found with locate libcudart.so.7.0 after running updatedb).

  14. Build SparkNet with

    cd ~/SparkNet
    git pull
    sbt assembly
  15. Create the file ~/.bash_profile and add the following:

    if [ "$BASH" ]; then
      if [ -f ~/.bashrc ]; then
        . ~/.bashrc
    export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

    Spark expects JAVA_HOME to be set in your ~/.bash_profile and the launch script SparkNet/ec2/spark-ec2 will give an error if it isn't there.

  16. Clear your bash history cat /dev/null > ~/.bash_history && history -c && exit.

  17. Now you can create an image of your instance, and you're all set! This is the procedure that we used to create our AMI.

JavaCPP Binaries

We have built the JavaCPP binaries for a couple platforms. They are stored at the following locations:

  1. Ubuntu with GPUs: http://www.eecs.berkeley.edu/~rkn/snapshot-2016-03-05/
  2. Ubuntu with CPUs: http://www.eecs.berkeley.edu/~rkn/snapshot-2016-03-16-CPU/
  3. CentOS 6 with CPUs: http://www.eecs.berkeley.edu/~rkn/snapshot-2016-03-23-CENTOS6-CPU/
SparkNet open issues Ask a question     (View All Issues)
  • almost 4 years I run the apps.CifarApp,but there were some mistakes.
  • almost 4 years Tensorflow GPU
  • almost 4 years sbt assembly failed in Ubuntu 14.04
  • almost 4 years Current Spark Net cannot build for both CPU and GPU
  • almost 4 years I wonder that does anyone run SparkNet in single node instead of a cluster?
  • about 4 years Unresolved Dependency
  • about 4 years GoogleNet training successful using PSGD method-- will Adam, AdaDelta work?
  • about 4 years Unable to install on macosx
  • about 4 years Can i compile SparkNet without using CUDA ( want to run with CPU)
  • about 4 years how to run on ppc64le platform?
  • about 4 years build issue
  • about 4 years can sparknet support a single GPU of each node on the sparknet cluster?
  • about 4 years Multi GPUs do not run on the sparknet cluster
  • about 4 years error while running CifarApp
  • about 4 years is current ubuntu library GPU cudnn enabled?
  • about 4 years Multi GPU for Caffe
  • over 4 years exception trying to run Mnist example
  • over 4 years Error : Check failed: error == cudaSuccess (30 vs. 0) unknown error
  • over 4 years UnsatisfiedLinkError when running Caffe Test
  • over 4 years Hyperparameters optimizzation
  • over 4 years Provide instructions for setting up SparkNet on custom cluster
  • over 4 years Re-enable Caffe tests
  • over 4 years CaffeOnSpark
  • over 4 years Support saving the Net and Solver state
  • over 4 years In ImageNetApp, leave JPEGs compressed so the full dataset fits in memory more easily
  • over 4 years Machine Learning Provider Abstraction Layer
  • over 4 years More of a question than an issue.. what IDE/env is used by committers to do SparkNet development?
  • over 4 years Number of iterations
SparkNet open pull requests (View All Pulls)
  • Multi GPU integration
  • Training googlenet from disk
  • remove unnecessary allocations from caffenet
  • fix memory leak during broadcasting
  • SBT files updated & created
  • Study googlenet using PSGD with LMDB and hadoop parquet
SparkNet list of languages used
Other projects in C++