Are you happy with your logging solution? Would you help us out by taking a 30-second survey? Click here

fast-neural-style

Feedforward style transfer

Subscribe to updates I use fast-neural-style


Statistics on fast-neural-style

Number of watchers on Github 2366
Number of open issues 109
Average time to close an issue about 16 hours
Main language Lua
Average time to merge a PR 2 days
Open pull requests 9+
Closed pull requests 0+
Last commit over 2 years ago
Repo Created about 3 years ago
Repo Last Updated over 1 year ago
Size 18.6 MB
Organization / Authorjcjohnson
Contributors7
Page Updated
Do you use fast-neural-style? Leave a review!
View open issues (109)
View fast-neural-style activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating fast-neural-style for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

fast-neural-style

This is the code for the paper

Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Justin Johnson, Alexandre Alahi, Li Fei-Fei
Presented at ECCV 2016

The paper builds on A Neural Algorithm of Artistic Style by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge by training feedforward neural networks that apply artistic styles to images. After training, our feedforward networks can stylize images hundreds of times faster than the optimization-based method presented by Gatys et al.

This repository also includes an implementation of instance normalization as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization by Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. This simple trick significantly improves the quality of feedforward style transfer models.

Stylizing this image of the Stanford campus at a resolution of 1200x630 takes 50 milliseconds on a Pascal Titan X:

In this repository we provide:

If you find this code useful for your research, please cite

@inproceedings{Johnson2016Perceptual,
  title={Perceptual losses for real-time style transfer and super-resolution},
  author={Johnson, Justin and Alahi, Alexandre and Fei-Fei, Li},
  booktitle={European Conference on Computer Vision},
  year={2016}
}

Setup

All code is implemented in Torch.

First install Torch, then update / install the following packages:

luarocks install torch
luarocks install nn
luarocks install image
luarocks install lua-cjson

(Optional) GPU Acceleration

If you have an NVIDIA GPU, you can accelerate all operations with CUDA.

First install CUDA, then update / install the following packages:

luarocks install cutorch
luarocks install cunn

(Optional) cuDNN

When using CUDA, you can use cuDNN to accelerate convolutions.

First download cuDNN and copy the libraries to /usr/local/cuda/lib64/. Then install the Torch bindings for cuDNN:

luarocks install cudnn

Pretrained Models

Download all pretrained style transfer models by running the script

bash models/download_style_transfer_models.sh

This will download ten model files (~200MB) to the folder models/.

Models from the paper

The style transfer models we used in the paper will be located in the folder models/eccv16. Here are some example results where we use these models to stylize this image of the Chicago skyline with at an image size of 512:


Models with instance normalization

As discussed in the paper Instance Normalization: The Missing Ingredient for Fast Stylization by Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky, replacing batch normalization with instance normalization significantly improves the quality of feedforward style transfer models.

We have trained several models with instance normalization; after downloading pretrained models they will be in the folder models/instance_norm.

These models use the same architecture as those used in our paper, except with half the number of filters per layer and with instance normalization instead of batch normalization. Using narrower layers makes the models smaller and faster without sacrificing model quality.

Here are some example outputs from these models, with an image size of 1024:



Running on new images

The script fast_neural_style.lua lets you use a trained model to stylize new images:

th fast_neural_style.lua \
  -model models/eccv16/starry_night.t7 \
  -input_image images/content/chicago.jpg \
  -output_image out.png

You can run the same model on an entire directory of images like this:

th fast_neural_style.lua \
  -model models/eccv16/starry_night.t7 \
  -input_dir images/content/ \
  -output_dir out/

You can control the size of the output images using the -image_size flag.

By default this script runs on CPU; to run on GPU, add the flag -gpu specifying the GPU on which to run.

The full set of options for this script is described here.

Webcam demo

You can use the script webcam_demo.lua to run one or more models in real-time off a webcam stream. To run this demo you need to use qlua instead of th:

qlua webcam_demo.lua -models models/instance_norm/candy.t7 -gpu 0

You can run multiple models at the same time by passing a comma-separated list to the -models flag:

qlua webcam_demo.lua \
  -models models/instance_norm/candy.t7,models/instance_norm/udnie.t7 \
  -gpu 0

With a Pascal Titan X you can easily run four models in realtime at 640x480:

The webcam demo depends on a few extra Lua packages:

You can install / update these packages by running:

luarocks install camera
luarocks install qtlua

The full set of options for this script is described here.

Training new models

You can find instructions for training new models here.

Optimization-based Style Transfer

The script slow_neural_style.lua is similar to the original neural-style, and uses the optimization-based style-transfer method described by Gatys et al.

This script uses the same code for computing losses as the feedforward training script, allowing for fair comparisons between feedforward style transfer networks and optimization-based style transfer.

Compared to the original neural-style, this script has the following improvements:

  • Remove dependency on protobuf and loadcaffe
  • Support for many more CNN architectures, including ResNets

The full set of options for this script is described here.

License

Free for personal or research use; for commercial use please contact me.

fast-neural-style open issues Ask a question     (View All Issues)
  • almost 3 years Training new model with OpenCL backend
  • almost 3 years What does the style_image_size do
  • almost 3 years Commercial Use
  • almost 3 years How to use this code for super-resolution?
  • almost 3 years Style transfer not much
  • almost 3 years about the style loss function
  • almost 3 years Cannot load pre-trained model file on android and iOS
  • almost 3 years prisma seems to preserve more detail
  • almost 3 years Run train on AWS EC2
  • almost 3 years Using two models at the same time
  • almost 3 years Training is very slow
  • almost 3 years More trained models
  • almost 3 years Can I run this on Mac
  • almost 3 years preprocessing
  • almost 3 years default model
  • almost 3 years Why my model file size is so big (551MB)?
  • almost 3 years Minimum number of images of dataset
  • about 3 years Training installation instructions vs Ubuntu 14.04.5
  • about 3 years Training parameters and procedure
  • about 3 years Fast Neural Style
  • about 3 years Avoid checkerboard pattern
  • about 3 years problem with output_dir
  • about 3 years Why model file size is not fixed, but seems to increase with the training schedule?
  • about 3 years File vgg16_nograd.t7
  • about 3 years this can support video?
  • about 3 years Generate image very slow
  • about 3 years resume from checkpoint
  • about 3 years Training killed
  • about 3 years About the style layer
  • about 3 years Error in CuDNN: CUDNN_STATUS_ALLOC_FAILED in training
fast-neural-style open pull requests (View All Pulls)
  • Input dir files without extension
  • dockerfile for quick & painless setup
  • Flip webcam image to make it a mirror
  • Typo in /fast_neural_style/StyleLoss.lua
  • Add RiseML Demo
  • Follow 301 redirects in the bash script that downloads the models
  • allow fast_neural_style.lua to use multiple models
  • Added support for histogram matching
  • Docker Image with Nvidia support
fast-neural-style list of languages used
Other projects in Lua