Are you happy with your logging solution? Would you help us out by taking a 30-second survey? Click here

crosscat

A domain-general, Bayesian method for analyzing high-dimensional data tables

Subscribe to updates I use crosscat


Statistics on crosscat

Number of watchers on Github 290
Number of open issues 26
Average time to close an issue 2 months
Main language Python
Average time to merge a PR 5 days
Open pull requests 7+
Closed pull requests 9+
Last commit almost 2 years ago
Repo Created about 6 years ago
Repo Last Updated over 1 year ago
Size 8.82 MB
Homepage http://probcomp.c...
Organization / Authorprobcomp
Contributors15
Page Updated
Do you use crosscat? Leave a review!
View open issues (26)
View crosscat activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating crosscat for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Crosscat

.. image:: https://travis-ci.org/probcomp/crosscat.svg?branch=master :target: https://travis-ci.org/probcomp/crosscat

CrossCat is a domain-general, Bayesian method for analyzing high-dimensional data tables. CrossCat estimates the full joint distribution over the variables in the table from the data, via approximate inference in a hierarchical, nonparametric Bayesian model, and provides efficient samplers for every conditional distribution. CrossCat combines strengths of nonparametric mixture modeling and Bayesian network structure learning: it can model any joint distribution given enough data by positing latent variables, but also discovers independencies between the observable variables.

A range of exploratory analysis and predictive modeling tasks can be addressed via CrossCat, including detecting predictive relationships between variables, finding multiple overlapping clusterings, imputing missing values, and simultaneously selecting features and classifying rows. Research on CrossCat has shown that it is suitable for analysis of real-world tables of up to 10 million cells, including hospital cost and quality measures, voting records, handwritten digits, and state-level unemployment time series.

Installation

Local (Ubuntu)

You can install CrossCat using pip (no need to clone from git)::

$ pip install crosscat

If you'd like to install from source, CrossCat can be successfully installed locally on bare Ubuntu server 14.04 systems with::

$ sudo apt-get install build-essential cython python
$ sudo apt-get install python-setuptools python-numpy
$ git clone https://github.com/probcomp/crosscat.git

$ cd crosscat
$ python setup.py build
$ python setup.py install  # or python setup.py develop

CrossCat can also be installed in a local Python virtual environment: ::

$ cd crosscat
$ virtualenv --system-site-packages /path/to/venv
$ . /path/to/venv/bin/activate
$ python setup.py build
$ python setup.py install  # or python setup.py develop

A similar process has been found to work on OSX.

Tests

To run the automatic tests:

$ ./check.sh

Documentation

Note: The VM is only meant to provide an out-of-the-box usable system setup. Its resources are limited and large jobs will fail due to memory errors. To run larger jobs, increase the VM resources or install directly to your system.

Python Client_

.. _Python Client: https://docs.google.com/file/d/0B_CtKGJ4pH2TdmNRZkhmamg5aVU/edit?usp=drive_web)

C++ backend_

.. _C++ backend: https://docs.google.com/file/d/0B_CtKGJ4pH2TeVo0Zk5IT3V6S0E/edit?usp=drive_web)

Example

dha_example.py (github_) is a basic example of analysis using CrossCat. For a first test, run the following from above the top level crosscat dir

.. _github: https://github.com/probcomp/crosscat/blob/master/examples/dha_example.py

::

python crosscat/examples/dha_example.py crosscat/www/data/dha.csv --num_chains 2 --num_transitions 2

Note: the default argument values take a considerable amount of time to run and are best suited to a cluster.

License

Apache License, Version 2.0_

.. _Apache License, Version 2.0: https://github.com/probcomp/crosscat/blob/master/LICENSE)

crosscat open issues Ask a question     (View All Issues)
  • almost 3 years missing dependencies / environment compatibilities
  • over 3 years pls help run on windows 10 machine with anaconda python 2.7
  • over 3 years test_multiple_col_ensure.py is stochastic
  • over 3 years Multistate conditional sampling is inaccurate
  • over 3 years Crosscat should use 256-bit seeds
  • over 3 years Crosscat hyperprior grid on variance parameter is broader than it needs to be
  • over 3 years continuous component model can't handle constant column
  • over 3 years Hadoop state non-functional
  • over 3 years Python 3 compatibility
  • over 3 years Allow multistate queries in sample_utils to utilize multiprocessing
  • almost 4 years Speed up inference (and ensure ergodicity) in the presence of ENSURE DEPENDENT
  • almost 4 years Get a jenkins job testing crosscat with python3
  • almost 4 years Revising algorithm for computing joint pdf
  • almost 4 years Distribute binary builds
  • almost 4 years our PRNG sucks
  • almost 4 years crosscat does stupidly many deep copies
  • almost 4 years suspicious implementation of sampling from categorical for cluster assignments
  • about 4 years Issues with impute_confidence.
  • about 4 years numpy and cython should not be required to be installed before setup.py runs
  • about 4 years Building for Anaconda on OS X requires extra linker flags
  • about 4 years BOOST_ROOT not being picked up when doing pip build
  • about 4 years vonmises test in test_mixture_inference_quality.py fails
  • over 4 years associate random seed with each model
  • over 2 years rename pypi package: CrossCat -> crosscat
crosscat open pull requests (View All Pulls)
  • use cmath, not math.h, in C++ code
  • Randomize seeds for chain states better
  • 20160303 alxempirical pass bdb seed to crosscat engine
  • Remove dependency on boost.
  • Release the GIL, crosscat!!!
  • Add sanity to the poorly formatted State.pyx
  • 20160920 fsaad fixup
crosscat list of languages used
More projects by probcomp View all
Other projects in Python