Want to take your software engineering career to the next level? Join the mailing list for career tips & advice Click here

irmin

Irmin is a distributed database that follows the same design principles as Git

Subscribe to updates I use irmin


Statistics on irmin

Number of watchers on Github 1327
Number of open issues 81
Average time to close an issue 11 days
Main language OCaml
Average time to merge a PR 3 days
Open pull requests 26+
Closed pull requests 12+
Last commit 2 months ago
Repo Created over 7 years ago
Repo Last Updated 2 months ago
Size 14.5 MB
Homepage https://irmin.org
Organization / Authormirage
Latest Release2.2.0
Contributors16
Page Updated
Do you use irmin? Leave a review!
View open issues (81)
View irmin activity
View on github
Book a Mock Interview With Me (Silicon Valley Engineering Leader, 100s of interviews conducted)
Software engineers: It's time to get promoted. Starting NOW! Subscribe to my mailing list and I will equip you with tools, tips and actionable advice to grow in your career.
Evaluating irmin for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Irmin - A distributed database built on the same principles as Git

OCaml-CI Build Status Travis-CI Build Status docs

Irmin is an OCaml library for building mergeable, branchable distributed data stores.

Features

  • Built-in snapshotting - backup and restore
  • Storage agnostic - you can use Irmin on top of your own storage layer
  • Custom datatypes - (de)serialization for custom data types, derivable via ppx_irmin
  • Highly portable - runs anywhere from Linux to web browsers and Xen unikernels
  • Git compatibility - irmin-git uses an on-disk format that can be inspected and modified using Git
  • Dynamic behavior - allows the users to define custom merge functions, use in-memory transactions (to keep track of reads as well as writes) and to define event-driven workflows using a notification mechanism

Documentation

Documentation can be found online at https://mirage.github.io/irmin

Installation

Prerequisites:

Please ensure to install the minimum Opam and Ocaml versions which are 2 and 4.07 respectively.

opam list    // listing the installed packages

To install Irmin, the command-line tool and all optional dependencies using opam:

opam install irmin-unix

Note : If you face installation issues due to pinned packages. Run the below commands to unpin and try to install again,

Unpinning irmin-unix package,

opam info irmin-unix // to get available versions
opam pin -s | grep irmin | xargs opam unpin

A minimal installation, with no storage backends can be installed by running:

opam install irmin

To only install the in-memory storage backend:

opam install irmin-mem

The following packages have been made available on opam:

  • irmin - the base package, no storage implementations
  • irmin-chunk - chunked storage
  • irmin-fs - filesystem-based storage using bin_prot
  • irmin-git - Git compatible storage
  • irmin-graphql - GraphQL server
  • irmin-http - a simple REST interface
  • irmin-mem - in-memory storage implementation
  • irmin-mirage - mirage compatibility
  • irmin-mirage-git - Git compatible storage for mirage
  • irmin-mirage-graphql - mirage compatible GraphQL server
  • irmin-unix - unix compatibility
  • irmin-pack - compressed, on-disk, posix backend
  • ppx_irmin - PPX deriver for Irmin content types (see README_PPX.md)
  • irmin-containers - collection of simple, ready-to-use mergeable data structures

For more information about an individual package consult the online documentation.

Development version

To install the development version of Irmin, clone this repository and opam install the packages inside:

git clone https://github.com/mirage/irmin
cd irmin/
opam install .

Examples

Below is a simple example of setting a key and getting the value out of a Git based, filesystem-backed store.

open Lwt.Infix

(* Irmin store with string contents *)
module Store = Irmin_unix.Git.FS.KV(Irmin.Contents.String)

(* Database configuration *)
let config = Irmin_git.config ~bare:true "/tmp/irmin/test"

(* Commit author *)
let author = "Example <example@example.com>"

(* Commit information *)
let info fmt = Irmin_unix.info ~author fmt

let main =
  (* Open the repo *)
  Store.Repo.v config >>=

  (* Load the master branch *)
  Store.master >>= fun t ->

  (* Set key "foo/bar" to "testing 123" *)
  Store.set_exn t ~info:(info "Updating foo/bar") ["foo"; "bar"] "testing 123" >>= fun () ->

  (* Get key "foo/bar" and print it to stdout *)
  Store.get t ["foo"; "bar"] >|= fun x ->
  Printf.printf "foo/bar => '%s'\n" x

(* Run the program *)
let () = Lwt_main.run main

The example is contained in examples/readme.ml. It can be compiled and executed with dune:

$ dune build examples/readme.exe
$ dune exec examples/readme.exe
foo/bar => 'testing 123'

The examples/ directory also contains more advanced examples, which can be executed in the same way.

Command-line

The same thing can also be accomplished using irmin, the command-line application installed with irmin-unix, by running:

$ echo "root: ." > irmin.yml
$ irmin init
$ irmin set foo/bar "testing 123"
$ irmin get foo/bar

irmin.yml allows for irmin flags to be set on a per-directory basis. You can also set flags globally using $HOME/.irmin/config.yml. Run irmin help irmin.yml for further details.

Also see irmin --help for list of all commands and either irmin <command> --help or irmin help <command> for more help with a specific command.

Issues

Feel free to to report any issues using the Github bugtracker.

Conditions

See the LICENSE file.

Acknowledgements

Development of Irmin was supported in part by the EU FP7 User-Centric Networking project, Grant No. 611001.

irmin open issues Ask a question     (View All Issues)
  • almost 4 years Issues with pulling/pushing remote repo
  • about 4 years Documentation for `bare` repository unclear.
  • over 4 years Compiling the examples seem broken
  • over 4 years View.update is really expensive
  • over 4 years Error loading irmin.mirage from utop.
  • over 4 years Dependency problem with sexplib?
  • over 4 years Expose `/remotes/` branches for the Git backend
  • over 4 years Expose some kind of "mtime"
  • over 4 years Problem while trying to sync a remote using Irmin_mirage
  • almost 5 years http: error format
  • almost 5 years http: internal error, uncaught exception if too many connections
  • almost 5 years Setting compression level to 0 in Irmin_git doesn't disable compression.
  • almost 5 years Must `BC` require `HRW`?
  • almost 5 years Issue with recursive merging of deleted values
  • almost 5 years Better IO functions
  • almost 5 years Use Jsont
  • almost 5 years Use Fmt
  • about 5 years rest api: /view/merge-path returns "ok": null on success
  • about 5 years rest api: /view/create returns "-" separated string in json output
  • about 5 years rest api: /view/create is /view/create/create
  • about 5 years Implement `Irmin.compare_and_set`
  • about 5 years REST API: JSON parse error when numbers are not in quotes
  • about 5 years REST API: remove-rec without parameters deletes everything
  • about 5 years Review `Task.uid`
  • about 5 years Document the binary format of the REST API
  • about 5 years More consistent types for params and results in the REST API
  • about 5 years Review the error path of the REST API
  • about 5 years Type the JSON fields
  • about 5 years Document the `.irminconfig` file
  • about 5 years The `-s` option of the irmin command-line is obscure
irmin open pull requests (View All Pulls)
  • Fix performance problem in lcas
  • First tentative for Ir_funview.
  • Convenience functors to to customize node metadata
  • ptime is used by the mirage backend only, so moving it to depopts
  • Test windows
  • remove unused META files
  • irmin-chunk: Fix content types
  • Use the new ocaml-git in irmin-git
  • Depend on cohttp 1.0.0
  • HTTP REST API
  • Better pretty-printer output.
  • irmin-pack: make instance sharing explicit
  • irmin-pack: fixes to work with index's master after mirage/index#188
  • irmin-pack: update the file format to v2
  • Upgrade to ocamlformat 0.15.0
  • Clear + ro
  • Use MDX to test the README example
  • Fixed "illegal literal" error when JSON-decoding NaN.
  • RO opens pack before RW
  • Improve last_modified performances
  • [WIP] Layered store
  • Use digestif default variant
  • Add a simple diagnostic tool
  • Improved default GraphQL presentation
  • Fuzzing Irmin.Types
  • [WIP] Add metrics to irmin operations
irmin list of languages used
irmin latest release notes

CHANGES:

Added

  • irmin:

    • Added Irmin.Type.empty to represent an uninhabited type. (#961, @CraigFe)
    • Added Store.Tree.concrete_t. (#1003, @CraigFe)
  • ppx_irmin

    • Added support for the @nobuiltin attribute, which can be used when shadowing primitive types such as unit. See README_PPX for details. (#993, @CraigFe)
    • Added support for a lib argument, which can be used to supply primitive type representations from modules other than Irmin.Type. (#994, @CraigFe)

Changed

  • irmin:

    • Require OCaml 4.07 (#961, @CraigFe)
    • Add sanity checks when creating Irmin.Type records, variants and enums (#956 and #966, @liautaud):
      • Irmin.Type.{sealr,sealv,enum} will now raise Invalid_argument if two components have the same name;
      • Irmin.Type.{field,case0,case1} will now raise Invalid_argument if the component name is not a valid UTF-8 string.
    • Changed the JSON encoding of options and unit to avoid ambiguous cases (#967, @liautaud):
    • () is now encoded as {};
    • None is now encoded as null;
    • Some x is now encoded as {"some": x};
    • Fields of records which have value None are still omitted;
    • Fields of records which have value Some x are still unboxed into x.
    • Changed pretty-printing of Irmin types to more closely resemble OCaml types. e.g. pair int string prints as int * string. (#997, @CraigFe)
    • The type Irmin.S.tree is now abstract. The previous form can be coerced to/from the abstract representation with the new functions Irmin.S.Tree.{v,destruct} respectively. (#990, @CraigFe)
  • irmin-mem

    • Stores created with KV now expose their unit metadata type. (#995, @CraigFe)

Fixed

  • irmin-graphql
    • Fixed an issue with keys inside get_{contents,tree} fields having incorrect ordering (#989, @CraigFe)

CHANGES:

Added

  • ppx_irmin (new):

    • Created a new package, ppx_irmin, which provides a PPX deriving plugin for generating Irmin generics.
  • irmin-unix:

    • Added a --hash parameter to the command-line interface, allowing the hash function to be specified. For BLAKE2b and BLAKE2s, the bit-length may be specified with a trailing slash, as in --hash=blake2b/16. The hash function may also be specified in the configuration file. (#898, @craigfe)
  • irmin:

    • Added Irmin.Hash.Make_BLAKE2B and Irmin.Hash.Make_BLAKE2S functors for customizing the bit-length of these hash functions. (#898, @craigfe)
    • Added iter function over a closure graph (#912, @ioana)
    • Added Type.pp_ty for pretty-printing Irmin generics. (#926, @craigfe)
    • Added Merge.with_conflict for modifying the conflict error message of a merge function. (#926, @craigfe)

Changed

  • irmin-pack:

    • Changed the bit-length of serialized hashes from 60 to 30. (#897, @icristescu)
    • integrity_check can now try to repair corrupted values. (#947, @pascutto)
  • irmin-graphql:

    • Changed default GraphQL type names to ensure uniqueness. (#944, @andreas)

CHANGES:

Added

  • irmin-pack (new):

    • Created a new Irmin backend, irmin-pack, which uses a space-optimised on-disk format.
  • irmin-graphql (new):

    • Created a new package, irmin-graphql, which provides a GraphQL server implementation that can be used with both the MirageOS and Unix backends. Additionally, a graphql command has been added to the command-line interface for starting irmin-graphql servers. (#558, @andreas, @zshipko)
    • Contents can now be queried directly using irmin-graphql with Irmin_graphql.Server.Make_ext and the Irmin_graphql.Server.PRESENTER interface. (#643, @andreas)
  • irmin-test (new):

    • Added a new package, irmin-test, which allows for packages to access the Irmin test-suite. This package can now be used for new packages that implement custom backends to test their implementations against the same tests that the core backends are tested against. (#508, @zshipko)
  • irmin-unix:

    • Add Cli module to expose some methods to simplify building command-line interfaces using Irmin. (#517, @zshipko)
    • Add global config file $HOME/.irmin/config.yml which may be overridden by either $PWD/.irmin.yml or by passing --config <PATH>. See irmin help irmin.yml for details. (#513, @zshipko)
  • irmin-git:

    • Allow import/export of Git repositories using Irmin slices. (#561, @samoht)
  • irmin-http:

    • Expose a /trees/merge route for server-side merge operations. (#714, @samoht)
  • irmin:

    • Add Json_value and Json content types. (#516 #694, @zshipko)
    • Add optional seed parameter to the Irmin.Type generic hash functions. (#712, @samoht)
    • Add V1 submodules in Commit, Contents and Hash to provide compatibility with 1.x serialisation formats. (#644 #666, @samoht)
    • Add Store.last_modified function, which provides a list of commits where the given key was modified last. (#617, @pascutto)
    • Add a Content_addressable.unsafe_add function allowing the key of the new value to be specified explicitly (for performance reasons). (#783, @samoht)
    • Add save_contents function for saving contents to the database. (#689, @samoht)
    • Add pretty-printers for the results of Sync operations. (#789, @craigfe)
    • Private.Lock now exposes a stats function returning the number of held locks. (#704, @samoht)

Changed

  • irmin-unix:

    • Rename irmin read to irmin get and irmin write to irmin set. (#501, @zshipko)
    • Switch from custom configuration format to YAML. (#504, @zshipko)
  • irmin-git:

    • Require ocaml-git >= 2.0. (#545, @samoht)
    • Cleanup handling of remote stores. (#552, @samoht)
  • irmin-http:

    • Rename CLIENT to HTTP_CLIENT and simplify the signatures necessary to construct HTTP clients and servers. (#701, @samoht)
  • irmin-mirage

    • Split irmin-mirage into irmin-{mirage,mirage-git,mirage-graphql} to allow for more granular dependency selection. Any instances of Irmin_mirage.Git should be replaced with Irmin_mirage_git. (#686, @zshipko)
  • irmin:

    • Update to use dune (#534, @samoht) and opam 2.0. (#583, @samoht)
    • Replace Irmin.Contents.S0 with Irmin.Type.S.
    • Rename Type.pre_digest -> Type.pre_hash and Type.hash -> Type.short_hash. (#720, @samoht)
    • Change Irmin.Type to use incremental hash functions (functions of type 'a -> (string -> unit) -> unit) for performance reasons. (#751, @samoht)
    • Simplify the Irmin.Type.like constructor and add a new Irmin.Type.map with the previous behaviour.
    • Improvements to Irmin.Type combinators. (#550 #538 #652 #653 #655 #656 #688, @samoht)
    • Modify Store.set to return a result type and create a new Store.set_exn with the previous exception-raising behaviour. (#572, @samoht)
    • Rename store module types to be more descriptive:
      • replace Irmin.AO with Irmin.CONTENT_ADDRESSABLE_STORE;
      • replace Irmin.AO_MAKER with Irmin.CONTENT_ADDRESSABLE_STORE_MAKER;
      • replace Irmin.RW with Irmin.ATOMIC_WRITE_STORE;
      • replace Irmin.RW_MAKER with Irmin.ATOMIC_WRITE_STORE_MAKER. (#601, @samoht)
    • Rename export_tree to save_tree (#689, @samoht) and add an option to conditionally clear the tree cache (#702 #725, @samoht).
    • Change hash function for Irmin_{fs,mem,unix}.KV to BLAKE2b rather than SHA1 for security reasons. (#811, @craigfe)
    • Move Irmin.remote_uri to Store.remote, for stores that support remote operations. (#552, @samoht)
    • Simplify the error cases of fetch/pull/push operations. (#684, @zshipko)
    • A batch function has been added to the backend definition to allow for better control over how groups of operations are processed. (#609, @samoht)
    • A close function has been added to allow backends to close any held resources (e.g. file descriptors for the FS backend). (#845, @samoht)
    • Simplify Private.Node.Make parameters to use a simpler notion of 'path' in terms of a list of steps. (#645, @samoht)
    • Rename Node.update to Node.add. (#713, @samoht)

Fixed

  • irmin-unix:

    • Fix parsing of commit hashes in revert command. (#496, @zshipko)
  • irmin-git:

    • Fix Node.add to preserve sharing. (#802, @samoht)
  • irmin-http:

    • Respond with a 404 if a non-existent resource is requested. (#706, @samoht)
  • irmin:

    • Fix a bug whereby S.History.is_empty would return true for a store with exactly one commit. (#865, @pascutto)

Removed

  • irmin:

    • Remove pp and of_string functions from Irmin.Contents.S in favour of Irmin.Type.to_string and Irmin.Type.of_string.
    • Remove Bytes content type. (#708, @samoht)
    • Remove Cstruct dependency and content type. If possible, switch to Irmin.Contents.String or else use Irmin.Type.map to wrap the Cstruct type. (#544, @samoht)
Other projects in OCaml
Powered by Autocode - Instant Webhooks, Scripts and APIs
Autocode logo wordmark