Are you happy with your logging solution? Would you help us out by taking a 30-second survey? Click here

libkv

Distributed key/value store abstraction library

Subscribe to updates I use libkv


Statistics on libkv

Number of watchers on Github 486
Number of open issues 75
Average time to close an issue about 1 month
Main language Go
Average time to merge a PR 3 days
Open pull requests 38+
Closed pull requests 16+
Last commit over 1 year ago
Repo Created about 4 years ago
Repo Last Updated over 1 year ago
Size 311 KB
Organization / Authordocker
Contributors18
Page Updated
Do you use libkv? Leave a review!
View open issues (75)
View libkv activity
View on github
Fresh, new opensource launches πŸš€πŸš€πŸš€
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating libkv for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

libkv

GoDoc Build Status Coverage Status Go Report Card

libkv provides a Go native library to store metadata.

The goal of libkv is to abstract common store operations for multiple distributed and/or local Key/Value store backends.

For example, you can use it to store your metadata or for service discovery to register machines and endpoints inside your cluster.

You can also easily implement a generic Leader Election on top of it (see the docker/leadership repository).

As of now, libkv offers support for Consul, Etcd, Zookeeper (Distributed store) and BoltDB (Local store).

Usage

libkv is meant to be used as an abstraction layer over existing distributed Key/Value stores. It is especially useful if you plan to support consul, etcd and zookeeper using the same codebase.

It is ideal if you plan for something written in Go that should support:

  • A simple metadata storage, distributed or local
  • A lightweight discovery service for your nodes
  • A distributed lock mechanism

You can find examples of usage for libkv under in docs/examples.go. Optionally you can also take a look at the docker/swarm or docker/libnetwork repositories which are using docker/libkv for all the use cases listed above.

Supported versions

libkv supports:

  • Consul versions >= 0.5.1 because it uses Sessions with Delete behavior for the use of TTLs (mimics zookeeper's Ephemeral node support), If you don't plan to use TTLs: you can use Consul version 0.4.0+.
  • Etcd versions >= 2.0 because it uses the new coreos/etcd/client, this might change in the future as the support for APIv3 comes along and adds more capabilities.
  • Zookeeper versions >= 3.4.5. Although this might work with previous version but this remains untested as of now.
  • Boltdb, which shouldn't be subject to any version dependencies.

Interface

A storage backend in libkv should implement (fully or partially) this interface:

type Store interface {
    Put(key string, value []byte, options *WriteOptions) error
    Get(key string) (*KVPair, error)
    Delete(key string) error
    Exists(key string) (bool, error)
    Watch(key string, stopCh <-chan struct{}) (<-chan *KVPair, error)
    WatchTree(directory string, stopCh <-chan struct{}) (<-chan []*KVPair, error)
    NewLock(key string, options *LockOptions) (Locker, error)
    List(directory string) ([]*KVPair, error)
    DeleteTree(directory string) error
    AtomicPut(key string, value []byte, previous *KVPair, options *WriteOptions) (bool, *KVPair, error)
    AtomicDelete(key string, previous *KVPair) (bool, error)
    Close()
}

Compatibility matrix

Backend drivers in libkv are generally divided between local drivers and distributed drivers. Distributed backends offer enhanced capabilities like Watches and/or distributed Locks.

Local drivers are usually used in complement to the distributed drivers to store informations that only needs to be available locally.

Calls Consul Etcd Zookeeper BoltDB
Put X X X X
Get X X X X
Delete X X X X
Exists X X X X
Watch X X X
WatchTree X X X
NewLock (Lock/Unlock) X X X
List X X X X
DeleteTree X X X X
AtomicPut X X X X
Close X X X X

Limitations

Distributed Key/Value stores often have different concepts for managing and formatting keys and their associated values. Even though libkv tries to abstract those stores aiming for some consistency, in some cases it can't be applied easily.

Please refer to the docs/compatibility.md to see what are the special cases for cross-backend compatibility.

Other than those special cases, you should expect the same experience for basic operations like Get/Put, etc.

Calls like WatchTree may return different events (or number of events) depending on the backend (for now, Etcd and Consul will likely return more events than Zookeeper that you should triage properly). Although you should be able to use it successfully to watch on events in an interchangeable way (see the docker/leadership repository or the pkg/discovery/kv package in docker/docker).

TLS

Only Consul and etcd have support for TLS and you should build and provide your own config.TLS object to feed the client. Support is planned for zookeeper.

Roadmap

  • Make the API nicer to use (using options)
  • Provide more options (consistency for example)
  • Improve performance (remove extras Get/List operations)
  • Better key formatting
  • New backends?

Contributing

Want to hack on libkv? Docker's contributions guidelines apply.

Copyright and license

Copyright 2014-2016 Docker, Inc. All rights reserved, except as follows. Code is released under the Apache 2.0 license. The README.md file, and files in the docs folder are licensed under the Creative Commons Attribution 4.0 International License under the terms and conditions set forth in the file LICENSE.docs. You may obtain a duplicate copy of the same license, titled CC-BY-SA-4.0, at http://creativecommons.org/licenses/by/4.0/.

libkv open issues Ask a question     (View All Issues)
  • almost 3 years can not get lock after release.
  • almost 3 years make etcd periodic member sync configurable and optional
  • about 3 years Do we have plan upgrade to etcd clientv3?
  • about 3 years Question: Does swarm can repopulate netorking infos to backend (consul..) when backend is restarted ?
  • over 3 years Add support for username/password auth with Consul and Zookeeper
  • over 3 years generic configuration?
  • over 3 years Design review on redis/rediscluster store
  • over 3 years DynamoDB backend for libkv
  • over 3 years WatchTree (zookeeper) doesn't send event when value of a existing child key changed
  • over 3 years Zookeeper AtomicPut doesn't respect store.WriteOptions parameter
  • over 3 years Watchtree sends back whole list of specified directory in every event, how can a watcher know the changes?
  • over 3 years Fix Lock abstraction across distributed store backends
  • over 3 years health endpoint query
  • over 3 years Migrate etcd backend to APIv3 client
  • over 3 years Request: Non Recursive "directory" listing
  • over 3 years support for mkdir?
  • almost 4 years Expose recursive parameter of WatchTree through WatchOptions
  • almost 4 years Test TLS client configurations with travis
  • almost 4 years Enhancement: Automatic and periodic renew of Key lease for basic Put operation
  • about 4 years Add mapping layer for keys
  • about 4 years Add support for ACL
  • about 4 years Watch/WatchTree error return is useless, should be changed for an error channel
  • about 4 years puts that would succeed on consul & zookeeper fail on etcd
  • about 4 years libkv should have a way to watch a directory that doesn't exist yet.
  • about 4 years Mechanism to abort Lock
libkv open pull requests (View All Pulls)
  • Update README.md: typo
  • Improve Consul lock
  • fix zookeeper problems
  • fix etcd list recursive
  • Add Consul client TLS
  • Dockerfile & Makefile
  • Allow full URLs in connection specification, with etcd tests
  • fix wrong error code in Zookeeper.List
  • Use fork of go-zookeeper
  • Increase transientTimeout of boltdb to 2mimute
  • make etcd periodic member sync configurable and optional
  • Allow client to set boltdb timeout
  • use error channel for Watch/WatchTree
  • etcd: Implement PersistConnection config option
  • [redis] add redis backend
  • store/boltdb: Allow paths with no slash at the beginning
  • Fix broken headings in Markdown files
  • Zookeeper missed watched events
  • Zookeeper: Relinquish locks on session close
  • Fix CI SSL issue
  • Allow to specify consul scheme (only HTTP is allowed at the moment)
  • ingore error of etcd member auto sync
  • Zookeeper: Make Create+Put on a new znode atomic
  • Support Consul ACL Tokens
  • add dynamodDB
  • Fix to markdown parsing issues mentioned in tic…
  • configurable zookeeper logging
  • consul: set List options for consistent results
  • Fix AutoSync when a etcd cluster is unhealthy
  • specify the default http.transport
  • Update examples.md
  • do not wait lock when network broken
  • add MySQL backend implementant and support all methods
  • Add dimrok as a maintainer.
  • Travis: switch to trusty
  • Fixing markdown rendering
  • Add support for memo backend.
  • Update go, etcd, and consul version
libkv questions on Stackoverflow (View All Questions)
  • libkv / ectd go client : how to use config.TLS object?
libkv list of languages used
Other projects in Go