Want to take your software engineering career to the next level? Join the mailing list for career tips & advice Click here

multilevel

Expose a LevelDB over the network

Subscribe to updates I use multilevel


Statistics on multilevel

Number of watchers on Github 313
Number of open issues 19
Average time to close an issue about 2 months
Main language JavaScript
Average time to merge a PR 20 days
Open pull requests 7+
Closed pull requests 3+
Last commit over 2 years ago
Repo Created over 7 years ago
Repo Last Updated over 2 years ago
Size 196 KB
Organization / Authorjuliangruber
Contributors12
Page Updated
Do you use multilevel? Leave a review!
View open issues (19)
View multilevel activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Software engineers: It's time to get promoted. Starting NOW! Subscribe to my mailing list and I will equip you with tools, tips and actionable advice to grow in your career.
Evaluating multilevel for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

multilevel

Expose a levelDB over the network, to be used by multiple processes, with levelUp's API.

Build Status downloads

Usage

Expose a db on the server:

var multilevel = require('multilevel');
var net = require('net');
var level = require('level');

var db = level('/my/db');

net.createServer(function (con) {
  con.pipe(multilevel.server(db)).pipe(con);
}).listen(3000);

And connect to it from the client:

var multilevel = require('multilevel');
var net = require('net');

var db = multilevel.client();
var con = net.connect(3000);
con.pipe(db.createRpcStream()).pipe(con);

// asynchronous methods
db.get('foo', function () { /* */ });

// streams
db.createReadStream().on('data', function () { /* */ });

Compatibility and binary data

multilevel works in the browser too - via browserify - and has full support for binary data. For getting a connection between browser and server I recommend either websocket-stream, which treats binary data well, or engine.io-stream, which has websocket fallbacks.

When using a binary capable transport, require multilevel like this:

var multilevel = require('multilevel/msgpack');

This way it uses msgpack-stream instead of json-buffer which uses way less bandwidth - but needs a binary capable transport.

Plugins

You can also expose custom methods and sublevels with multilevel!

When using plugins, you must generate a manifest and require it in the client.

Here's an example:

// server.js
// create `db`
var level = require('level');
var multilevel = require('multilevel');
var db = level(PATH);

// extend `db` with a foo(cb) method
db.methods = db.methods || {};
db.methods['foo'] = { type: 'async' };
db.foo = function (cb) {
  cb(null, 'bar');
};

// now write the manifest to a file
multilevel.writeManifest(db, __dirname + '/manifest.json');

// then expose `db` via shoe or any other streaming transport.
var shoe = require('shoe');
var sock = shoe(function (stream) {
  stream.pipe(multilevel.server(db)).pipe(stream);
});
sock.install(http.createServer(/* ... */), '/websocket');

Manifests are generated using level-manifest, which doesn't only support async functions but e.g. streams as well. For more, check its README.

Then require the manifest on the client when bundling with browserify or in any other nodejs compatible environment.

// client.js
// instantiate a multilevel client with the `manifest.json` we just generated
var multilevel = require('multilevel');
var manifest = require('./manifest.json');
var db = multilevel.client(manifest);

// now pipe the db to the server
var stream = shoe('/websocket');
stream.pipe(db.createRpcStream()).pipe(stream);

// and you can call the custom `foo` method!
db.foo(function (err, res) {
  console.log(res); // => "bar"
});

Authentication

You do not want to expose every database feature to every user, you might e.g. only want to provide read-only access to some users.

Auth controls may be injected when creating the server stream.

In this example, allow read only access, unless logged in as root.

//server.js
var db = require('./setup-db'); //all your database customizations
var multilevel = require('multilevel');

//write out manifest
multilevel.writeManifest(db, __dirname + '/manifest.json');

shoe(function (stream) {
  stream.pipe(multilevel.server(db, {
    auth: function (user, cb) {
      if (user.name == 'root' && user.pass == 'toor') {
        //the data returned will be attached to the mulilevel stream
        //and passed to `access`
        cb(null, {name: 'root'});
      } else {
        cb(new Error('not authorized');
      }
    },
    access: function (user, db, method, args) {
      //`user` is the {name: 'root'} object that `auth`
      //returned. 

      //if not a privliged user...
      if (!user || user.name !== 'root') {
        //do not allow any write access
        if (/^put|^del|^batch|write/i.test(method)) {
          throw new Error('read-only access');
        }
      }        
    })
  })).pipe(stream);
});

// ...

The client authorizes by calling the auth method.

var multilevel = require('multilevel');
var shoe = require('shoe');

var stream = shoe();
var db = multilevel.client();
stream.pipe(db.createRpcStream()).pipe(stream);

db.auth({ name: 'root', pass: 'toor' }, function (err, data) {
  if (err) throw err
  //later, they can sign out, too.

  db.deauth(function (err) {
    //signed out!
  });
});

API

The exposed DB has the exact same API as levelUp, except

  • db#close() closes the connection, instead of the database.
  • the synchronous versions of db#isOpen() and db#isClose() tell you if you currently have a connection to the remote db.
  • events, like db.on("put", ...) are not emitted. If you need updates, you can use level-live-stream.

multilevel.server(db[, authOpts])

Returns a server-stream that exposes db, an instance of levelUp. authOpts is optional and should be of this form:

var authOpts = {
  auth: function (userData, cb) {
    //call back an error, if the user is not authorized.
  },
  access: function (userData, db, method, args) {
    //throw if this user is not authorized for this action.
  }
}

It is necessary that you create one server stream per client.

multilevel.writeManifest(db, path)

Synchronoulsy write db's manifest to path.

Also returns the manifest as json.

var db = multilevel.client([manifest])

Return a new client db. manifest may optionally be provided, which will grant the client access to extensions and sublevels.

db.createRpcStream()

Pipe this into a server stream.

Listen for the error event on this stream to handle / swallow reconnect events.

db.auth(data, cb)

Authorize with the server.

db.deauth (cb)

Deauthorize with the server.

Performance

On my macbook pro one multilevel server handles ~15k ops/s over a local tcp socket.

   bench (master) : node index.js 

writing "1234567890abcdef" 100 times

 native             : 2ms (50000 ops/s)
 multilevel direct  : 21ms (4762 ops/s)
 multilevel network : 14ms (7143 ops/s)

writing "1234567890abcdef" 1000 times

 native             : 12ms (83333 ops/s)
 multilevel direct  : 71ms (14085 ops/s)
 multilevel network : 77ms (12987 ops/s)

writing "1234567890abcdef" 10000 times

 native             : 88ms (113636 ops/s)
 multilevel direct  : 594ms (16835 ops/s)
 multilevel network : 590ms (16949 ops/s)

writing "1234567890abcdef" 100000 times

 native             : 927ms (107875 ops/s)
 multilevel direct  : 10925ms (9153 ops/s)
 multilevel network : 9839ms (10164 ops/s)

Installation

With npm do:

$ npm install multilevel

Contributing

$ npm install
$ npm test
$ npm run test-browser

Contributors

Sponsors

This module is proudly supported by my Sponsors!

Do you want to support modules like this to improve their quality, stability and weigh in on new features? Then please consider donating to my Patreon. Not sure how much of my modules you're using? Try feross/thanks!

License

(MIT)

Copyright (c) 2013 Julian Gruber <julian@juliangruber.com>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the Software), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

multilevel open issues Ask a question     (View All Issues)
  • about 5 years EventEmitter memory leak detected when running 'get'
  • over 5 years evaluate mux-demux alternatives
  • over 5 years multilevel-down
  • over 5 years custom encoding options on db from client.
  • over 5 years No indication that connection is lost
  • almost 6 years Proper approach to close/disconnect
  • over 6 years implement chained batch
  • over 6 years client.createReadableStream()
  • over 6 years handling disconnect/reconnect
  • over 6 years live-stream needs timeout
  • almost 7 years create sublevels from client
  • about 7 years expose db.options on the client
  • about 7 years Encoding
  • about 7 years handling disconnect/reconnect
multilevel open pull requests (View All Pulls)
  • Update memdb to version 1.3.1 ?
  • Updated tape to version 4.2.1
  • Pinned all dependencies
  • wip client side custom encodings
  • Support manifest passed db options
  • propagating end events to server
  • level-live-stream@1.4.12 breaks build 🚨
multilevel questions on Stackoverflow (View All Questions)
  • Parsing the multilevel XML File using java (DOM Parser)
  • Adaptive Multilevel Threshold Algorithms / Way for MATLAB
  • multilevel tree with struts2
  • How to create dynamic multilevel related data version table in informatica
  • Multilevel jQuery menu plugin, accessing and setting up options
  • How to generate multilevel menu items programmatically in angularjs?
  • Jenkins: MultiLevel Access control using Role Based Strategy Plugin
  • Simulink: Perform math operations with multilevel bus signals
  • How can I toggle the multilevel numbering in Word?
  • How do I convert a wide dataframe to a long dataframe for a multilevel structure with 'quadruple nesting'?
  • python key dict error multilevel dict
  • containable behavior in cakephp 3.0 not working in multilevel association
  • multilevel angular js dropdown
  • R package for Multilevel Factor Analysis
  • Meta analysis with multilevel apporach
  • Apply similar function to multilevel pandas dataframe
  • Responsive Multilevel Menu
  • multilevel menu in pure-drawer
  • Multilevel header UI issue in jqgrid
  • Interpreting multilevel negative binomial models in glmmadmb
  • Multilevel summarize with dplyr in R
  • Converting multilevel pandas index to datetime
  • How to merge multilevel (i.e. MultiIndex) dataframes?
  • Multilevel transclusion in Angular 1.5
  • Bootstrap Multilevel Dropdown Menu Not Extending
  • How to get Protected Friend proprty from multilevel inherited class using reflection?
  • laravel-soap multilevel tag structure
  • Android multilevel list of different depth
  • When you have a multilevel list [python], how does the point of reference change in that list?
  • Multilevel Feedback queue preemptive - resumption
multilevel list of languages used
Other projects in JavaScript