Want to take your software engineering career to the next level? Join the mailing list for career tips & advice Click here

genesis

A tool for data center automation

Subscribe to updates I use genesis


Statistics on genesis

Number of watchers on Github 153
Number of open issues 17
Average time to close an issue 1 day
Main language Ruby
Average time to merge a PR 2 days
Open pull requests 8+
Closed pull requests 1+
Last commit over 3 years ago
Repo Created about 6 years ago
Repo Last Updated over 2 years ago
Size 8.4 MB
Homepage http://tumblr.git...
Organization / Authortumblr
Latest Releasev1.3.0
Contributors9
Page Updated
Do you use genesis? Leave a review!
View open issues (17)
View genesis activity
View on github
Book a Mock Interview With Me (Silicon Valley Engineering Leader, 100s of interviews conducted)
Software engineers: It's time to get promoted. Starting NOW! Subscribe to my mailing list and I will equip you with tools, tips and actionable advice to grow in your career.
Evaluating genesis for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Genesis

Introduction and motivation

Genesis is a tool for data center automation. The primary motivation for developing Genesis at Tumblr was to streamline the process of discovering new machines and reporting their hardware details to Collins, our inventory management system, without having to do a bunch of data entry by hand. In addition, we've also extended Genesis to be a convenient way to do hardware configuration such as altering BIOS settings and configuring RAID cards before provisioning an operating system on to the host.

From a high-level point of view, Genesis consists of a stripped down linux image suitable to boot over PXE and a ruby DSL for describing tasks to be executed on the host.

This repository also includes a test environment which is suitable for building the linux image.

Framework

genesis_framework framework

genesis_retryingfetcher retryingfetcher

genesis_promptcli promptcli

The Genesis framework and supporting gem can be found in src

Tasks

Tasks are created using the Genesis DSL which makes it easy to run commands, install packages, etc. in the stripped down environment.

Examples of tasks are the TimedBurnin task, which performs a stress test on the system to rule out hardware errors before putting it into production, and BiosConfigrR720, which sets up the BIOS on Dell R720s just the way we want it.

General workflow

There are a couple of systems apart from Genesis that need to be in place for a successful deployment. These are

  • a DHCP server,
  • a TFTP server,
  • and a file server (serving static files over HTTP)

More detail on setting these up is documented in INSTALL.md.

When a machine boots, the DHCP server tells the PXE firmware to chain boot into iPXE. We then use iPXE to present a list of menu choices, fetched from a remote server. When the user makes a choice we load the Genesis kernel and initrd (from the file server) along with parameters on the kernel command line. Once the Genesis OS has loaded, the genesis-bootloader fetches and executes a ruby script describing a second stage where we install gems, a few base RPMs, and fetch our tasks from a remote server. Finally, we execute the relevant tasks.

For a real world example; consider a brand new server that boots up. It makes a DHCP request and loads the iPXE menu. In this case, we know that we haven't seen this MAC address before, so it must be a new machine. We boot Genesis in to discovery mode, where the tasks it runs are written to fetch all the hardware information we need and report it back to the Collins. In our setup this includes information such as hard drives and their capacity and the number of CPUs, but also more detailed information such as service tags, which memory banks are in use, and even the name of the switchports all interfaces are connected to. We then follow this up with 48 hours of hardware stress-test using the TimedBurnin task.

Test environment

To avoid testing Genesis in production, we've set up a virtual test environment based on VirtualBox. This allows for end-to-end testing of changes to the framework, new tasks, etc.

More information about the test environment and setting it up can be found in testenv/README.md.

Building an Image

To make it easy to get started with genesis, we have included a Dockerfile that will allow you to compile a bootable live image without needing a lot of client side configuration. If you have docker running on your machine and just want to build the latest images:

# mkdir output
# docker run --privileged=true -v $(pwd)/output:/output tumblr/genesis-builder
# ls output

To build a custom image, if you have tweaked something about genesis, you can present build a custom builder image:

# docker build -f Dockerfile -t genesis-builder .
# docker run --privileged=true -v $(pwd)/output:/output genesis-builder
# ls output

NOTE: the genesis-builder uses livecd-creator which depends on loopback mounts. These don't currently (2016-08-25) work with Docker for Mac. On linux you should make sure you have at least 2 spare /dev/loop* devices, losetup -a will show which ones are busy and mknod /dev/loop# -m0600 b 7 # a couple if needed. Also you may need to cleanup/free (losetup -d) devices after this has run.

Contact

Please feel free to open issues on GitHub for any feedback or problems you might run in to. We also actively encourage pull requests. Please also make sure to check CONTRIBUTING.md.

License

Copyright 2016 Tumblr Inc.

Licensed under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

genesis open issues Ask a question     (View All Issues)
  • about 4 years vagrant test image 404s
  • over 4 years Task failure doesnt include any backtrace
  • over 4 years Log DSL method has no support for severity levels
  • almost 5 years Releases should include precompiled bootable images
  • almost 5 years Use a more robust configuration mechanism for customizing images
  • almost 5 years UEFI Support
  • almost 6 years Clean up config.yaml.erb
  • almost 6 years Documentation on setting up a Genesis environment in production
  • almost 6 years "General workflow" doesn't align with the features released
  • almost 6 years Suggestion to use vagrant-vbguest plugin
  • almost 6 years Separate task specific configs from main genesis config
genesis open pull requests (View All Pulls)
  • Genesis config access by string keys
  • Marantz retries array
  • base image improvements
  • Add task to disable CPU frequency scaling
  • Add serial consoles to ACTIVE_CONSOLES
  • Convert GENESIS_PROMPT_TIMEOUT to integer
  • Add TESTING.md and Makefile
  • [Tasks] Modules/logging - Added condition for Collins
genesis questions on Stackoverflow (View All Questions)
  • Customizing Wordpress (Genesis) Themes
  • Where to set viewport size in wordpress theme genesis
  • How to remove wp customize sections in genesis
  • Remove - before leave a comment on Wordpress Genesis theme
  • How to register a widget in WordPress Genesis child theme?
  • Genesis custom header implementing add_action within new function
  • Register widget area above posts in WordPress Genesis child theme
  • Wordpress, Genesis theme, streamline pro, How to add an extra .PHP file?
  • How can I create a customized form handler in wordpress genesis framework?
  • Add custom field in Genesis Featured Posts widget output
  • Secondary menu not showing on Wordpress Genesis
  • Wordpress Genesis – Create two menu locations in header (primary + secondary)
  • Remove Genesis Default Sidebar not working
  • Genesis: How can headline of post title
  • I am trying to create new genesis block Twister-Core using Ubuntu and Qt Creator IDE
  • WordPress Genesis Framework: <!--nextpage--> not working
  • Wordpress Genesis Framework add Class
  • Post loop shortcode in Genesis Framework
  • If I completely remake the home page of a studiopress child theme, what will the effects be if the genesis framework gets updated?
  • Need help showing pagination on genesis custom archive template
  • Genesis Framework - How to switch main hook and sidebar hook order
  • Equal column height for each row - Genesis Columns
  • WordPress Genesis: adding co-authors to custom genesis loop
  • WordPress Theme (Genesis) Login Button in PHP
  • Ethereum Genesis Block Private Network
  • How to know that a existing plugin is compatible to genesis framework or not. If not then how to make it compatible to it
  • Add CDN CSS before style.css in genesis
  • Adding Google phone conversion tracking to Genesis Framework
  • wp-types plugin : Cretae Custom Fields Groups is not working in genesis
  • Dynamically include html page in Genesis
genesis list of languages used
genesis latest release notes
v1.3.0 2017-07-12

Most visible changes:

  • Genesis run by init, terminals can tail the log
  • Added basic support for loglevel in log messages
  • support array AND enumerable for retries values
  • facter caching improvements

Plus bug fixes

v1.2.0 2016-10-06

Added dmidecode to the base image

v1.1.0 2016-03-18
  • includes bind-utils
  • include ntpdate in the bootable image and sets the system time if possible to help prevent https certificate validation errors that could happen
Other projects in Ruby
Powered by Autocode - Instant Webhooks, Scripts and APIs
Autocode logo wordmark