Technology moves fast! ⚑ Don't get left behind.🚢 Subscribe to our mailing list to keep up with latest and greatest in open source projects! πŸ†


Subscribe to our mailing list

Singularity

Scheduler (HTTP API and webapp) for running Mesos tasksβ€”long running processes, one-off tasks, and scheduled jobs. #hubspot-open-source

Subscribe to updates I use Singularity


Statistics on Singularity

Number of watchers on Github 713
Number of open issues 41
Average time to close an issue 3 days
Main language Java
Average time to merge a PR 5 days
Open pull requests 189+
Closed pull requests 50+
Last commit about 1 year ago
Repo Created over 5 years ago
Repo Last Updated about 1 year ago
Size 29 MB
Homepage http://getsingula...
Organization / Authorhubspot
Latest ReleaseSingularity-0.19.1
Contributors35
Page Updated
Do you use Singularity? Leave a review!
View open issues (41)
View Singularity activity
View on github
Fresh, new opensource launches πŸš€πŸš€πŸš€
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating Singularity for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Singularity

Latest Release Version: 0.14.1

Mesos Version: 0.28.2

Build Status

Overview

Singularity is an API and web application for running and scheduling Apache Mesos tasks including long running processes, scheduled jobs, and one-off tasks.

It focuses on a batteries-included approach: Singularity and its components provide an entire Platform as a Service (PaaS) to end-users. It has many features which have been introduced to reduce developer friction and ensure proper operation and reliable deployment of tasks. Users may even be unfamiliar with and shielded from the details of Mesos.

For a more thorough explanation of the concepts behind Singularity and Mesos click here.


Features


Try It Out!

If you want to give Singularity a try, you can install docker and docker-compose to run our example cluster.

Run docker-compose pull first to get all of the needed images. Note: This may take a few minutes

Then simply run docker-compose up and it will start containers for...

  • mesos master
  • mesos slave (docker/mesos containerizers enabled)
  • zookeeper
  • Singularity
  • Baragon Service for load balancer management
  • Baragon Agent + Nginx as a load balancer

...and the following UIs will be available:

if using boot2docker or another vm, replace localhost with the ip of your vm


Getting Started

Requirements
Contact

Reference

Installation

Deployment

Development


Singularity is built and used by a number of different companies. Check out the list of adopters here.

Singularity open issues Ask a question     (View All Issues)
  • over 2 years The method runSingularityRequest() on SingularityClient should return the JSON response
  • over 2 years Missing "command" field in deploy request from UI when using custom executor
  • over 2 years Add a timeout for decommissioning
  • over 2 years UIs unavailable after running docker-compose commands in docs
  • over 2 years Categorization on /slaves page is confusing and hard to read
  • over 2 years Singularity disaster detection itself needs rate limiting
  • over 2 years api-docs link not working
  • over 2 years Singularity should support healthchecks on statically defined service ports
  • over 2 years Would be nice if SingularityDeploy env vars in could be interpolated
  • over 2 years singularity doesnt support aws IAM credentials
  • over 2 years expose particular docker-related ID
  • almost 3 years Container IP
  • almost 3 years Run same OnDemand Request with different parameters set in commandline
  • almost 3 years Support for multi-job dependency
  • about 3 years Swagger docs - invalid $ref fields
  • about 3 years Integration with Mesos maintenance mode
  • about 3 years OVERDUE task state should indicate which resource it is lacking
  • over 3 years Continuous health checks
  • over 3 years handle TASK_ERROR appropriately
  • over 3 years Partial request updates via UI and API
  • over 3 years Allow changing request type from SERVICE to WORKER and vice versa
  • about 4 years investigate a new request type for running a task on every slave
  • about 4 years UI file browser should show images properly
  • about 4 years API Docs for SingularityDeploy should include defaults
  • over 4 years Support and documentation for custom Mesos slave resources
  • over 4 years Singularity should be able to handle registration, re-registration, and leadership changes without aborting
  • over 4 years make sure that jade4j works with java 8
  • over 4 years Singularity should abort if it does not successfully negotiate a driver connection
  • about 5 years user input via stdin for on-demand tasks
  • over 5 years basic auth
Singularity open pull requests (View All Pulls)
  • Task kill checkbox label
  • Deploy Failure Information
  • cut down on amount of task data stored in ZK
  • use pushState for log line links
  • Task Search UI
  • add support for max numbers of objects in ZK when no database is configured
  • Allow destination targets to be specified
  • Add ability to post task metadata to a task
  • (WIP) Task search
  • automatically run DB migrations at Singularity startup
  • Incremental deploys
  • More interrupt exception logic clarification.
  • smarter ordering for queries that don't use zk first
  • Don't display healthcheck message for tasks in state TASK_LOST or TASK_FINISHED
  • Tail Logs From Bottom of File (WIP)
  • RFC 5545 recur schedule support
  • missing an early return for numPorts == 0
  • get memory info from deploy object instead of mesos task resources
  • Die brunch
  • Upgrade basepom
  • New task checker healthcheck retries
  • Healthcheck notification won't display if task state is TASK_FAILED
  • Even number across racks
  • Fix time display emails
  • Fix and Rename getQuartzScheduleSafe() Method
  • Better failure message when creating a scheduled reqeust with no schedule
  • fix tasks failed count for deploy failure info
  • Improvements to Task Overview on Task page
  • wire up new LB fields
  • Remove cleanup after bounce expire and slave reactivation
  • WIP: Log Tailer Rewrite
  • ability to update request data upon successful deploy
  • Thread check types
  • optional redirect in ui when receiving a 401 from the api
  • allow multiple on demand requests to be queued up
  • Allow glob matching in addition to fuzzy matching on Requests and Tasks page
  • Better Request and Deploy ID Validation
  • Count non-active requests towards request totals on dashboard page
  • Show TASK_ERROR state as danger in the UI
  • Transition Resource Usage Badge & Bar Back to Normal When CPU Safe
  • Aggregate tailer better handle logs not found
  • add support for per-task labels + surfacing a label as a LB upstream group
  • Aggregate tailer tooltip host on instance number and tasks dropdown
  • Reactify racks and slaves pages
  • Changes to user requested task kills in custom executor
  • Tweaks to s3UploaderAdditionalFiles
  • Upgrade Mesos Version
  • add support for disk resource
  • UI improvements with global search tool
  • draft of writeup for 0.6.0 release
  • avoid index out of bounds when finding tasks to shut down
  • Add a "defaultPortMapping" option which exposes all Mesos provided ports
  • Change package.json 'vex' dependency to 'vex-js'
  • (WIP) Support task history search for 'running at'
  • UTF-8 support in chunked logs (WIP)
  • Kill tasks below a certain priority level
  • try pull one extra time for a DockerRequestException
  • configurable max metadata message length
  • only show Task killed in message for healthchecks if not in running state
  • Add support for hourly logrotate
  • add downloadUrl field, for explicitly downloading from S3
  • support overriding the log level in tests
  • fix docker integration environment
  • Avoid race condition in deploy to unpause
  • Fix table paging
  • Only show wait for replacement checkbox if a WORKER or a SERVICE
  • Remove extraneous console.logs
  • Fix broken shell log link
  • handle new lines in docker environment variables correctly
  • SingularityUI Conversion Phase 1: Coffee => ES6, Stylus => SCSS, Reactify/Reduxify Pages
  • Reimplementation of RFC5545 Schedule
  • use 0.28.2 to fix MESOS-5449
  • Concurrent run once tasks
  • Bump moment minor version to mitigate CVE-2016-4055
  • New Requests page (WIP)
  • Always launch docker containers in the correct parent cgroup
  • Link to tail_of_finished_service.log
  • React/redux Requests Page
  • New new deploy page (WIP)
  • New tasks page
  • allow resource override to be set on a run now request
  • add method for grabbing a snapshot of the master metrics to MesosClient
  • Fix filename truncation in sandbox endpoint
  • sort task history updates by ExtendedTaskState enum ordinal, not timestamp
  • fail differently if request data not present during deploy check
  • remove the now > start check in RFC5545
  • Do not remove from LB if not yet added
  • Add TASK_ID environment variable with the singularity task_id
  • WIP: support multiple docker parameters and task labels
  • Redo gulp & webpack (WIP)
  • Singularity Request - Timezone field (API)
  • Docker auth config
  • SingularityService configuration & DC/OS support
  • surface taskReconciliationStartedAt in SingularityState object
  • Traffic splitting round 2 (WIP)
  • Tags input tweaks (for cmd line input)
  • Starred requests persistence frontend (WIP)
  • (WIP) Get count for blended history calls
  • Starred requests persistence
  • Use direct string matching with glob patterns instead of fuzzy matching
  • Request group actions
  • Redeploy
  • allow searching of deleted request logs
  • Guarantee durationMillis is present in getExpiringBounce response
  • Request Form Improvements
  • deploy failure message for non-task-specific failures
  • Code cleanliness improvements
  • support for task id var substitution in env
  • Better cleanup of incremental tasks
  • Execution timeout for tasks
  • undo click-to-copy changes
  • Support for setting user in default executor
  • logfetch support for bzip2
  • support for alternate logrotate compression
  • fix typo in task alerts
  • Tailer
  • add team requests to dashboard
  • Automatically refresh some tables
  • Disaster detection and disabled actions
  • Starred requests persistence frontend
  • Use direct string matching with glob patterns instead of fuzzy matching
  • Redeploy
  • Multiple deploy groups
  • use saveTaskCleanup() instead of createTaskCleanup() for deploys
  • expand requestHistory.createdAt column to millisecond precision
  • (WIP) Ability to search task history by runId
  • Add global read only groups
  • Add pending request on failed deploy as well
  • Shortcut to task by instance number
  • (WIP) Smarter Healthchecks
  • More even distribution among racks
  • Consider tasks with skipped healthchecks in cleaner
  • duplicate code refactoring
  • added threshold for startup tasks during singularity launch
  • Process status updates in separate thread
  • Refresh task lists appropriately
  • extra filtering on fuzzy match
  • WIP - Search UI Improvements
  • also grab containerId when grabbing directory
  • Fix another js TypeError on task detail page
  • Introduce expiring machine state changes
  • Fix custom executor command on new deploy page
  • replace instead of ignore so that we can update the offset
  • Sanitize file data before parsing with Jackson
  • Docs and deprecated fields updates
  • Nicer format for disaster email
  • Update uuid to version 3.0.0
  • Count losts tasks with a Meter
  • Check isEmpty on attributes for more accurate message
  • introduce offer cache to allow better resource allocation
  • Ability to override docker workdir
  • add slave and task usage tracking inside singularity ZK
  • Add ability to upload files immediately to S3
  • Clarifications on S3ArtifactSignature
  • [logfetch] Use the new paginated s3 search endpoint
  • S3 Search Improvements
  • Deleting state
  • [WIP] Force Deploy - allow a new deploy to override an existing deploy
  • Canary deploy
  • Add support for spread task to all slaves.
  • rephrase flagging host
  • Add skip lb removal flag to DeleteRequestRequest
  • Easy endpoint to check if user is authorized for request
  • Deleting request
  • Add flag to run scheduled task on deploy
  • Fix maxDeployIdSize and maxRequestIdSize validation
  • Better task balancing
  • Release lock on bounce when bounced with no running instances
  • Fix allowRunningTasksToFinish in pause modal
  • Baragon 0.5.0
  • Initial support for Mesos containerizing Docker images directly
  • (WIP) Immutables
  • Allow immediate run in pending queue with deploy
  • reduce S3 uploader mem usage
  • Support for gpus
  • Bug: Ignore healthcheck config when skipping checks
  • Smarter table pagination
  • Capital search
  • Singularity performance improvements
  • add support for overriding S3 endpoint URL for the downloader
  • (WIP) upgrade swagger and use open api 3.0
  • Add the ability to shuffle long running tasks when a host is overloaded
  • Better check for finish of a bounce
  • Uploader refactoring and additional attempts for immediate uploaders
  • Account for expected usage when scoring offers
  • Remove expiring scale when new scale has no expiration
  • Wire up slack notifications
  • Add a new scoring mode accounting for max historical usage of all tasks
  • WIP - Move to java.util.Optional and use immutables
Singularity questions on Stackoverflow (View All Questions)
  • nlme: Error message "Singularity in backsolve at level 0, block 1 "
  • how to avoid singularity in numerical integration?
  • VS Community 2015 Sass Singularity
  • using eigenvalues to test for singularity: identifying collinear columns
  • Sage Math elliptic curve with singularity points
  • How I can use Singularity Extras in a libsass workflow?
  • Using Singularity with LibSass
  • singularity css grids add one gutter to the width
  • Singularity zero-width gutters
  • singularity gs remove the left/right margins
  • How to handle singularity in MATLAB LMS filter?
  • Singularity: can I pass an argument to grid span to change the float direction?
  • How can i use Singularity GS with only 1 master grid?
  • Thumbnail Grid - using Singularity
  • How to use Singularity and Breakpoints without Ruby Sass on Drupal?
  • Troubles with singularity of log when trying to plot a function in Scilab
  • An Integral with Singularity in MATLAB
  • Why don't my floats in my Singularity grid fill all the columns?
  • A double integral with singularity MATLAB
  • How to try-except an illegal matrix operation due to singularity in NumPy
  • Dymola-Modelica. Singularity Error
  • Singularity multiple gutters being ignored
  • Is there any way to run susy or singularity with libsass?
  • Backward selection in LME, singularity in backsolve occured
  • Fixed width column in a singularity grid?
  • Singularity for inverse matrix
  • Singularity Grid System showing the grid when using a 'container'
  • What is correct markup for float-spans within breakpoints in a singularity grid
  • singularity error regressing lag in R
  • Singularity responsive gutter widths
Singularity list of languages used
Singularity latest release notes
Singularity-0.19.1 Singularity 0.19.1

Changes in 0.19.1

This is a bugfix release

Check out the 0.19.1 milestone to see bugfixes in detail.

  • 1740 - Only send healthcheck object from ui when setting all fields
Singularity-0.17.1 Singularity 0.17.1

Changes in 0.17.1

This is a bugfix release

Check out the 0.17.1 milestone to see bugfixes in detail.

  • 1740 - Only send healthcheck object from ui when setting all fields
Singularity-0.19.0 Singularity 0.19.0

Changes in 0.19.0

Check out the 0.19.0 milestone to see new features / bugfixes in detail.

New Features

  • 1668/1650 Adds support for more overrides in run now requests. You can now override items like environment variables for individual runs of a SingularityRequest by POSTing json in the form of a SingularityRunNowRequest
  • 1690 - Adds initial support for Mesos containers, with volume sources and network mapping. These can be specified in the containerInfo.volumes and containerInfo.mesos.image sections of the SingularityContainerInfo in your SingularityDeploy
  • 1702 - Updates the internal locking scheme for Singularity to allow more parallel processing. As a result, the concurrency of offer and status update processing can now be tuned, with increased concurrency coming with a cost of increased memory/cpu usage by the scheduler. The following parameters in the mesos section of the SingularityConfiguration impact concurrency and tuning of the scheduler:
    • statusUpdateConcurrencyLimit - The number of status updates that can be processed in parallel. Defaults to 500 and is backed by its own cached thread pool
    • maxStatusUpdateQueueSize - A semaphore limits the number of submissions to the status update cached thread pool. If there are currently no more permits available (i.e. > statusUpdateConcurrencyLimit status updates), these are added to a queue where they wait until more capacity is available. This configuration parameter controls the max size of that queue. It is recommended that this be set a bit above the maximum number of tasks you expect to have active in Singularity at any one time, due to the fact that during reconciliation a status update for each task is sent in rapid succession.
    • offersConcurrencyLimit - The number of offer scoring calculations and checks to be done in parallel. Defaults to 100. This should generally not need to be updated.

Improvements

API/Scheduler

  • 1666 - Set AVAILABILITY_ZONE on the default task environment.
  • 1681 - Provide option to prevent emails for scale events
  • 1653 - Endpoint to allow users to delete pending on-demand tasks
  • 1657 - Include disk resources when scoring offers
  • 1683 - Support usage of DefaultServerFactory to find port
  • 1692 - Wire up overrides for the S3 uploader path.
  • 1616 - Remove swagger jackson version override
  • 1600 - Report disk usage
  • 1690 - Second pass at Mesos containers, with volume sources and network mapping
  • 1682 - Ability to update authorized groups separately from full request
  • 1695 - Allow the cors bundle to be more configurable
  • 1697 - Support uploads to GCS
  • 1703 - Better webhook auth timeouts and exception messages
  • 1699 - add support for overriding S3 endpoint URL for the downloader
  • 1707 - Enable server side encryption params for uploads
  • 1717 - Collect cpu usage for tasks that have just started

UI

  • 1654 - Ability to specify quick links for requests
  • 1636 - New dashboard in Singularity + UI performance improvements
  • 1604 - Disk usage ui
  • 1687 - Correct copy button on InfoBlocks of Task tab. Also, remove clipboard.js.
  • 1704 - Add support for configurable navbar title links.
  • 1701 - Permalinks for bounce and scale modals
  • 1705 - Capitals search on Singularity requests page

Bug Fixes

  • 1609 - Use HostAndPort#getHostText instead of HostAndPort#getHost
  • 1658 - Corrected path for executor download fallback
  • 1659 - Make sandbox logs/ dir world-readable.
  • 1685 - Allow a request in FINISHED state to be redeployed
  • 1688 - Also check pending requests on the track task endpoint
  • 1693 - Properly send task destroy message to executor
  • 1698 - Don't show s3 logs error message as a pop up
  • 1708 - Better catch for statusUpdate exceptions
  • 1718 - Account for task level overrides in usage collection
  • 1719 - Remove unneeded call to unsafeProcessStatusUpdate, fix tasksPerOfferHost check
  • 1711 - Take system usage into account when scoring offers
  • 1696 - Fix nav bar for mobile view
  • 1715 - Use total system cpus, not totalCpus in system calculation
  • 1713 - Fix mobile menu responsiveness
  • 1726 - Don't operate directly on pending tasks during statusUpdate
  • 1721 - Fix cached offer checkin in resourceOffers

Documentation

  • 1700 - Fixed task webhook docs

Thanks

  • @ssalinas
  • @benheng
  • @tpetr
  • @kwm4385
  • @Tamba32
  • @baconmania
  • @stevegutz
  • @stevenschlansker
  • @darcatron
Other projects in Java