Are you happy with your logging solution? Would you help us out by taking a 30-second survey? Click here

cf-mysql-release

Cloud Foundry MySQL Release

Subscribe to updates I use cf-mysql-release


Statistics on cf-mysql-release

Number of watchers on Github 56
Number of open issues 14
Average time to close an issue 18 days
Main language Shell
Average time to merge a PR 12 days
Open pull requests 12+
Closed pull requests 28+
Last commit over 1 year ago
Repo Created about 6 years ago
Repo Last Updated over 1 year ago
Size 12.7 MB
Organization / Authorcloudfoundry
Latest Releasev36.11.0
Contributors55
Page Updated
Do you use cf-mysql-release? Leave a review!
View open issues (14)
View cf-mysql-release activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating cf-mysql-release for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Cloud Foundry MySQL Service

Table of contents

Components

Downloading a Stable Release

Development

Release notes & known issues

Deploying

Registering the Service Broker

Security Groups

Smoke Tests

Deregistering the Service Broker

CI

Components

A BOSH release of a MySQL database-as-a-service for Cloud Foundry using MariaDB Galera Cluster and a v2 Service Broker.

Component Description
CF MySQL Broker Advertises the MySQL service and plans. Creates and deletes MySQL databases and credentials (bindings) at the request of Cloud Foundry's Cloud Controller.
MySQL Server The MySQL instances, either single or 3-node cluster. Currently using MariaDB 10 (versions vary by release).
Proxy Switchboard; proxies to MySQL, severing connections on MySQL node failure.

Proxy

Traffic to the MySQL cluster is routed through one or more proxy nodes. The current proxy implementation is Switchboard. This proxy acts as an intermediary between the client and the MySQL server, providing failover between MySQL nodes. The number of nodes is configured by the proxy job instance count in the deployment manifest.

NOTE: If the number of proxy nodes is set to zero, apps will be bound to the IP address of the first MySQL node in the cluster. If that IP address should change for any reason (e.g. loss of a VM) or a proxy was subsequently added, one would need to re-bind all apps to the IP address of the new node.

For more details see the proxy documentation.

Dashboard

A user-facing service dashboard is provided by the service broker that displays storage utilization information for each service instance. The dashboard is accessible by users via Single Sign-On (SSO) once authenticated with Cloud Foundry. The dashboard URL can be found by running cf service MY_SERVICE_INSTANCE.

Service authors interested in implementing a service dashboard accessible via SSO can follow documentation for Dashboard SSO.

Prerequisites

  1. SSO is initiated when a user navigates to the URL found in the dashboard_url field. This value is returned to cloud controller by the broker in response to a provision request, and is exposed in the cloud controller API for the service instance. A users client must expose this field as a link, or it can be obtained via curl (cf curl /v2/service_instances/:guid) and copied into a browser.

  2. SSO requires the following OAuth client to be configured in cf-release. This client is responsible for creating the OAuth client for the MySQL dashboard. Without this client configured in cf-release, the MySQL dashboard will not be accessible but the service will be otherwise functional. Registering the broker will display a warning to this effect.

    properties:
      uaa:
        clients:
          cc-service-dashboards:
            secret: cc-broker-secret
            scope: cloud_controller.write,openid,cloud_controller.read,cloud_controller_service_permissions.read
            authorities: clients.read,clients.write,clients.admin
            authorized-grant-types: client_credentials
    
  3. SSO was implemented in v169 of cf-release; if you are on an older version of cf-release you'll encounter an error when you register the service broker. If upgrading cf-release is not an option, try removing the following lines from the cf-mysql-release manifest and redeploy.

    dashboard_client:
      id: p-mysql
      secret: yoursecret
    

Implementation Notes

The following links show how this release implements Dashboard SSO integration.

  1. Update the broker catalog with the dashboard client properties
  2. Implement oauth workflow with the omniauth-uaa-oauth2 gem
  3. Use the cf-uaa-lib gem to get a valid access token and request permissions on the instance
  4. Before showing the user the dashboard, the broker checks to see if the user is logged-in and has permissions to view the usage details of the instance.

Broker Configuration

Require HTTPS when visiting Dashboard

The dashboard URL defaults to using the https scheme. This means any requests using http will automatically be redirected to https instead. To override this, you can change jobs.cf-mysql-broker.ssl_enabled to false.

Keep in mind that changing the ssl_enabled setting for an existing broker will not update previously advertised dashboard URLs. Visiting the old URL may fail if you are using the SSO integration, because the OAuth2 client registered with UAA will expect users to both come from and return to a URI using the scheme implied by the ssl_enabled setting.

Note: If using https, the broker must be reached through an SSL termination proxy. Connecting to the broker directly on https will result in a port 443: Connection refused error.

Trust Self-Signed SSL Certificates

By default, the broker will not trust a self-signed SSL certificate when communicating with cf-release. To trust self-signed SSL certificates, you can change jobs.cf-mysql-broker.skip_ssl_validation to true.

Downloading a Stable Release

Stable releases, also known as final releases, are available for general use. Release notes and source code are available on github. Instructions for uploading a final release to your BOSH director can be found on bosh.io.

Note: If your BOSH director's able to access the Internet, you don't need to download and upload a release to your BOSH director. When using cf-mysql-deployment, the correct release is referenced in the manifest, and will be automatically retrieved by the BOSH director.

Development

See our contributing docs for instructions on how to make a pull request.

This BOSH release doubles as a $GOPATH. It will automatically be set up for you if you have direnv installed.

# fetch release repo
mkdir -p ~/workspace
cd ~/workspace
git clone https://github.com/cloudfoundry/cf-mysql-release.git
cd cf-mysql-release/

# switch to develop branch (not master!)
git checkout develop

# automate $GOPATH and $PATH setup
direnv allow

# initialize and sync submodules
./scripts/update

If you do not wish to use direnv, you can simply source the .envrc file in the root of the release repo. You may manually need to update your $GOPATH and $PATH variables as you switch in and out of the directory.

Release Notes, Known Issues, and Documentation

Release Notes

Known Issues

For more information, check out the documentation.

Deploying

See https://github.com/cloudfoundry/cf-mysql-deployment to deploy cf-mysql release.

cf-mysql-release open issues Ask a question     (View All Issues)
  • almost 3 years How to apply max_connect_errors, wait_timeout, interactive_timeout to mysql?
  • about 3 years When does cf-mysql-release support for Openstack?
  • about 4 years Encrypted connection support for MySQL (MariaDB) in Cloud Foundry
  • over 4 years Is it possible to run this without a CF?
cf-mysql-release open pull requests (View All Pulls)
  • Add ability to override deployment name
  • Add ability to override number of compilation workers
  • Trim release size
  • Add openstack stub
  • Allow the main mysql host to be specified via config.
  • Default cf_mysql.host to nil
  • `./scripts/generate-deployment-manifest` now takes a real cf manifest
  • Add a way to set `ulimit -n' for the proxy job
  • Allow for override of the innodb_buffer_pool_instances
  • Specify the cf-mysql-broker ip via manifest
  • add performance_schema property
  • Mysql evs props
cf-mysql-release questions on Stackoverflow (View All Questions)
  • How to install bosh-lite cf-release and cf-mysql-release
cf-mysql-release list of languages used
cf-mysql-release latest release notes
v36.11.0 v36.11.0

v36.11.0

  • New feature: As a Space Developer, I'd like to be able to create read-only bindings to a Service Instance [#129885695], [#154575192]
  • Improved: The Operator would like pre-seeded users' passwords to be updated when changed in the manifest [#153529269]
  • Improved: Many small usability updates to download-logs, including now relying on bosh ssh rather than having manual SSH set up independently.
    • As an Operator, download-logs should leverage bosh CLI where possible so that I don't have to figure out ssh on each IaaS [#154609725]
    • As an Operator, I'd like to be able to distinguish the artifacts generated by download-logs [#154299373]
    • Bug fix: download_logs should not fail when a customer has other jobs installed that aren't readable by user vcap [#148156397]
    • As an Operator, it would be nice if download logs could contain the output of bosh vms and bosh instances [#154299563]
    • As a support team member, I'd like download-logs to include the sst log files [#154013852]

Dependency Updates and Bug Fixes

  • cf-mysql-release shall use MariaDB 10.1.30 [#154149018]
    • MariaDB 10.1.31 failed to build cleanly, we'll look into it and catch up in a coming release.
  • Bump cf-mysql-release to use xtrabackup 2.4.8 [#153101615]
  • Bug fix: Cluster nodes should not hang in pre-start after unclean shutdown of cluster [#153572034]
  • Bug fix: remove logging to syslog from cf-mysql-release [#153301323]

We discovered that in switching over to syslog-release, we hadn't fully stopped independently sending logs to syslog. We've pared that back.

  • A known issue remains with routing-release, which may be addressed some time in the future. Track issue #97 for any updates.

Deprecation Notice: ARP Cache

When a MySQL server VM disappears, BOSH will recreate that VM with the same IP address. When that new VM does not share the same ethernet MAC address as the old one, switchboard will not be able to direct connections to the new VM.

BOSH has a feature to clear the old ethernet address from switchboard's cache, but this feature used to default to off. For this reason, switchboard contained a duplicate feature to ensure that it is able to reach the new VM.

Recently, we discovered that this code does not work when BOSH DNS is enabled. This bug has been fixed, see below. However, in doing, we've learned that the switchboard feature does not work on GCP, and possibly other IaaSes.

The BOSH feature is designed to work on any IaaS that BOSH supports. In a coming release, we will drop the duplicate switchboard feature.

We strongly suggest that you run all of your Directors with flush-arp enabled. If you do not have flush-arp enabled, you may encounter issues where switchboard is not able to reach a back end node, even after successful resurrection.

  • Bug fix: ARP Cache flushing broken in switchboard when BOSH DNS is enabled [does not fix GCP] [#154302130]

Changes to Manifest

  • No changes
v36.10.0 v36.10.0

Syslog Changes

  • Sally can see that cf-mysql-release generates syslogs using a method that does not interfere with other software in the deployment [#152567983]

Breaking Change: We've cut independent calls to syslog from cf-mysql-release and now depend on syslog-release to do that work for us. If you are using cf-mysql-deployment, simply use the enable-syslog.yml operations file, and provide the following variables:

  • syslog_address
  • syslog_port
  • syslog_transport

Deploying with the enable-syslog.yml operations file will affect only the cf-mysql deployment. You don't need to use it if you already use syslog-release as a BOSH add-on.

Minor Features and Bug Fixes

Changed default

In #144637793, we introduced the new property, cf_mysql.mysql.enable_local_file and set the default to false. This is inconvenient, because it affects commonly used tools such as mysqlimport. The property is still there, but we've switched the default to true.

  • enable_local_file default overly restrictive [#148389277]

Other Changes

  • [PR #185] Configureable slow query threshold [#151987215]
    • Introduces a new property, cf_mysql.mysql.long_query_time, which you can set to the minumum number of seconds a query must run before showing up in the slow query log.
  • Update versions of ruby and gems used by broker (security) [#148614637]
    • Updates dependencies to address security concerns.
  • Bug fix: cloudfoundry/cf-mysql-release #184: Roadmin user doesn't have read privs for admin operations? [#151643206]
  • Bug fix: MySQL fails to start if the node once had IPSec installed but IPSec is no longer installed [#152740268]
  • Bug fix: Binky and other applications should not be able to write directly to a database that is in upgrade mode. [#151236116]

New Documentation

  • cloudfoundry/cf-mysql-release #192: We've added a document, backup-restore.md, that covers how to scale down and ease the restriction on the storage engine so that you can do a full restore without having issues with the MySQL system tables.

Manifest Changes

  • Removed: syslog_aggregator.address
  • Removed: syslog_aggregator.port
  • Removed: syslog_aggregator.transport
v36.9.0 v36.9.0

Minor updates

The focus of this release is to catch up on a few items that didn't make v36.8.0.

  • [PR #163] Disable running tests when packaging galera [#145631809]
  • Bug fix: IPsec causes mariadb_ctrl to be left in an Execution Failed state when taking time to start [#152650758]
  • Sally does not want to see a new warning from BOSH when running broker-registrar [#151862716]
  • Sally would like the Interruptor turned off by default [#152651619]
    • Although we had some troubles last year, we are well past them. The default has long been set to disable automatic SST, which can be inconvenient especially when cf-mysql-release is used by cf-release and cf-deployment. This paranoid option is still available, we're just changing the default to false.

Changes to Manifest

  • No changes

As always, please refer to cf-mysql-deployment for deployment instructions, manifest and sample overrides files.

Other projects in Shell