Are you happy with your logging solution? Would you help us out by taking a 30-second survey? Click here

ice

AWS Usage Tool

Subscribe to updates I use ice


Statistics on ice

Number of watchers on Github 2449
Number of open issues 101
Average time to close an issue about 1 month
Main language Java
Average time to merge a PR about 1 month
Open pull requests 38+
Closed pull requests 17+
Last commit over 2 years ago
Repo Created over 6 years ago
Repo Last Updated over 1 year ago
Size 2.44 MB
Organization / Authorteevity
Latest Releasev1.1.2
Contributors23
Page Updated
Do you use ice? Leave a review!
View open issues (101)
View ice activity
View on github
Fresh, new opensource launches 🚀🚀🚀
Trendy new open source projects in your inbox! View examples

Subscribe to our mailing list

Evaluating ice for your project? Score Explanation
Commits Score (?)
Issues & PR Score (?)

Ice

Build Status

Intro

Ice provides a birds-eye view of our large and complex cloud landscape from a usage and cost perspective. Cloud resources are dynamically provisioned by dozens of service teams within the organization and any static snapshot of resource allocation has limited value. The ability to trend usage patterns on a global scale, yet decompose them down to a region, availability zone, or service team provides incredible flexibility. Ice allows us to quantify our AWS footprint and to make educated decisions regarding reservation purchases and reallocation of resources.

Ice is a Grails project. It consists of three parts: processor, reader and UI. Processor processes the Amazon detailed billing file into data readable by reader. Reader reads data generated by processor and renders them to UI. UI queries reader and renders interactive graphs and tables in the browser.

What it does

Ice communicates with AWS Programmatic Billing Access and maintains knowledge of the following key AWS entity categories:

  • Accounts
  • Regions
  • Services (e.g. EC2, S3, EBS)
  • Usage types (e.g. EC2 - m1.xlarge)
  • Cost and Usage Categories (On-Demand, Reserved, etc.) The UI allows you to filter directly on the above categories to custom tailor your view.

In addition, Ice supports the definition of Application Groups. These groups are explicitly defined collections of resources in your organization. Such groups allow usage and cost information to be aggregated by individual service teams within your organization, each consisting of multiple services and resources. Ice also provides the ability to email weekly cost reports for each Application Group showing current usage and past trends.

When representing the cost profile for individual resources, Ice will factor the depreciation schedule into your cost contour, if so desired. The ability to amortize one-time purchases, such as reservations, over time allows teams to better evaluate their month-to-month cost footprint.

Screenshots

  1. Summary page grouped by accounts Summary page grouped by accounts

  2. Detail page with throughput metrics and grouped by products Detail page with throughput metrics and grouped by products

  3. Reservation page grouped by on-demand, un-used, reserved, upfront costs Reservation page

  4. Reservation page with on-demand cost and grouped by instance types Reservation page with on-demand cost and grouped by instance types

  5. Breakdown page of Application Groups Breakdown page of Application Groups

Prerequisite:

  1. First sign up for Amazon's programmatic billing access here to receive detailed billing(hourly) reports. Verify you receive monthly billing file in the following format: <accountid>-aws-billing-detailed-line-items-<year>-<month>.csv.zip.
  2. Install Grails 2.4.4 and set GRAILS_HOME and JAVA_HOME
  3. Ice uses highstock to generate interactive graphs. Please make sure you acquire the proper license before using it.

Basic setup:

Using basic setup, you don't need any extra code change and you will use the provided bootstrap.groovy. You will need to construct your own ice.properties file and have it in your classpath (I put it at src/java). You can use sample.properties (located at src/java) file as the template.

  1. Processor configuration

1.1 Enable processor in ice.properties:

      ice.processor=true

1.2 In ice.properties, set up the local directory where the processor can copy the billing file to and store the output files. The directory must pre-exist. For example:

      ice.processor.localDir=/mnt/ice_processor

1.3 In ice.properties, set up the working s3 bucket and working s3 bucket file prefix to upload the processed output files which will be read by reader. Ice must have read and write access to the s3 bucket. For example:

      ice.work_s3bucketname=work_s3bucketname
      ice.work_s3bucketprefix=work_s3bucketprefix/

1.4 If running locally, set the following system properties at runtime. ice.s3AccessToken is optional.

      ice.s3AccessKeyId=<accessKeyId>
      ice.s3SecretKey=<secretKey>
      ice.s3AccessToken=<accessToken>

If running on a ec2 instance and you want to use the credentials in the instance metadata, you can leave the above properties unset.

1.5 In ice.properties, specify the start time in millis for the processor to start processing billing files. For example, if you want to start processing billing files from April 1, 2013. If this property is not set, Ice will set startmillis to be the beginning of current month.

      ice.startmillis=1364774400000

1.6 Set up s3 billing bucket in ice.properties. If you have multiple payer accounts, you will need to specify multiple values for each property.

      # s3 bucket name where the billing files are. multiple bucket names are delimited by ",". Ice must have read access to billing s3 bucket.
      ice.billing_s3bucketname=billing_s3bucketname1,billing_s3bucketname2
      # prefix of the billing files. multiple prefixes are delimited by ","
      ice.billing_s3bucketprefix=,
      # location for the billing bucket.  It should be specified for buckets using v4 validation
      ice.billing_s3bucketregion=eu-west-1,eu-central-1
      # specify your payer account id here if across-accounts IAM role access is used. multiple account ids are delimited by ",". "ice.billing_payerAccountId=,222222222222" means assumed role access is only used for the second bucket.
      ice.billing_payerAccountId=,123456789012
      # specify the assumed role name here if you use IAM role access to read from billing s3 bucket. multiple role names are delimited by ",". "ice.billing_accessRoleName=,ice" means assumed role access is only used for the second bucket.
      ice.billing_accessRoleName=,ice
      # specify external id here if it is used. multiple external ids are delimited by ",". if you don't use external id, you can leave this property unset.
      ice.billing_accessExternalId=

Tip: If you have multiple payer accounts, or Ice is running from a different account of the s3 billing bucket, for example Ice is running in account test, while the billing files are written to bucket in account prod, account test does not have read access to those billing files because Amazon created them. In this case, the recommended way is to use cross-accounts IAM roles. E.g. you can create an assumed role ice. In prod account, grant assumed role ice with read access to billing bucket, then specify ice.billing_accessRoleName=ice. You can also create a secondary s3 bucket in account prod and grant read access to account test, and then create a billing file poller running in account prod to copy billing files to the secondary bucket.

1.7 Specify account id and account name mappings in ice.properties. This is for readability purposes. For example:

      ice.account.account1=123456789011
      ice.account.account2=123456789012
      ice.account.account3=123456789013

1.8 If you have reserved ec2 instances, please also make sure Ice has the permission to make ec2 call describeReservedInstancesOfferings, which is used to get ri prices.

  1. Reader configuration

2.1 Enable reader in ice.properties:

      ice.reader=true

2.2 In ice.properties, set up the local directory where the reader will copy files to. The local directory must pre-exist. For example:

      ice.reader.localDir=/mnt/ice_reader

Make sure the local directory is different if you run processor and reader on the same instance.

2.3 Same as 1.3

2.4 Same as 1.4

2.5 Same as 1.5

2.6 Specify your organization name in ice.properties. This will show up in the UI header.

      ice.companyName=Your Company Name

2.7 You can choose to show cost in currency other than $. To enable other currency, specify the following properties in ice.properties:

      # Specify your currency sign here. The default value is $. For other currency symbols, you can use UTF-8 code, e.g. for , you can use ice.currencySign=\u00A5
      ice.currencySign=
      # Specify your currency conversion rate here. The default value is 1. If 1 pound = 1.5 dollar, then the rate is 0.6666667.
      ice.currencyRate=0.6666667

2.8 By default, Ice pulls in Highstock from its CDN.

      ice.highstockUrl=https://example.com/js/highstock.js
  1. Running Ice

After the processor and reader setup, you can choose to run the processor and reader on the same or different instances. Running on different instances is recommended. For the first time, you should FIRST RUN PROCESSOR. Make sure you see non-empty output files in your working s3 bucket. Then run reader and browse to http://localhost:8080/ice/dashboard/summary.

Here are the steps of getting ice running locally:

3.1 Pull the project

3.2 Run grails wrapper from the project root directory. This step will pull all necessary jar from maven central.

3.3 Construct ice.properties for processor and make sure ice.properties is added to directory src/java

3.4 Run Ice processor. From project root directory, run ./grailsw run-app. Note you may need to add system properties like ./grailsw -Dice.s3AccessKeyId=<s3AccessKeyId> -Dice.s3SecretKey=<s3SecretKey> run-app. To verify Ice processor runs successfully, make sure you see un-empty output files in your working S3 bucket, e.g. tagdb_all, cost_weekly_all, cost_daily_all_2013, etc.

3.5 Repeat steps 3.3 and 3.4 to run Ice reader.

Tip: Sometimes you want to re-start from a clean slate. To do that:

a) Get the latest code

b) Delete all files from your working s3 bucket under the working prefix

c) Delete all files from your local ice directory (for processor and reader)

d) Start Ice in processor mode. Make sure it runs correctly.

e) Then start Ice in reader mode.

Ice Cookbook:

  1. A community cookbook is available for deploying Ice via Chef here https://github.com/mdsol/ice_cookbook.

Ice Docker Image:

  1. A community image is available for deploying Ice via Docker here https://github.com/jonbrouse/docker-ice

Advanced options:

Options with * require writing your own code.

  1. Basic reservation service

If you have reserved instances in your accounts, you may want to make use of the reservation view in the UI, where you can browse/analyze your on-demand, unused reserved instance usage & cost of different instance types in different regions, zones and accounts. in Bootstrap.groovy, BasicReservationService is used. You can specify reservation period and reservation utilization type in ice.properties:

      # reservation period, possible values are oneyear, threeyear
      ice.reservationPeriod=threeyear
      # reservation utilization, possible values are LIGHT, HEAVY
      ice.reservationUtilization=HEAVY
  1. Reservation capacity poller

To use BasicReservationService, you should also run reservation capacity poller, which will call ec2 API (describeReservedInstances) to poll reservation capacities for each reservation owner account defined in ice.properties. The reservation capacities history is stored in a file in s3 bucket. To run reservation capacity poller, following steps below:

2.1 Set ice.reservationCapacityPoller=true in ice.properties

2.2 Make sure you set up reservation owner accounts in ice.properties. For example:

      ice.owneraccount.account1=
      ice.owneraccount.account2=account3,account4
      ice.owneraccount.account5=account6

2.3 If you need to poll reservation capacity of different accounts, set up IAM roles. Then specify the assumed roles and external ids in ice.properties. For example, if assumed role "ice" is used:

      ice.owneraccount.account1.role=ice
      ice.owneraccount.account2.role=ice
      ice.owneraccount.account5.role=ice

    If you use external id too, specify it like following:

      ice.owneraccount.account1.externalId=
  1. On-demand instance cost alert

You can set set an on-demand instance cost threshold so that alert emails will be sent through Amazon SES if the threshold is reached within last 24 hours. The alert emails will be sent no more than once a day. The following properties are needed in ice.properties:

      # url prefix, e.g. http://ice.netflix.com/. This is used to create the link in alert email
      ice.urlPrefix=
      # from email address, this email must be registered in ses.
      ice.fromEmail=
      # ec2 ondemand hourly cost threshold to send alert email. The alert email will be sent at most once per day.
      ice.ondemandCostAlertThreshold=250
      # ec2 ondemand hourly cost alert emails, separated by ","
      ice.ondemandCostAlertEmails=
  1. Sharing reserved instances among accounts (*)

All linked accounts under the same payer account can share each other's reservations (see http://docs.aws.amazon.com/awsaccountbilling/latest/about/AboutConsolidatedBilling.html).

If reserved instances are shared among accounts, please specify them in ice.properties. For example:

      # set reservation owner accounts. In the example below, account1, account2, account3 and account4 are linked under the same payer account. account5, account6 are linked under the same payer account.
      # if reservation capacity poller is enabled, the poller will try to poll reservation capacity through ec2 API (describeReservedInstances) for each reservation owner account.
      ice.owneraccount.account1_name=account2_name,account3_name,account4_name
      ice.owneraccount.account2_name=account1_name,account3_name,account4_name
      ice.owneraccount.account5_name=account6_name

If different accounts have different AZ mappings, you will also need to subclass BasicAccountService and override method getAccountMappedZone to provide correct AZ mapping.

  1. Customized reservation service (*)

Reserved instance prices in BasicReservationService are copied from Amazon's ec2 price page as of Jun 1, 2013. Your accounts may have different reservation prices (e.g. Amazon may change prices in the future). In this case, you need to write a subclass of BasicReservationService to provide the correct pricing.

  1. Resource service (*)

To use the breakdown feature and application group feature, first make sure you signed up the beta version of detailed billing file with resources and tag. Verify you receive monthly billing file in the this format: <accountid>-aws-billing-detailed-line-items-with-resources-and-tags-<year>-<month>.csv.zip. Then you will need to subclass abstract class ResourceService and have your own bootstrap.groovy create ProcessorConfig and ReaderConfig. See SampleMapDbResourceService for a sample of subclass.

If your custom tags have limited number of value combinations (e.g. < 100), you can choose to set the following parameter in ice.properties, and Ice will generate resource group values for each line item in the billing file. Please be VERY careful about using this feature. Resource group values are generated by concatenating values of of all custom tags. If it results in a long list of resource group values, Ice performance will be greatly affected. Please make sure the custom tags exist in the header of the billing file.

      # specify your custom tags here. Multiple tags are delimited by ",". If specified, BasicResourceService will be used to generate resource groups for you.
      # PLEASE MAKE SURE you have limited number (e.g. < 100) of unique value combinations from your custom tags, otherwise Ice performance will be greatly affected.
      ice.customTags=tag1,tag2

You will need to ensure that any tag you wish to use in ICE is ticked in the Manage Cost allocation report page here: https://portal.aws.amazon.com/gp/aws/developer/account?ie=UTF8&action=cost-allocation-report

Any tag that you have created yourself (as opposed to being automatically generated by AWS) will require you to use the ice.customTags= parameter in the following way. See this example:

ice.customTags=user:CostCenter,User:Environment

  1. Weekly cost email per application group (*)

If you have resource service enabled, you can use BasicWeeklyCostEmailService to send weekly cost emails. You can use the default BasicS3ApplicationGroupService, or you can have your own ApplicationGroupService implementation.

  1. Throughput metric service (*)

You may also want to show your organization's throughput metric alongside usage and cost. You can choose to implement interface ThroughputMetricService, or you can simply use the existing BasicThroughputMetricService. Using BasicThroughputMetricService requires the throughput metric data to be stores monthly in files with names like _2013_04, _2013_05. Data in files should be delimited by new lines. is specified when you create BasicThroughputMetricService instance.

  1. Blended Costs By default, unblended costs are shown. You can show Blended costs with the following configuration:

    ice.use_blended=true
    
  2. Extra Grails configuration file

If you need to setup custom Grails settings, you can specify an additional configuration file to be loaded by Grails by setting the ice.config.location system property to the location of that file.

See http://docs.grails.org/2.4.4/guide/single.html#configExternalized for more information.

Example IAM Permissions

Grant the following permissions to either an instance role, or the user running the reports:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1421551747000",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeReservedInstances",
        "ec2:DescribeReservedInstancesOfferings"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Sid": "Stmt1418665415000",
      "Effect": "Allow",
      "Action": [
        "s3:DeleteObject",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListAllMyBuckets",
        "s3:ListBucket",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::work-bucket-name/*"
      ]
    },
    {
      "Sid": "Stmt1418665415001",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::work-bucket-name"
      ]
    },
    {
      "Sid": "Stmt1418665415000a",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListAllMyBuckets",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::billing-reports-bucket/*"
      ]
    },
    {
      "Sid": "Stmt1418665415001a",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::billing-reports-bucket"
      ]
    }
  ]
}

Support

Please use the Ice Google Group for general questions and discussion.

Download Snapshot Builds

Download snapshot builds here: https://netflixoss.ci.cloudbees.com/job/ice-master/

License

Copyright 2014 Netflix, Inc.

Licensed under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

ice open issues Ask a question     (View All Issues)
  • about 3 years Fatal error during compilation org.apache.tools.ant.BuildException
  • about 3 years Error when I open the dashboard page
  • about 3 years ICE crashes due to missing regions and availability zones.
  • about 3 years install.sh fails with dependencies error.
  • about 3 years Hours accounted always return 24 (hours), it used to work fine a week or more ago.
  • over 3 years Empty dashboard after running app
  • over 3 years Cannot get property 'companyName' on null object
  • over 3 years By default Ice fails to use S3 on eu-central-1
  • over 3 years ice/dashboard/summary shows spinning wheel
  • over 3 years how do i run ice reader
  • over 3 years getData: parseDateTime returns Invalid format: ""
  • over 3 years Items without "region" cloumn are being ignored, so total bill amount is showing less than actual
  • over 3 years Set custom polling interval instead of default 5 minutes
  • over 3 years Faias to start with error: ERROR basic.BasicReservationService - failed to poll reservation prices
  • over 3 years Unable to set the web application class loader property
  • over 3 years Trouble during launch docker-compose - basic.BasicReservationService
  • over 3 years Error when issueing docker-compose up.
  • over 3 years dockerice_ice_1 exited with code 0
  • over 3 years Faias to start with error: ERROR basic.BasicReservationService - No Zone for ap-southeast-2c
  • almost 4 years How to run ice on different port ?
  • almost 4 years failed to get RI price for region and instance type
  • about 4 years Memory leak error
  • about 4 years OnDemandInstance shown as ReservedInstanceHeavy when Custom Tags enabled
  • about 4 years Running ice in apache tomcat - can't find ice.properties
  • over 4 years Documentation clarification - Support for Custom Tags
  • over 4 years NPE in Dashboard.groovy when clicking Link in Weekly Cost Email
  • over 4 years Weekly Cost Email BCC not sent
  • over 4 years Consistently failing to get data after 6/1
  • over 4 years Key does not exist error when getLastMillis called
  • over 4 years NPE stops processor BillingFileProcessor thread
ice open pull requests (View All Pulls)
  • update install script to correct version of grails
  • Added region property for billing buckets to reference properly if v4…
  • Highstock CDN supports HTTPS now, let's use it by default
  • Grouping resources with missing resourceId
  • Spelling Correction from Dawson's Spelling Bee
  • Grammar, clarification in docs
  • Parse services with '/'. Fixes #100
  • Upgrade to Grails 2.4.4 at install.sh
  • Fixed typo in BasicReservationService
  • Feature/login framework
  • Feature/cumulative graph
  • Upgrade dependencies
  • Add Support for Credits/Refunds
  • Issue #148: Changed compile to build in BuildConfig.groovy
  • Allow either file-based or command-line AWS credentials
  • Some smaller improvements on error logs/exceptions, ignores, file-checks, ...
  • add code for addressing services with '/'
  • Provide complete IAM policy document.
  • Added commons-io dependency
  • Add ResourceGroup to /summary
  • Ability to replace "Application Groups" and "ResourceGroups" with user defined terms in the UI
  • Issue 11 fix
  • added initial .gitignore
  • avoid NullPointerException for invalid AZ
  • Fix RuntimeException when an AZ is missing
  • adding missing zone for southeast asia (1c & 2c)
  • minor fixes to install.sh
  • Pin highcharts version. Closes #210
  • Fix typo
  • Update aws-sdk to latest version
  • Adding ap-southeast-2c
  • Add missing regions and zones
  • changes to support running on amazon ecs
  • README: typo
  • Changed page title to read "AWS" instead of "Aws"
  • Update AWS Client
  • Change auth mechanism to support the standards.
  • Add region poperty for working bucket
ice questions on Stackoverflow (View All Questions)
  • SMS_RECEIVED not working on Ice Cream Sandwich?
  • I couldn't establish ice connection between two clients in same LAN using srvflx candidate
  • WebRTC ICE Failed outside LAN
  • WiX: Mysterious and hard-to-diagnose ICE validation errors on build server build
  • WebRTC p2p connection without ICE servers
  • WebRTC ICE Connection fails with recvonly
  • Kernel Sources for Android 4.0 - Ice Cream Sandwich
  • One way Streaming - ICE Connection fails before adding candidates
  • How do I hide address bar on ice cream sandwich
  • Materialistic design in android ice cream sandwich
  • PHONE_REGISTRATION_ERROR C2DM on Ice cream sandwich AVD
  • Big picture notification in ice cream sandwitch
  • Ice Cream Sandwich Style Tabs
  • Ruby: Ice cream with decorators
  • "Ice Cream Sandwich" and WRITE_APN_SETTINGS
  • InvalidSessionDescriptionError: Invalid description, no ice-ufrag attribute
  • Framework to test STUN/TURN/ICE protocols?
  • WebView does not render local HTML page on app launch in Android 4.0.x Ice Cream Sandwich
  • STUN, TURN and ICE
  • how do I resolve docker issues with ice login?
  • Getting timeout issue for ice candidate collection
  • ICE Default IO error in Spyder Ubuntu
  • Autosys command to identify jobs which are ON ICE | ON HOLD
  • ICE module query
  • Unix Script for finding Jobs on ICE
  • Ice namespace error
  • Unable to open serial port ttyUSB under Android Ice Cream Sandwich 4.0.3
  • Creating a container that runs IBM Containers Extension (ICE)
  • Ice Cream Sandwich(Api14 -4.0.1) users reporting StackOverflowError
  • The ice --cloud ps command does not show the Bluemix container in the Building state
ice list of languages used
ice latest release notes
v1.1.2 Release 1.1.2

Minor update that solves known problems when parsing the Products and creating the ResourceGroups. It also adds support for billing files with a localized name (India, China, ...) and fixed the install.sh script.

Changes:

  • Add the support for localized billing files (AWS India, AWS China, ...) (#260)
  • Add the support for Grails external configuration file (ice.config.location) (#259)
  • Add us-east-1f in the zone list (#255)
  • Use HTTPS endpoint for Highstock (#191)
  • Fix the problems with product with / in their names (AWS Import/Export Snowball, ...) (#184)
  • Fix the resource grouping for resource without ResourceId (#187)
  • Fix the install.sh script (Grails v2.4.4 + AWS AMI Linux compatibility) (#182, #200)
v1.1.1 Release 1.1.1

Minor updates to support more regions and zones (us-gov-west-1 + eu-central-1c and ap-northeast-2c). We have also set up Travis CI.

Changes:

  • Added missing zones (eu-central-1c, ap-northeast-2c)
  • Added support for us-gov-west-1 billing data (remark: does not poll reservation capacity and reservation prices for us-gov-west-1 accounts)
  • Configured Travis CI on master and integration branches

Merged PR:

  • ap-northeast-2c and support for us-gov-west-1: #250
v1.1 Release 1.1

Bringing back life into Ice, step 1 :-)

Several PRs have been merged and changes made in order to make Ice:

  • Usable in any AWS region
  • Able to process billing data related to the new regions and zones

We've also automated the integration with https://github.com/jonbrouse/docker-ice in order to create a new docker image on every new Ice release (thanks to @jonbrouse for his quick reaction).

Changes

  • Added missing regions and zones
  • Support for auth v4 regions (eu-central-1) with the ice.billing_s3bucketregion property
  • Updated AWS SDK to 1.11.136
  • Include an example of an IAM policy in the README
  • Capitalize the Aws abbreviation in the page titles

Merged PR:

  • New AWS regions support: #228 #225 #220 #193
  • Typos: #242 #221 #241 #179
  • Dependencies updates: #122 #220 #211
  • README improvments: #135
  • Others: #3
Other projects in Java