Contacts Us

Fill Out the Form below to learn more!

on 12 June 2020
  • AWS
  • QuickSight
  • EC2 SPOT Instances

Introduction


Lot of organizations use EC2 Spot instances to run their workload to save cost. Though Spot instances saves cost it is often difficult to quantify the savings. Its even more difficult when there are hundreds and thousands of instances running across many AWS child accounts and multiple regions. Spot historical report tries to solve this problem. You can have very rich information on the savings achieved in less than 20 min.


High-Level Design


  • The solution utilizes the AWS Cost and Usage Report (CUR). Spot instances billing-related information are picked up from this report.
  • Solution creates all required services to capture and store the pricing & cost data in the master and child accounts. The list is provided in subsequent sections.
  • It uses the AWS Pricing Bulk API to extract the current On-Demand and Reserved Instances pricing information.
  • The data collected is stored in the Parent account for reporting.
  • AWS Quicksight dashboard lets the user visualize the savings achieved. It also allows the user to create custom dashboards using the data captured.

High-Level Design Diagram

Figure 1: High-Level Architecture of EC2 SPOT Historical Reporting Solution


Services Required in AWS


Below is the list of services the solution would create in the parent and child AWS accounts.

  • Cloud Watch
  • Cloud Trail
  • SNS (Simple Notification Service)
  • SQS (Simple Queue Service)
  • Lambda
  • S3 (Simple Storage Service)
  • Athena
  • QuickSight – Premium Account

Cost and Usage Report


The AWS Cost & Usage Report plays a vital role in our solution. This lists AWS usage for each service category used by an account and its IAM users in hourly or daily line items, as well as any tags that has been activated for cost allocation purposes. It can also be customized to aggregate usage data to the daily or hourly level. The AWS Cost & Usage Report need to enabled in the parent/master account as perquisite for our solution to work. Please refer the link to enable AWS Cost and Usage Report


Billing data will be dropped into the bucket that was configured while enabling Cost & Usage Report in 24 hours.


Cost and Usage Reports created in master accounts

Figure 2: Cost and Usage Reports created in master account


Cost and Usage Report data loaded in S3 Bucket

Figure 3: Cost and Usage Report data loaded in S3 Bucket


Deployment


We use CloudFormation stacks to create the required service in the parent and child accounts.


Deployment steps for Parent Account


Follow the below steps to create the CloudFormation stacks in the parent account using the CloudFormation template.


Step 1: Launch the CloudFormation Stack by clicking the below button and chooose the region where you want to create the stack (CUR should be enabled in the chosen region).

Launch Stack




Step 2: Name your stack (Ex: EC2-REPORTING-PARENT-ACC-STACK).

Step 3: Fill the below 4 parameters and click on next

  1. ChildAccountId

  Description: The Child Account Id of the current parent account, this id will be used for creating a cross-account role. The cross-account role is used to access the parent account Athena service from the child account. If you want to create the cross-account role for multiple account ids, provide the account ids as comma-separated values.

  2. License Key

  Description: The Product Licence Key which you have received while SPOT Subscription process.

  3. S3BucketNameForFileUploadsAndAthenaData

  Description: The name of the S3 bucket where Athena’s source input and output files will be kept. The user should give a new bucket name for this parameter.

  4. S3BucketNameOfCURBillingReport

  Description: The name of the Cost and Usage Report bucket which the user has given while creating the cost and usage report in Billing Console.

Specify Stack Details

Step 4: In Configure Stack Option, Fill the Tags If you want to add.

Stack Configuration

Step 5: Click on Next and Review your stack creation.

Step 6: Acknowledge the capabilities and click on Create Stack.

Stack Review and Acknowledge

It will take around 4 minutes to complete the stack creation.

List of Parent and Child stacks created in Parent Account

Once the stacks are created in the parent account the below services will be created in the parent account using the child stacks.

List of services created by the Child Stacks
S.No. Child Stack Name AWS Service Name Resource(s) Detail Usage
1 GENERATEATHENADATABASECDK Athena Database: aws-ec2-rpt, Tables: (cur_raw_data, ec2_instance_tag_detail, ec2_network_detail, ec2_pricing_detail, spot_instance_detail, spot_price_history) To store the CUR Processed data, EC2 On-Demand and Reserved Instances pricing detail, EC2 Spot Instances Pricing detail, EC2 Instances interruption, tags and network-related detail
Amazon S3 A new bucket will be creation using the S3BucketNameForFileUploadsAndAthenaData value. This bucket is used for storing the Atheana datasource
2 EC2COSTANDUSAGEDATAPROCESSORCDK CloudTrail Trail CUR_NOTIFICATION_CLOUDTRAIL To process the Cost and Usage Report Bucket events
CloudWatch Event Rule CUR_NOTIFICATION_CLOUDWATCH This rule is used to receives the notification from the above trail and triggers an notification to CUR_NOTIFICATION_SNS
SNS Topic CUR_NOTIFICATION_SNS This topic is used to receive S3 notification from CloudWatch Events
SQS Queue CUR_NOTIFICATION_SQS This queue is used to receive message from the CUR_NOTIFICATION_SNS Topic. It will trigger the cost and usage data processor lambda function
SQS Queue CUR_NOTIFICATION_OD_RI_SQS This queue is used to receive message from the CUR_NOTIFICATION_SNS Topic. It will trigger the EC2 pricing downloader lambda function
Lambda Function EC2_COST_AND_USAGE_DATA_PROCESSOR_FUNCTION To process the Cost and Usage Report un processed data, and it additionally refresh the quicksight dashboard's dataset
Athena Views: aws_billing_detail, ec2_on_demand_reserved_pricing_detail, ec2_spot_pricing_detail, spot_instance_network_detail, spot_instance_tag_detail, od_spot_prod_map, ec2_cost_usage_summary, ec2_instance_comparision This views are used for final report preparation
3 EC2ONDMDRESVDPRICEDOWNLOADCDK Lambda Function EC2_ONDMD_RESVD_PRICE_DOWNLOAD_FUNCTION This function is used to process the EC2 On-Demand and Reserved Instances pricing files
4 EC2SPOTPRICEDOWNLOADCDK Lambda Function EC2_SPOT_PRICE_DOWNLOAD_FUNCTION This function is used to download and process the EC2 SPOT Instances latest pricing details
CloudWatch Event Rule SPOT_PRICE_DOWNLOAD_CLOUDWATCH This rule is used to receives the notification from the above trail and triggers an notification to CUR_NOTIFICATION_SNS
5 PARENTSTACKINITIALIZATIONCDK Lambda Function PARENT_STACK_INITIALIZATION_FUNCTION This function is used for creating the cross-account roles for the child account


Watch our parent stack deployment video in YouTube...






Deployment steps for Child Account


Follow the below steps to create the CloudFormation stacks in the child account using the CloudFormation templates.


Step 1: Launch the CloudFormation Stack by clicking the below button and choose the region where your parent account stack is deployed.

Launch Stack




Step 2: Name your stack (Ex: EC2-REPORTING-CHILD-ACC-STACK).

Step 3: Fill the below 4 parameters and click on next.

  1. AthenaRegion

  Description: The Athena database region where the Athena Database is created.

  2. License Key

  Description: The Product License Key which the customer have received while subscription

  3. parentAccountID

  Description: The parent account Id where the Athena database is created.

  4. S3DataBucket

  Description: The name of the S3 bucket where Athena’s source input and output files will be kept.(Give the same bucket name which you have given in the Parent Stack creation)


Specify Stack Details

Step 4: In Configure Stack Option, Fill the Tags If you want to add.


Configure Stack

Step 5: Click on Next and Review your stack creation.

Step 6: Acknowledge the capabilities and click on Create Stack.


Stack Review and Acknowledge

It will take around 5 minutes to complete the stack creation.


Parent and Child Stacks created in Child Account

Once the stacks are created in the child account the below services will be created in the child account using the child stacks.

List of services created by the Child Stacks
S.No. Child Stack Name AWS Service Name Resource(s) Detail Usage
1 EC2NOTIFICATIONCDK CloudWatch Event Rule EC2_NOTIFICATION_CLOUDWATCH This rule is used to capture the SPOT Instances interruption detail and trigger the notifications to the EC2_NOTIFICATION_SNS Topic
SNS Topic EC2_NOTIFICATION_SNS It receives EC2 event notification from CloudWatch Event. Whenever it receives the notification it publishes to its subscribed queues
SQS Queue EC2_NOTIFICATION_INTERRUPTION_SQS It receives messages from the EC2_NOTIFICATION_SNS Topic and triggers the SpotInterruptionLambdaCDK lambda function
SQS Queue EC2_NOTIFICATION_INTERRUPTION_SQS It receives messages from the EC2_NOTIFICATION_SNS Topic and triggers the SpotInterruptionLambdaCDK lambda function
SQS Queue EC2_NOTIFICATION_TAGS_SQS It receives messages from the EC2_NOTIFICATION_SNS Topic and triggers the ec2_instance_tags_lambda_func lambda function
2 EC2INTERRUPTIONDETAILCDK Lambda Function EC2_INTERRUPTION_DETAIL_FUNCTION This function is used to process the EC2 Instances interruption details and stores the detail in Athena Tables
3 EC2NETWORKDETAILCDK Lambda Function EC2_NETWORK_DETAIL_FUNCTION This function is used to process the EC2 Instances network details and stores the detail in Athena Table
4 EC2TAGDETAILCDK Lambda Function EC2_TAG_DETAIL_FUNCTION This function is used to process the EC2 Instances tags details and stores the detail in Athena Table
5 CHILDSTACKINITIALIZECDK Lambda Function CHILD_STACK_INITIALIZE_FUNCTION This function is used to validate the customer's Product License Key

Note:

EC2 Instances Networks, Interruptions and Tags related details are stored in Athena for the instances which are created after the CloudFormation Templates creation in the Child account.


Deployment steps for Quicksight


Amazon QuickSight is a fast, cloud-powered business intelligence service that makes it easy to deliver insights to everyone in your organization. As a fully managed service, QuickSight lets you easily create and publish interactive dashboards that include ML Insights. Dashboards can then be accessed from any device, and embedded into your applications, portals, and websites.


Prerequisites


1. .NET Core SDK - for executing the deployment application from your local system.

  Ref:.NET Core SDK Download URL

2. QuickSight should be enabled in customer's master/parent account.


Create reports in the target account


Follow the below steps for deploying the report and viewing in AWS QuickSight.

Step 1: Download our QuickSight Deployment package from the below URL and unzip it.

URL: QuickSight Deployment Package Download URL

Step 2: Open the config.json file in any editor from the bin folder and configure the below information.

tarAccessKey - Your parent account access key
tarSecretAccessKey - Your parent account secret access key
tarAccountID - Your parent account id
tarUserArn - Replace your parent account id the ARN
tarDatasourceName - Give any name for the data source
tarDatasetName - Give any name for the dataset
tarDashboardName - Give any name for the dashboard

Note: For your convenience give the same name for the data source, dataset, and for the dashboard.

Setp 3: Now click on the QuicksightDeployment.exe file for deploying the report.

QuicksightDeployment.exe

It will take couple of minutes to complete the deployment and you will see the response like in the below image.

template.json

Provide S3 bucket access to your Data Set


Follow the below steps to provide S3 bucket(The bucket which contains Athena source data) access to your data set.

Step 1 : Navigate to QuickSight and click on Manage QuickSight under User Menu.

Step 2 : Click on Security & Permissions.

Step 3 : Click on Add or remove which is availbe under QuickSight access to AWS services.

template.json

Step 4 : Click on S3 and select the bucket which contains the Athena source data.

template.json

Step 5 : Click on Finish and Click on Update.

Step 6 : Navigate to Home Page and click on Manage Data.

Step 7 : Select your Data Set and click on Refresh Now.

It will take couple of minutes to complete the refresh.


View the report


Navigate to QuickSight and click on all dashboards.

You can see the dashboard with the name you have given in the config.json file.

template.json

If you click on the dashboard your can see the cost-saving report.

template.json

The user can easily understand the savings calculation in an effective way in the report.

The user can identify the savings by the following ways

  • Savings per Instance type
  • Savings per Month
  • Savings per Account
  • Savings per Region

and the total spot interruptions can be viewed against instances.


Watch our QuickSight Dashboard explanation video in YouTube...



Troubleshooting


During deployment you may face some of the below errors and these solutions will help you to resolve the problems.


Problems in CloudFormation Stack Creation


1. Error in Amazon S3 Template URL

TemplateURL must be an Amazon S3 URL.

Solution

  • Replace region_name with your current region code in the template URL.

2. Error in Stack Creation

Stack Creation Failed

Solution

  • Make sure that the required permissions are added to the IAM user.

3. Error in the resources created using CloudFormation Stacks

Solution

  • Please check the AWS CloudWatch Logs.

Problems in Report Deployment


1. Error in Source Account Id

Amazon.QuickSight.Model.AccessDeniedException: User: arn:aws:iam::src_account_id:user/admin is not authorized to perform: quicksight:DescribeDashboard on resource: arn:aws:quicksight:ap-southeast-2:src_account_id:dashboard/dasboard_id
                                        

Solution

  • Make sure that you have successfully configured our source account id in the config.json file.

2. Error in Dashboard Id

Amazon.QuickSight.Model.ResourceNotFoundException: Dashboard arn:aws:quicksight:ap-southeast-2:src_account_id:dashboard/dashboard_id is not found
                                        

Solution

  • Make sure that the dasboard id is correct.

3. Error in Dataset Id

Amazon.QuickSight.Model.ResourceNotFoundException: DataSet arn:aws:quicksight:ap-southeast-2:src_account_id:dataset/dataset_it is not found
                                        

Solution

  • Make sure that the dataset id is correct.

4. Error in Datasource Id

Amazon.QuickSight.Model.ResourceNotFoundException: DataSource arn:aws:quicksight:ap-southeast-2:682853212408:datasource/datasource_id is not found
                                        

Solution

  • Make sure that the datasource id is correct.

Problems in QuickSight


1. Error in Dataset

Import failed: This is a general SQL error. This can be caused by query timeouts, resource constraints, unexpected DDL alterations before or during a query, and other database errors. Check your database settings and your query, and try again. 
sourceErrorMessage = Amazon S3; Status Code: 403; Error Code: AccessDenied;

Solution

  • Click on Account / User Icon
  • Click on Manage QuickSight
  • You will be redirected to Admin page
  • Click on Security and Permissions from the left side menu
  • In QuickSight Access to AWS Services, Click on Add or remove
  • Click on AmazonS3, Select S3 Buckets
  • Select the bucket where your datasource is available and click on update
  • Next refresh your dataset. It will take couple of minutes to refresh the data

Please contact us for further assistance aws@bi3technologies.com