EC2 inventory detail calculations

Inventory Details for EC2 in Clumio Discover provides insight into utilized sizes, change rates, and growth rates of EC2 instances in AWS accounts, based on EBS snapshots of the attached EBS volumes. All calculations are based on EBS snapshots that meet the requirements explained in the following sections. 

Utilized size of EC2 instance

EBS volumes attached to EC2 instances provide durable block-level storage for the instances. AWS offers different options for EBS volume sizes for specifying the storage options instances. For example, a General Purpose SSD (gp2) volume can have a volume size between 1GiB to 16TiB. If you spin up a volume of size 1GiB, then this volume size is called the provisioned size of the EBS volume. The utilized size of the EBS volume is the size of the data that is stored in this provisioned volume. For example, if you write 512MiB of data, then the utilized size of the EBS volume is 512MiB. 

The following sections explain how Clumio calculates the utilized sizes of EC2.

EBS snapshots based calculations

EBS snapshots are a point-in-time copy of the data, and each snapshot contains all of the information that is needed to restore the data to a new EBS volume - which means that we can infer the EBS volume’s utilized size based on its snapshot size. For an EC2 instance, Clumio looks for the most recent snapshot of each attached EBS volume in the last 90 days and this snapshot’s size is treated as the utilized size of the EBS volume.

How Clumio calculates the utilized size of an EC2 instance:

For example, assume that an EC2 instance has two attached EBS volumes that are provisioned for 4 blocks each.

Attached volume 1:

Attached volume 2:

• For both attached volumes, snapshot A, taken on July 15, 2022, is seeding backups which store the data on the blocks of volumes.
• For subsequent snapshots taken on the following days, some blocks were overwritten and other blocks were newly written to.
• AWS snapshots are incremental backups, which means that only the blocks on the volume that have changed after the most recent snapshot are stored. This is indicated in the table by empty cells for the snapshots, where empty cells are referencing the previous snapshot storing the data.
• If we calculate the utilized size of the EC2 instance on Aug 01, 2022, then Clumio looks for the most recent snapshot in the last 90 days for both volumes- which in this case is snapshot D, taken on July 18, 2022.
• The size of snapshot D of attached volume 1 includes the bolded letters in the table with data (A2, B2, C1, D1) starting from block 0 through block 3, and the total snapshot size is the sum of these blocks. Since each EBS volume block is 512KiB in size, the snapshot size is 4 blocks x 512KiB = 2MiB. The utilized size of attached volume 1 is 2MiB.
• The size of snapshot D of attached volume 2 includes the bolded letters in the table with data (A3, B3, C1) starting from block 0 through block 2, and the total snapshot size is the sum of these blocks. Since each EBS volume block is 512KiB in size, the snapshot size is 3 blocks x 512KiB = 1.5MiB. The utilized size of attached volume 2 is 1.5MiB.
• By adding the utilized sizes of both attached volumes, the utilized size of the instance is 2MiB + 1.5MiB = 3.5MiB

Summarized utilized size of EC2 instances

For example, assume that you have two EC2 instances which have a number of attached volumes. These volumes have a number of snapshots taken on different dates. 

If today is Aug 01, 2022, Clumio looks back 90 days in time and uses the most recent snapshot’s size as the attached volume’s utilized size. The most recent snapshot is chosen because this snapshot is indicative of the latest utilized size of the volume. To calculate the utilized size of the instance, the sizes of volumes are added up.

 

The summarized utilized size of instances calculated by adding the utilized sizes of individual instances.  

Utilized size of EC2 instance 1: 2 GiB + 2GiB ≅ 4GiB  
  
Utilized size of EC2 instance 2: 2 GiB + 2GiB ≅ 4GiB  
  
Summarized utilized size of EC2 instances: 4 GiB + 4GiB ≅ 8GiB

 

Change vs Growth

Data can be written to blocks storing data or new blocks in an EBS volume. When a block storing data is overwritten with new data, we refer to it as “change”, and when data is written to an entirely new block, we refer to it as “growth”.

Change in EBS volume

The following table shows how change occurs in an EBS volume. For simplicity, consider an EBS volume with 10 blocks with indexes 0 through 9. On Day 1, data was written to 6 blocks starting from block 0 through block 5. On Day 2, block 1’s data B was overwritten with new data B1 and we refer to this as “change”. Similarly, block 3’s data D changed from D to D1 on Day 2. All the changed blocks from Day 1 to Day 3 are highlighted with bolded letters in the cells. 

 

 

Growth in EBS volume

The following table shows how growth occurs in an EBS volume. For simplicity, consider an EBS volume with 10 blocks with indexes 0 through 9. On Day 1, data is written to 5 blocks starting from block 0 through block 4. On Day 2, data is written to new blocks i.e. block 5 and block 6 and we refer to this as “growth”. Similarly, data is written to block 7 and block 8 on Day 3. The cells with bolded letters indicate growth from Day 1 to Day 3 in this EBS volume.

 

Why Change and growth are Important

In the backup world, change and growth are important to understand as both impact your backup storage and the corresponding costs. If a web application writes 10GiB newly to an EBS volume of size 10GiB, it results in a growth of 10GiB and total size of 20GiB. In this case, you can expect your backup storage to be 20GiB. The 10GiB growth will be held onto forever in the backup. On the other hand, if the web application is changing the existing data stored in the 10GiB EBS volume, then your backup storage can vary widely due to the changes. This is because change causes multiple versions of the same block to be stored in the backups, and these versions are retained for the duration specified by your backup policy. Therefore, change and growth in your EBS volumes are important to understand what is contributing to your backup sizes.

To provide insights on how EC2 instances in your AWS accounts are changing and growing based on the change and growth of attached EBS volumes, Clumio calculates daily and monthly change rates and growth rates that can be used to estimate backup sizes if you are taking daily and monthly backups.

Change rate of an EC2 instance

The change rate of an EC2 instance refers to the rate at which the blocks of attached volumes of the instance change over time i.e. existing blocks of data getting overwritten by data for a specific time period. 

Daily change rate of an individual EC2 instance

A daily change rate explains the average rate of change in an EC2 instance on a daily basis. 

How Clumio calculates the average daily change rate of an EC2 instance:

For example, let's look at an EC2 instance that has two attached volumes with snapshots taken on different days.

Assuming that today is Aug 01, 2022, Clumio calculates the average daily change rate of this instance in the following steps.

Step 1:

For the last 3 months i.e. from May 01, 2022 to Aug 01, 2022, identify snapshots that are 20 to 28 hours apart in time for each attached volume. The time difference is required to ensure that the two snapshots are approximately a day apart for calculating the daily change rate. Some snapshots that meet this requirement are:

Attached volume 1:

Attached volume 2:

Note: 

• Each attached volume should have snapshots that meet Clumio’s requirements and should result in 2 to 5 snapshot pairs for Clumio to calculate the change rate successfully. If not, the change rate of the attached volume is shown as “-” in the UI.
• Snapshots that are copied to a secondary region are not used in the calculations.
• If you have multiple snapshots taken on two days, then the latest snapshot of each day will be used to create a snapshot pair to prioritize recency. For example, if you have snapshot A, snapshot B taken in order on Day 1 and snapshot C, snapshot D taken in order on Day 2, then we use snapshot B and snapshot D for calculating the daily change rate.

 Step 2:

An EBS volume can have the same blocks overwritten or new blocks written over time. Snapshots are point-in-time and capture the volume state when they are taken. To account for same blocks being overwritten and new blocks written, Clumio calculates two bounds for each snapshot pair:

1. Same blocks changing: Consider an EBS volume where the same block i.e. block 1 changes every 6 hours. The average change rate for a day would be 25%, implying that 1 out of 4 blocks changed.

 

2. Different blocks changing:Consider the same EBS volume where a different block changes every 6 hours. The average change rate for a day would be 100%, implying that 4 out of 4 blocks changed.

 

As described above, the change rate of the attached EBS volume in question can vary from 25% to 100% depending on whether the same blocks changed or different blocks changed. To account for this, Clumio calculates the two bounds for each snapshot pair as follows.

1. For the first snapshot pair S1 and S2, calculate the utilized sizes of S1 and S2 by making ListSnapshotBlocks API calls for each snapshot.
2. Calculate the delta between S1 and S2 (i.e. the blocks that are different between the two snapshots) by making ListChangedBlocks API calls.
3. Bound 1: We take the delta between the snapshots and subtract the growth between them for fetching changes alone using the formula:

(delta- (snapS2UtilizedSize - snapS1UtilizedSize))/(snapS1UtilizedSize)

For attached volume 1: Utilized sizes of S1 (Snap1Size) and S2 (Snap2Size) are 2GiB and 2.25GiB respectively.

For attached volume 2: Utilized sizes of S5 (Snap1Size) and S6 (Snap2Size) are 2.6GiB and 2.8GiB respectively.

Assuming that the delta between snapshot pairs is 1GiB, bound 1 will be:

Attached volume 1's Bound 1: (1GiB - (2.25GiB - 2GiB)) / 2GiB ≅ 0.375 or 37.5%   
  
Attached volume 2's Bound 1: (1GiB - (2.8GiB - 2.6GiB)) / 2.6GiB ≅ 0.307 or 30.7% 

 4. Bound 2: We divide the previous results by the number of days between the snapshots.

Attached volume 1's Bound 2: 0.375/ 0.83 ≅ 0.451 or 45.1%  
  
Attached volume 2's Bound 2: 0.307/ 1 ≅ 0.307 or 30.7

5. Bound 1 and Bound 2 will be displayed as min. change rate and max. change rate for each volume’s snapshot pair depending on their values. The avg. change rate is calculated by averaging min. and max. change rates.
6. The snapshot pair for both volumes will be displayed as follows.

Attached volume 1:

Attached volume 2:

 

Step 3:

Clumio identifies a number of snapshot pairs for each attached volume and calculates min. change rate, max. change rate and avg. change rate for each pair. Assuming that the delta from the ListChangedBlocks API calls is 1GiB for all snapshot pairs of the EBS volume, the following data is generated.

Attached volume 1:

 

The average change rate of attached volume 1 is calculated by taking an average of avg. change rates of all snapshot pairs. 

Attached volume 1's avg. daily change rate: (41.3+41.82+22.93+18.65+15.12)/5 ≅ 27.96% 

Attached volume 2:

The average change rate of attached volume 2 is calculated by taking an average of avg. change rates of all snapshot pairs. 

Attached volume 2's avg. daily change rate: (30.7+ 10.31+12.19+17.49+ 18.65)/5 ≅ 17.86% 

Lastly, the average daily change rate of the instance is calculated by averaging the change rates of attached volume 1 and attached volume 2.

Avg. daily change rate: (27.96 + 17.86)/2 ≅ 22.91% 

Growth rate of an EC2 instance

The growth rate of an EC2 instance refers to the rate at which the blocks of attached volumes of the instance grow by writing to new blocks over a specific time period. 

Daily growth rate of an individual EC2 instance

A daily growth rate explains the average rate of growth in an EC2 instance on a daily basis. 

How Clumio calculates the average daily growth rate of an EC2 instance

For example, consider an EC2 instance that has two attached volumes with snapshots taken on different days.

Assuming that today is Aug 01, 2022, Clumio calculates the average daily growth rate of this EBS volume in the following steps.

Step 1:

For the last 3 months i.e. from May 01, 2022 to Aug 01, 2022, identify snapshots that are 20 to 28 hours apart in time for each attached volume. The time difference is required to ensure that the two snapshots are approximately a day apart for calculating the daily growth rate. Some snapshots that meet this requirement are:

Attached volume 1:

Attached volume 2:

Note: 

• Each attached volume should have snapshots that meet Clumio’s requirements and should result in 2 to 5 snapshot pairs for Clumio to calculate the change rate successfully. If not, the change rate of the attached volume is shown as “-” in the UI.
• Snapshots that are copied to a secondary region are not used in the calculations.
• If you have multiple snapshots taken on two days, then the latest snapshot of each day will be used to create a snapshot pair, to prioritize recency. For example, if you have snapshot A, snapshot B taken in order on Day 1 and snapshot C, snapshot D taken in order on Day 2, then we use snapshot B and snapshot D for calculating the daily growth rate.

Step 2:

For each snapshot pair of the attached volume, calculate the daily growth by using the following formula.

(snapS2UtilizedSize - snapS1UtilizedSize)/snapS1UtilizedSize/(snapS2TimeInDays-snapS1TimeInDays)

For attached volume 1: Utilized sizes of S1 (Snap1Size) and S2 (Snap2Size) are 2GiB and 2.25GiB respectively and the snapshots are 22 hours apart (0.83 day).

Attached volume 1: (2.25GiB - 2GiB)) / 2GiB / 0.83 ≅ 0.1506 or 15.06%

For attached volume 2: Utilized sizes of S5 (Snap1Size) and S6 (Snap2Size) are 2.6GiB and 2.8GiB respectively and the snapshots are 24 hours apart.

 Attached volume 2:  (2.8GiB - 2.6GiB)) / 2.6GiB/ 1  ≅ 0.0769 or 7.69% 

Attached volume 1:

Attached volume 2:

 

Step 3:

Clumio identifies a number of snapshot pairs for each attached volume and calculates growth rate for each pair that will result in the following table.

Attached volume 1:

The average growth rate of attached volume 1 is calculated by taking an average of the daily growth rates of all snapshot pairs. 

Attached volume 1's avg. daily growth  rate: (15.06+0+22.24+4.83+6.57)/5 ≅ 9.74% 

Attached volume 2:

The average growth rate of attached volume 2 is calculated by taking an average of daily growth rates of all snapshot pairs. 

Attached volume 2's avg. daily growth rate: (7.69+ 23.14+14.7+7.8+ 4.9)/5 ≅ 11.64% 

Lastly, the average daily growth rate of the instance is calculated by averaging the growth rates of the attached volumes.

Avg. daily growth  rate: (9.74 + 11.64)/2 ≅ 10.69% 

 

Aggregated daily change rate and daily growth rate of EC2 instances

In the previous sections we covered how daily change rate and daily growth rate are calculated for an individual EC2 instance. Clumio calculates summarized daily change rates and daily growth rates by using EC2 instances that have non NULL values (NULL values are represented by a “-”) for utilized sizes, change rates and growth rates. 

For example, you have 10 EC2 instances, and Clumio’s calculations for utilized sizes, daily change rates, and daily growth rates of these 10 instances resulted in the following data.

 

 

Summarized utilized size of EC2 instances:

All EC2 instances that have the utilized size calculated successfully are added to calculate the summarized utilized size.

Utilized size: 100+50+16+10+200+30+50 ≅ 456GiB

Summarized daily change rate of EC2 instances:

This is calculated by a weighted average of the utilized sizes of EC2 instances in order to consider an instance’s contribution to the summarized daily change rate based on its utilized size.

A: (100GiB (6.5%)+ 50GiB (4.25%) + 16GiB (5%) + 10GiB(5%) + 200GiB(3.75%) + 30GiB(3.5%) + 50GiB(6.25%))

B: 100GiB + 50GiB + 16GiB + 10GiB + 200GiB + 30GiB + 50GiB ≅ 456GiB

Avg. daily change rate of EC2 instances: A/B ≅ 2160/456 = 4.73%

Summarized daily growth rate of EC2 instances:

This is calculated by a weighted average of the utilized sizes of EC2 instances in order to consider an instance’s contribution to the summarized daily growth rate based on its utilized size.

A: (100GiB (1.5%)+ 50GiB (3.5%) + 16GiB (0%) + 10GiB(4%) + 200GiB(6%) + 30GiB(1.5%) + 50GiB(3.2%))

B: 100GiB + 50GiB + 16GiB + 10GiB + 200GiB + 30GiB + 50GiB ≅ 456GiB

Avg. daily growth rate of EC2 instances: A/B = 1770/4565 ≅ 3.88%  
  

 

Change rate of an EC2 instance

The change rate of an EC2 instance refers to the rate at which the blocks of attached volumes of the instance change over time, i.e., existing blocks of data getting overwritten by data for a specific time period. 

Monthly change rate of an individual EC2 instance

A daily change rate explains the average rate of change in an EC2 instance over a month. 

How Clumio calculates the average monthly change rate of an EC2 instance:

For example, consider an EC2 instance that has two attached volumes with snapshots taken on different days.

Assuming that today is Aug 01, 2022, Clumio calculates the average monthly change rate of this EC2 instance in the following steps.

Step 1:

For the last year, i.e., from Aug 01, 2021 to Aug 01, 2022, identify snapshots that are 20 to 40 days apart in time for each attached volume. The time difference is necessary to ensure that the two snapshots are approximately a month apart for calculating the monthly change rate. Some snapshots that meet this requirement are:

Attached volume 1:

Attached volume 2:

Note:

• Each attached volume should have snapshots that meet Clumio’s requirements and should result in 2 to 5 snapshot pairs for Clumio to calculate the change rate successfully. If not, the change rate of the attached volume is shown as “-” in the UI.
• Snapshots that are copied to a secondary region are not used in the calculations.
• If you have multiple snapshots taken in two months, then the latest snapshot of each month will be used to create a snapshot pair to prioritize recency. For example, if you have two snapshots in January: snapshot A taken on Jan 10th, snapshot B taken on Jan 28th, and two snapshots in February: snapshot C taken on Feb 15th, snapshot D taken on Feb 25th, then we use snapshot B and snapshot D for calculating the monthly change rate.

Step 2:

As explained in the section of daily change rate, an attached volume can have the same blocks overwritten or new blocks written over time. To account for this, Clumio calculates two bounds for each snapshot pair. 

Assuming that the delta between snapshot pairs is 1GiB, bound 1 will be:

Attached volume 1 Bound 1: (1GiB - (2.5GiB - 2GiB))/2GiB = 0.25 or 25%

Attached volume 2 Bound 1: (1GiB - (6.1GiB - 5.75GiB))/5.75GiB = 0.113 or 11.3%

For bound 2, we divide the previous results by the number of months between the snapshots.

  Attached volume 1's Bound 2: 0.25/ 1.1 ≅ 0.227 or 22.7%

  Attached volume 2's Bound 2: 0.113/ 1.06 ≅ 0.106 or 10.6%

The two bounds of a snapshot pair will be shown as min. change rate and max. change rate. The avg. change rate is calculated by averaging min. and max. change rates.

Attached volume 1:

Attached volume 2:

Step 3:

Clumio identifies a number of snapshot pairs for each attached volume and calculates min. change rate, max. change rate, and avg. change rate for each pair. Assuming that the delta from the ListChangedBlocks API calls is 1GiB for all snapshot pairs of the EBS volume, the following data is generated.

Attached volume 1:

The average change rate of attached volume 1 is calculated by taking an average of avg. change rates of all snapshot pairs. 

Avg. monthly change rate: (23.5+27.55+30.02+22+13.4)/5 ≅ 23.29% 

Attached volume 2:

The average change rate of attached volume 2 is calculated by taking an average of avg. change rates of all snapshot pairs. 

Avg. monthly change rate: (10.95+13.5+11.88+8.96+6.54)/5 ≅  10.37% 

Lastly, the average monthly change rate of the instance is calculated by averaging the change rates of attached volume 1 and attached volume 2.

Avg. monthly change rate: (23.29 + 10.37)/2 ≅ 16.83% 

Growth rate of an EC2 instance

The growth rate of an EC2 instance refers to the rate at which the blocks of attached volumes of the instance grow by writing to new blocks over a specific time period. 

Monthly growth rate of an individual EC2 instance

A monthly growth rate explains the average rate of growth in an EC2 instance over a month. 

How Clumio calculates the average monthly growth rate of an EC2 instance

For example, consider an EC2 instance that has two attached volumes with snapshots taken on different days.

Assuming that today is Aug 01, 2022, Clumio calculates the average monthly growth rate of this EBS volume in the following steps.

Step 1:

For the last year, i.e., from Aug 01, 2021 to Aug 01, 2022, identify snapshots that are 20 to 40 days apart in time for each attached volume. The time difference is required to ensure that the two snapshots are approximately a month apart for calculating the monthly growth rate. Some snapshots that meet this requirement are:

Attached volume 1:

Attached volume 2:

 

Step 2:

For each snapshot pair of the attached volume, calculate the monthly growth by using the following formula.

(snapS2UtilizedSize - snapS1UtilizedSize)/snapS1UtilizedSize/(snapS2TimeInMonths-snapS1TimeInMonths)

For attached volume 1: Utilized sizes of S1 (Snap1Size) and S2 (Snap2Size) are 2GiB and 2.5GiB respectively and the snapshots are 793 hours apart (1.1 months apart).

Attached volume 1: (2.5GiB - 2GiB)) / 2GiB / 1.1 ≅ 0.2272 or 22.72%% 

For attached volume 2: Utilized sizes of S3 (Snap1Size) and S4 (Snap2Size) are 5.75GiB and 6.1GiB respectively and the snapshots are 859.5 hours apart (1.19 months apart).

Attached volume 2:  (6.1GiB - 5.75GiB)) / 5.75GiB/ 1 .19 ≅ 0.0511 or 5.11 % 

The snapshot pair will be displayed as follows.

Attached volume 1:

Attached volume 2:

 

Step 3:

Clumio identifies a number of snapshot pairs for each attached volume and calculates growth rate for each pair that will result in the following table.

Attached volume 1:

The average growth rate of attached volume 1 is calculated by taking an average of the monthly growth rates of all snapshot pairs. 

Attached volume 1's avg. monthly growth  rate: (22.72+ 31.32+7.32+3.48+17.7)/5 ≅  16.5% 

Attached volume 2:

The average growth rate of attached volume 2 is calculated by taking an average of the monthly growth rates of all snapshot pairs. 

Attached volume 2's avg. monthly growth  rate: (5.11+ 2.26+0+1.98+9.3)/5 ≅  3.73% 

The average monthly growth rate of the instance is calculated by taking an average of growth rates of attached volumes. 

Avg. monthly growth rate: (16.5%+ 3.73)/5 ≅ 10.11% 

Aggregated monthly change rate and monthly growth rate of EC2 instances

In the previous sections, we showed how monthly change rate and monthly growth rate are calculated for an individual EC2 instance. Clumio calculates the summarized monthly change rate and monthly growth rate by using EC2 instances that have non NULL values (NULL values are represented by a “-”) for utilized sizes, change rates, and growth rates. 

For example, you have 10 EC2 instances, and Clumio’s calculations for utilized sizes, monthly change rates, and monthly growth rates of these 10 instances resulted in the following data.

 

Summarized utilized size of EC2 instances:

All EC2 instances that have the utilized size calculated successfully are added to calculate the summarized utilized size.

Utilized size: 100+50+16+10+200+30+50 ≅ 456GiB

Summarized monthly change rate of EC2 instances:

This is calculated by a weighted average of the utilized sizes of EC2 instances to consider an instance’s contribution to the summarized monthly change rate based on its utilized size.

A: (50GiB (3.25%)+ 16GiB (5%) + 10GiB (10.75%) + 200GiB(3.5%) + 30GiB(3.2%) + 50GiB(1.25%)

B: 50GiB + 16GiB + 10GiB + 200GiB + 30GiB + 50GiB  ≅ 356GiB

Avg. monthly change rate of EC2 instances: A/B = 1208.5/356 ≅ 3.39%

Summarized monthly growth rate of EC2 instances:

This is calculated by a weighted average of the utilized sizes of EC2 instances to consider an instance’s contribution to the summarized monthly growth rate based on its utilized size.

A: (50GiB (2%)+  16GiB (0%) + 10GiB(3.25%) + 200GiB(4.2%) + 30GiB(2.5%) + 50GiB(5.92%))

B: 50GiB + 16GiB + 10GiB + 200GiB + 30GiB + 50GiB  ≅ 356GiB

Avg. monthly growth rate of EC2 instances: A/B = 1343.5/356 ≅ 3.77%