EBS inventory detail calculations

Inventory Details for EBS in Clumio Discover provides insight into utilized sizes, change rates, and growth rates of EBS volumes in AWS accounts based on EBS snapshots. All calculations are based on EBS snapshots that meet the requirements explained in the following sections. 

Utilized Size of EBS Volume

AWS offers different options for EBS volume sizes when spinning up block-level storage devices in AWS. For example, a General Purpose SSD (gp2) volume can have a volume size between 1GiB to 16TiB. If you spin up a volume of size 1GiB, then this volume size is called the provisioned size of the EBS volume. The utilized size of the EBS volume is the size of the data that is stored in this provisioned volume. For example, if you write 512MiB of data, then the utilized size of the EBS volume is 512MiB. 

The following sections explain how Clumio calculates the utilized sizes of EBS volumes.

EBS snapshot-based calculations

EBS snapshots are a point-in-time copy of the data, and each snapshot contains all of the information needed to restore the data to a new EBS volume - which means that we can infer the EBS volume’s utilized size based on its snapshot size. Clumio looks for the most recent snapshot of the EBS volume in the last 90 days and this snapshot’s size is treated as the utilized size of the EBS volume.

How Clumio calculates the utilized size of an EBS volume:

• For example, there is an EBS volume that is provisioned with 10 blocks (provisioned size) on July 10, 2022. The volume has some blocks with data written to blocks 0 through 5 (utilized size) on July 13, 2022.
• Snapshot A, taken on July 15, 2022, is a seeding backup which stores the data on blocks 0 through 5 of the source volume.
• For subsequent snapshots taken on the following days, some blocks are overwritten and other blocks are newly written to. For example, after snapshot A was taken, data in block 1 of the volume changes from B0 to B1 and this change is captured in snapshot B taken on July 16, 2022. Similarly, new data G0 written to Block 6 is also captured in snapshot B.
• AWS snapshots are incremental backups, which means that only the blocks on the volume that have changed after the most recent snapshot are stored. This is indicated in the table by empty cells for the snapshots where empty cells are referencing the previous snapshot storing the data.
• If we calculate the utilized size of the EBS volume on Aug 01, 2022, then Clumio looks for the size of the most recent snapshot in the last 90 days - which in this case is snapshot D, taken on July 18, 2022.
• The size of snapshot D includes the bolded letters in the table with data (A2, B2, C1, D1, E0, F0, G0, and H0) starting from block 0 through block 7 and the total snapshot size is the sum of these blocks. Since each EBS volume block is 512KiB in size, the snapshot size is 8 blocks x 512KiB = 4MiB. Hence the utilized size of EBS volume is 4MiB.

Summarized utilized size of EBS volumes

As an example, you have three EBS volumes that have a number of snapshots taken on different dates. 

Assuming that today is Aug 01, 2022, Clumio looks back 90 days in time and uses the most recent snapshot’s size as the EBS volume’s utilized size. The most recent snapshot is chosen because this snapshot is indicative of the latest utilized size of the EBS volume. 

 

The summarized utilized size of EBS volumes is calculated by adding the utilized sizes of V1 and V3.  

Utilized size: 4.5 GiB + 32GiB ≅ 36.5GiB

 

Change vs Growth

Data can be written to blocks storing data or new blocks in an EBS volume. When a block storing data is overwritten with new data, we refer to it as “change”, and when data is written to an entirely new block, we refer to it as “growth”.

Change in EBS volume

The following table shows how change occurs in an EBS volume. For simplicity, consider an EBS volume with 10 blocks with indexes 0 through 9. On Day 1, data is written to 6 blocks starting from block 0 through block 5. On Day 2, block 1’s data B is overwritten with new data B1 and we refer to this as “change”. Similarly, block 3’s data D changes from D to D1 on Day 2. All the changed blocks from Day 1 to Day 3 are highlighted in bolded letters. 

Growth in EBS volume

The following table shows how growth occurs in an EBS volume. For simplicity, consider an EBS volume with 10 blocks with indexes 0 through 9. On Day 1, data is written to 5 blocks starting from block 0 through block 4. On Day 2, data is written to new blocks i.e. block 5 and block 6 and we refer to this as “growth”. Similarly, data is written to block 7 and block 8 on Day 3. The bolded letters indicate growth from Day 1 to Day 3 in this EBS volume.

Why change and growth are important

In the backup world, change and growth are important to understand as both impact your backup storage and the corresponding costs. If a web application writes 10GiB newly to an EBS volume of size 10GiB, it results in a growth of 10GiB and total size of 20GiB. In this case, you can expect your backup storage to be 20GiB. The 10GiB growth will be held onto forever in the backup. On the other hand, if the web application is changing the existing data stored in the 10GiB EBS volume, then your backup storage can vary widely due to the changes. This is because change causes multiple versions of the same block to be stored in the backups, and these versions are retained for the duration specified by your backup policy. Therefore, change and growth in your EBS volumes are important to understanding what is contributing to your backup sizes.

To provide insights on how EBS volumes in your AWS accounts are changing and growing, Clumio calculates daily and monthly change rates and growth rates that can be used to estimate backup sizes if you are taking daily and monthly backups. 

Change Rate of an EBS Volume

Change rate refers to the rate at which the blocks of an EBS volume change over time i.e. existing blocks of data getting overwritten by data for a specific time period. 

 

Daily change rate of an individual EBS volume

A daily change rate explains the average rate of change in an EBS volume on a daily basis.  

How Clumio calculates the average daily change rate of an EBS volume:

For example, consider an EBS volume that has a number of snapshots taken on different days.

Figure 1:

Assuming that today is Aug 01, 2022, Clumio calculates the average daily change rate of this EBS volume in the following steps.

Step 1:

For the last 3 months i.e. from May 01, 2022 to Aug 01, 2022, identify snapshots that are 20 to 28 hours apart in time. The time difference is required to ensure that the two snapshots are approximately a day apart for calculating the daily change rate. Some snapshots that meet this requirement are:

 

Note: 

• Each EBS volume should have snapshots that meet Clumio’s requirements and should result in 2 to 5 snapshot pairs for Clumio to calculate the change rate successfully. If not, the change rate of EBS volume is shown as “-” in the UI.
• Snapshots copied to a secondary region are not used in the calculations.
• If you have multiple snapshots taken on two days, then the latest snapshot of each day is used to create a snapshot pair to prioritize recency. For example, if you have snapshot A, snapshot B taken in order on Day 1 and snapshot C, snapshot D taken in order on Day 2, then we use snapshot B and snapshot D for calculating the daily change rate. 

Step 2:

An EBS volume can have the same blocks overwritten or new blocks written over time. Snapshots are point-in-time that capture the volume state when they are taken. To account for same blocks being overwritten and new blocks written, Clumio calculates two bounds for each snapshot pair:

1. Same blocks changing: Consider an EBS volume where the same block i.e. block 1 changes every 6 hours. The average change rate for a day would be 25%, implying that 1 out of 4 blocks changed.

 

2. Different blocks changing: Consider the same EBS volume where a different block changes every 6 hours. The average change rate for a day would be 100%, implying that 4 out of 4 blocks changed.

As described above, the change rate of the EBS volume in question can vary from 25% to 100% depending on whether the same blocks changed or different blocks changed. To account for this, Clumio calculates the two bounds for each snapshot pair as follows.

1. For the first snapshot pair S1 and S2, calculate the utilized sizes of S1 and S2 by making ListSnapshotBlocks API calls for each snapshot.
2. Calculate the delta between S1 and S2 (i.e. the blocks that are different between the two snapshots) by making ListChangedBlocks API calls.
3. Bound 1: We take the delta between the snapshots and subtract the growth between them for fetching changes alone using the formula:

(delta- (snapS2UtilizedSize - snapS1UtilizedSize))/(snapS1UtilizedSize)

From Figure 1, utilized sizes of S1 (Snap1Size) and S2 (Snap2Size) are 2GiB and 2.25GiB respectively. Assuming that the delta between S1 and S2 is 1GiB, bound 1 will be:

Bound 1: (1GiB - (2.25GiB - 2GiB)) / 2GiB ≅ 0.375 or 37.5% 

1. Bound 2: We divide the previous result by the number of days between the snapshots. S1 was taken at 10pm and S2 was taken at 6pm the following day. The time difference between the snapshots is 20 hours or 0.83 days.

Bound 2: 0.375/ 0.83 ≅ 0.451 or 45.1%

1. Bound 1 will be the min. change rate and Bound 2 will be the max. change rate for S1 and S2 snapshot pair. The avg. change rate is calculated by averaging min. and max. change rates.
2. The snapshot pair will be displayed as follows.

Step 3:

Clumio identifies a number of snapshot pairs for each EBS volume and calculates min. change rate, max. change rate and avg. change rate for each pair. Assuming that the delta from the ListChangedBlocks API calls is 1GiB for all snapshot pairs of the EBS volume, the following data is generated.

The average change rate of the EBS volume is calculated by taking an average of avg. change rates of all snapshot pairs. 

Avg. daily change rate: (41.3+41.82+22.93+18.65+15.12)/5 ≅ 27.96%

 

Growth Rate of an EBS volume

Growth rate refers to the rate at which an EBS volume grows over time by writing to new blocks.

Daily growth rate of an individual EBS volume

A daily growth rate explains the average rate of growth in an EBS volume on a daily basis. 

How Clumio calculates the average daily growth rate of an EBS volume

For example, consider an EBS volume that has a number of snapshots taken over different days. Clumio uses the same snapshots to calculate the daily change rate and daily growth rate of an EBS volume.

Figure 2:

Assuming that today is Aug 01, 2022, Clumio calculates the average daily growth rate of this EBS volume in the following steps.

Step 1:

For the last 3 months i.e. from May 01, 2022 to Aug 01, 2022, identify snapshots that are 20 to 28 hours apart in time. The time difference is required to ensure that the two snapshots are approximately a day apart for calculating the daily growth rate. Some snapshots that meet this requirement are:

 

Step 2:

For snapshots S1 and S2, which are 20 hours (0.83 day) apart in time, calculate the daily growth by using the following  formula. 

(snapS2UtilizedSize - snapS1UtilizedSize)/snapS1UtilizedSize/(snapS2TimeInDays-snapS1TimeInDays)

Daily growth rate: (2.25GiB - 2GiB)/2GiB/0.83 ≅ 15.06%

The snapshot pair will be displayed as follows.

 

Step 3:

Clumio identifies a number of snapshot pairs for each EBS volume and calculates growth rate for each pair resulting in the following table that is displayed in the UI.

 

The average daily growth rate of the EBS volume is calculated by taking an average of growth rates of all snapshot pairs. 

Avg. daily growth rate: (15.06+ 0 +22.24+4.83+6.57)/5 ≅ 9.74%

 

Aggregated daily change rate and daily growth rate of EBS volumes

In the previous sections, we discussed how daily change rate and daily growth rate are calculated for an individual EBS volume. Clumio calculates summarized daily change rate and daily growth rate by using the EBS volumes that have non NULL values (NULL values are represented by a “-”) for utilized sizes, change rates and growth rates. 

For example, you have 10 EBS volumes and Clumio’s calculations for utilized sizes, daily change rates, and daily growth rates of these 10 volumes result in the following data.

 

Summarized utilized size of EBS volumes:

All EBS volumes that have the utilized size calculated successfully are added to calculate the summarized utilized size.

Utilized size: 10+1.5+5+16+10+20+16+5 ≅ 83.5GiB

 

Summarized daily change rate of EBS volumes :

This is calculated by a weighted average of the utilized sizes of EBS volumes to consider a volume’s contribution to the summarized daily change rate based on its utilized size.

 

A: (10GiB (4%)+ 1.5GiB (3.5%) + 16GiB (5%) + 10GiB(5.2%) + 20GiB(3.2%) + 16GiB(1.5%) + 5GiB(3.25%))

B: 10GiB + 1.5GiB + 16GiB + 10GiB + 20GiB + 16GiB + 5GiB ≅ 78.5GiB

Avg. daily change rate of EBS volumes: A/B ≅ 281.5/78.5 = 3.58%

 

Summarized daily growth rate of EBS volumes :

This is calculated by a weighted average of the utilized sizes of EBS volumes to consider a volume’s contribution to the summarized daily growth rate based on its utilized size.

 

A: (10GiB (0.5%)+ 1.5GiB (1.2%) + 16GiB (0%) + 10GiB(4%) + 20GiB(3%) + 16GiB(0.5%) + 5GiB(1.2%))

 

B: 10GiB + 1.5GiB + 16GiB + 10GiB + 20GiB + 16GiB + 5GiB ≅ 78.5GiB

 

Avg. daily growth rate of EBS volumes: A/B = 120.8/78.5 ≅ 1.53%

 

Change Rate of EBS Volume

Change rate refers to the rate at which the blocks of an EBS volume change over time, i.e., existing blocks of data getting overwritten by data for a specific time period. 

 

Monthly change rate of an individual EBS volume

A monthly change rate explains the average rate of change in an EBS volume on a monthly basis. 

How Clumio calculates the average monthly change rate of an EBS volume

For example, an EBS volume that has snapshots taken in different months.

 

Figure 1:

Assuming that today is Aug 01, 2022, Clumio calculates the average monthly change rate of this EBS volume in the following steps.

 

Step 1:

For the last year, i.e., from Aug 01, 2021 to Aug 01, 2022, identify snapshots that are between 20 to 40 days apart in time. The time difference is required to ensure that the two snapshots are approximately a month apart for calculating the monthly change rate. Some snapshots that meet this requirement are:

 

 

Note: 

• Each EBS volume should have snapshots that meet Clumio’s requirements and should result in 2 to 5 snapshot pairs for Clumio to calculate the change rate successfully. If not, the change rate of EBS volume is shown as “-” in the UI.
• Snapshots that are copied to a secondary region are not used in the calculations.
• If you have multiple snapshots taken in two months, then the latest snapshot of each month will be used to create a snapshot pair to prioritize recency. For example, if you have two snapshots in January: snapshot A taken on Jan 10th, snapshot B taken on Jan 28th, and two snapshots in February: snapshot C taken on Feb 15th, snapshot D taken on Feb 25th, then we use snapshot B and snapshot D for calculating the monthly change rate.

Step 2:

As explained in the section of daily change rate, an EBS volume can have the same blocks overwritten or new blocks written over time. To account for this, Clumio calculates two bounds for each snapshot pair. 

Consider S1 and S2 snapshots from Figure 2 which are 793 hours apart in time and assuming that the delta between the snapshots is 1GiB, the two bounds are calculated as follows.

Bound 1: (delta - (snapS2UtilizedSize - snapS1UtilizedSize))/snapS1UtilizedSize

Bound 1: (1GiB - (2.5GiB - 2GiB)) / 2GiB ≅ 0.25 or 25% 

 

For Bound 2, we divide the previous result by the number of months between the two snapshots i.e. 793 hours = 793/24/30 = 1.1 month

Bound 2: 0.25 / 1.1 ≅ 0.22 or 22% 

The two bounds of a snapshot pair will be shown as min. change rate and max. change rate. The avg. change rate is calculated by averaging min. and max. change rates.

 

Step 3:

Clumio identifies a number of snapshot pairs for each EBS volume and calculates min. change rate, max. change rate and avg. change rate for each pair. Assuming that delta from ListChangedBlocks API calls is 1GiB for all snapshot pairs of the EBS volume, we generate the following data for all snapshot pairs.

 

 

The average change rate of the EBS volume is calculated by taking an average of avg. change rates of all snapshot pairs. 

Avg. monthly change rate: (23.5+27.55+30.02+22+13.4)/5 ≅ 23.29%

 

EBS volume growth rate

Growth rate refers to the rate at which an EBS volume grows over time by writing to new blocks.

Monthly growth rate of an individual EBS volume

A monthly growth rate explains the average rate of growth in an EBS volume on a monthly basis. 

 

How Clumio calculates the average monthly growth rate of an EBS volume

For example, consider an EBS volume that has a number of snapshots taken over different days. Clumio uses the same snapshots to calculate monthly change rate and monthly growth rate of an EBS volume.

 

Figure 2

Assuming that today is Aug 01, 2022, Clumio calculates the average monthly growth rate of this EBS volume in the following steps.

 

Step 1:

For the last year, i.e., from Aug 01, 2021 to Aug 01, 2022, identify snapshots that are between 20 to 40 days apart in time. The time difference is required to ensure that the two snapshots are approximately a month apart for calculating the monthly growth rate. Some snapshots that meet this requirement are:

 

Step 2:

For snapshots S1 and S2, which are 793 hours (1.1 month) apart in time, calculate the monthly growth by using the following  formula.

(snapS2UtilizedSize - snapS1UtilizedSize)/snapS1UtilizedSize/(snapS2TimeInMonths-snapS1TimeInMonths)

Monthly growth rate: (2.5GiB - 2GiB)/2GiB/1.1 ≅ 22.72%

The snapshot pair will be displayed as follows.

 

Step 3:

Clumio identifies a number of snapshot pairs for each EBS volume and calculates growth rate for each pair resulting in the following table.

 

The average monthly growth rate of the EBS volume is calculated by taking an average of growth rates of all snapshot pairs. 

Avg. monthly growth rate: (22.72+ 31.32 +7.32+3.48+17.7)/5 ≅ 16.5%

 

Aggregated monthly change rate and monthly growth rate of EBS volumes

In the previous sections we discussed how monthly change rate and monthly growth rate are calculated for an individual EBS volume. Clumio calculates summarized monthly change rate and monthly growth rate by using the EBS volumes that have non-NULL values (NULL values are represented by a “-”) for utilized sizes, change rates and growth rates.  

For example, assume you have 10 EBS volumes and Clumio’s calculations for utilized sizes, monthly change rates and monthly growth rates of these 10 volumes resulted in the following data.

 

Summarized utilized size of EBS volumes:

All EBS volumes that have the utilized size calculated successfully are added to calculate the summarized utilized size.

Utilized size: 10+1.5+5+16+10+20+16+5 ≅ 83.5GiB

 

Summarized monthly change rate of EBS volumes :

This is calculated by a weighted average of the utilized sizes of EBS volumes in order to consider a volume’s contribution to the summarized monthly change rate based on its utilized size.

A: (10GiB (15%)+ 1.5GiB (12.5%) + 16GiB (10.2%) + 10GiB(9.5%) + 20GiB(19.75%) + 16GiB(11.5%) + 5GiB(13%))

B: 10GiB + 1.5GiB + 16GiB + 10GiB + 20GiB + 16GiB + 5GiB ≅ 78.5GiB

Avg. monthly change rate of EBS volumes: A/B = 1070.95/78.5 ≅ 13.64%

 

Summarized monthly growth rate of EBS volumes

This is calculated by a weighted average of the utilized sizes of EBS volumes in order to consider a volume’s contribution to the summarized monthly growth rate based on its utilized size.

A: (10GiB (2%)+ 1.5GiB (1.2%) + 16GiB (0%) + 10GiB(4%) + 20GiB(3.5%) + 16GiB(1.5%) + 5GiB(5.2%))

B: 10GiB + 1.5GiB + 16GiB + 10GiB + 20GiB + 16GiB + 5GiB ≅ 78.5GiB

Avg. monthly growth rate of EBS volumes: A/B = 181.8/78.5 ≅ 2.31%