AWS S3: Troubleshooting AWS API Errors Encountered During S3 Bucket Backup
This document provides guidance on troubleshooting AWS API errors encountered during the backup of objects in an S3 bucket, which may result in partial or entire backup failures. Below, you'll find detailed information about the most common error messages, their causes, and suggested steps to resolve them.
Accessing Error Details
Before diving into specific errors, you can refer to the CSV file attached to the failed S3 backup task for more information about the failures encountered.
Steps to Access the CSV File:
- Login to the Clumio Portal with your credentials.
- Navigate to Activities in the left navigation pane.
- Under Tasks, apply the “Asset” filter and specify the name of the affected bucket.
- Find the task related to the “S3 asset SecureVault backup” and click on it to get to download csv link.
Download the CSV file associated with the failed task for a detailed breakdown of the errors.
ERROR # 1:
Not supported: Failed to get object …… :firmware_version_mapping/20250107-125229/_SUCCESS:operation error S3: GetObject, https response error StatusCode: 405, ………………., api error MethodNotAllowed: The specified method is not allowed against this resource.
The "405 MethodNotAllowed/InvalidRequest" error often occurs when attempting to back up a deleted object from an S3 bucket with versioning suspended.
Understanding the Cause
-
The error occurs when an object was previously present in the S3 inventory but was deleted before the Clumio backup job started. During the backup process, our API attempts to interact with this deleted object.
-
Since versioning is suspended on the S3 bucket, the object no longer has a regular version (i.e., it has no latest version available for retrieval). This means the API cannot locate a valid object version for backup.
-
When the API attempts to retrieve the object, it encounters a delete marker instead of a regular object version. The delete marker indicates that the object is deleted but doesn't provide the data needed for backup. As a result, the API call fails, triggering the 405 MethodNotAllowed/InvalidRequest error.
Next Steps for Resolution:
These errors are related to deleted objects that no longer exist in the source bucket. As such, they will be automatically resolved. In subsequent backup runs, Clumio will not attempt to back up these deleted objects once the inventory reflects the changes (i.e., once the deleted objects are removed from the inventory).
ERROR #2:
operation error S3: GetObject, https response error StatusCode: 412, ……………………………….., api error PreconditionFailed: At least one of the pre-conditions you specified did not hold
The PreconditionFailed error often occurs when an object is frequently updated throughout the day, but those updates are not reflected in the inventory report used by Clumio for the backup process.
Understanding the Cause:
-
During the backup process, Clumio relies on the inventory report from the source S3 bucket to identify which files need to be backed up. This report is sent over to Clumio when the daily backup job starts. The inventory report contains details such as file identifiers and properties, including the ETag (a unique identifier generated from the file’s hash).
-
Each time an object is modified or deleted, the ETag changes to reflect the new version of the object. If the inventory report being used by Clumio does not include these recent updates (for example, due to timing or delays in the report generation), the backup process will still reference the older ETag. This mismatch leads to failed API calls, which triggers the PreconditionFailed error in the AWS S3 API.
Next Steps for Resolution:
- Automatic Inclusion in Subsequent Backups: Objects that failed to back up due to the issue described above will typically be included in the next subsequent backup run. If the updates to the objects are infrequent, this may resolve the issue on its own.
- Enable Continuous Backup: However, if certain objects are frequently updated or deleted during the day, we recommend enabling continuous backup on the affected S3 bucket. This feature captures changes to the object every hour, ensuring that any updates are reflected in the backup and available for restoration.
To know more on continuous backups, refer this user guide: https://help.clumio.com/docs/s3-backups#s3-continuous-backup
ERROR #3:
operation error S3: CopyObject, https response error StatusCode: 403, RequestID: ………………………………, api error AccessDenied: Access Denied
These ”Access Denied” error messages, typically occur when the Clumio role lacks the necessary permissions to perform the required API calls for the backup. Below are some common causes of this issue:
Common Causes of "Access Denied" Errors:
-
Service Control Policies (SCPs) Restricting Actions: If Service Control Policies (SCPs) are in place, they may restrict certain actions on the S3 objects or bucket. SCPs provide an additional layer of control and can inadvertently block necessary operations needed for backups.
-
Boundary Permissions on Clumio Role Boundary permissions placed on the Clumio role may limit its access to the S3 bucket or its underlying objects. These boundary permissions act as a filtering mechanism and can prevent the role from accessing the required resources.
-
Explicit Deny in Bucket-Level Policies: A bucket-level policy (also known as a resource-based policy) may contain an explicit deny statement, which blocks access even if the role has the necessary permissions. It's important to verify whether such an explicit deny exists at the bucket level.
Next Steps for Resolution:
To resolve these "Access Denied" errors, ensure that the Clumio role has the appropriate permissions and that there are no conflicting policies (SCPs, boundary permissions, or explicit denies). For more detailed instructions on how to troubleshoot and resolve these issues, please refer to the following Knowledge Base article:
Updated about 1 month ago