AWS Snapshots Automation Using AWS Lifecycle Policy

AWS Elastic Block Storage (EBS) snapshots enable you to create point-in-time versions of EBS volumes. You can then use these snapshots for backup and recovery purposes, as well as version control. However, when dealing with a big quantity of volumes, it does not make sense to create each snapshot manually. This is what the AWS Lifecycle Policy is for. In this article, you will learn how to use the AWS Lifecycle Policy for automating snapshots.

What Are AWS EBS Snapshots?

AWS Elastic Block Storage (EBS) snapshots, sometimes called AWS snapshots, are images of EBS volumes. These snapshots enable you to backup and restore data from EBS volumes. This means you can shut down volumes that aren’t in use or recover damaged volumes without fear of data loss.

EBS volumes are used to store data for Elastic Cloud Compute (EC2) instances. Without EBS, EC2 instances only have ephemeral storage. This means that data is lost if EC2 instances fail or are shut down. By attaching EBS volumes and taking snapshots, you can retain data from EC2 instances indefinitely regardless of whether the instance continues to run. 

When creating snapshots of your EBS volumes, there are a few specifics you should be aware of, including:

• Data transfer to S3—snapshots are automatically transferred to AWS Simple Storage Service (S3) when complete. When this occurs, you are responsible for both transfer and storage costs for however long you keep the snapshot. The benefit is that S3 is cheaper storage than EBS. 

• Incremental backups—all snapshots created are incremental, with each reflecting on those changes made since the last snapshot. This helps reduce backup times and sizes, saving you money. When backups are restored, all previous backups are used to ensure that no data is left out. 

• Snapshots are AWS and region-specific—whatever region you create your snapshot in is the region it is saved to. You cannot access snapshots from other regions or restore to other regions without first transferring a copy of the snapshot. You also cannot use snapshots to create instances outside of AWS. For example, if you want to create an on-premises virtual machine with the same configurations.

• Encrypted volumes don’t offer incremental backups—if your volumes are encrypted, you can only create full backup snapshots. This means that encrypted volumes take longer to backup and snapshot transfer and storage costs more. However, backups of encrypted volumes do ensure that that your data remains protected even if snapshots are stolen.

AWS Snapshots Automation Process with Data Lifecycle Manager 

Data Lifecycle Manager (DLM) is a service offered by AWS that you can use to automate the creation and management of snapshots. It works according to lifecycle policies that you define and attaches to specific volumes according to metadata tags. In these policies, you can define how frequently backups are created and when, how many backups are retained, and how long backups are kept for.

When tagging your volumes, you can add multiple tags to each volume to target it with multiple backup policies. This enables you to manage backups according to volume purpose, customer, or any other trait you need. However you create tags, remember that you can also use this metadata to monitor and manage your volumes. Making tags functional and clear helps you maintain accurate backups and makes maintenance easier.

Once you set policies to run, your backups will continue to be created until you deactivate policies or run out of your allotted number of snapshots (10k). Although this number may seem high if you create many policies that overlap, or have very high retention numbers or long lifespans you can easily run out.

To prevent this from happening, try to set conservative policies and make sure that you are only creating snapshots when you need to. If you need frequent snapshots, you may want to manually manage older copies to reduce your numbers after a certain period.

The Pros and Cons of Amazon Data Lifecycle Manager

If you are planning to take advantage of DLM, you should make sure that you first understand it’s pros and cons. Understanding these can help you ensure that you are using the service as effectively as possible.
Pros of DLM include:

• Easy configuration—DLM policies can be written as JSON files or configured through the dashboard. This enables you to easily create either one-off or bulk policies. To use either of these methods you just need to create an IAM role with the proper permissions and attach it to the policies. 

• Monitoring—you can easily monitor your policies and snapshots through Amazon CloudWatch. This service enables you to see when policies run, confirm that snapshots are created successfully, to gauge the impacts of snapshot creation on performance, and to confirm your remaining storage.

       Cons of DLM include: 

• Configuration changes are not retroactive—if you decide to change a policy, it does not alter the status of any snapshots created with the old policy. This means that old snapshots’ lifecycles will no longer be managed automatically. 

• General backup times—when you schedule a time for your snapshots, creation occurs within an hour window of time. For example, if you specify that backups should be created at midnight, creation might not happen until 12:59 PM.

• Policy limits—you are only allowed to create 100 policies per region and one schedule per policy. While this may be enough for many users it may not allow the granular control you need.

• While DLM isn’t perfect, it does offer a good automation solution for those who do not wish to or aren’t able to create automatic snapshots otherwise. Additionally, using DLM does not prevent you from either creating snapshots manually or using other automation methods, such as Lambda scripts. 

Conclusion

The AWS Lifecycle Policy is a native AWS tool. That means it is a feature that integrates seamlessly with your EBS account. The process is fairly simple. You create a policy and then let the Data Lifecycle Manager create snapshots for you. You can then easily monitor the process through Amazon CloudWatch. 

As in any service, there are drawbacks to this process. Configuration changes, for example, are not retroactive. In addition, there is an hour lag between configuration time and actual backup. You are also limited to 100 policies per region and one schedule per policy. Take these limitations into account when considering this process, to ensure your backup is viable.