AWS has everything you need for secure and reliable data storage. With Amazon S3, you can easily build a low-cost and high-available solution. Together with the available features for regional replication, you can easily have automatic cross-region backups for all data in S3.
Overview
This example is a CDK project in TypeScript. Together with CloudFormation StackSets, you can deploy all resources in all needed regions with a single command:
- S3 Bucket in primary region with custom KMS key
- CloudFormation StackSet for regional replications
- IAM Role with access for primary region and replications
Besides the default IAM Roles for CloudFormation Stacks, you only use managed AWS resources and no custom source code or data processing:
The CDK project is available on GitHub. It’s ready-to-use and you can easily configure all parameters for the CloudFormation stack:
// Configured ./aws/index.ts
const option = {
env: {
region: 'eu-central-1'
},
replications: [
'eu-west-1',
'eu-north-1'
]
}
Afterwards, run npx cdk deploy to deploy everything.
Primary Region and Data Source
All changes to data in the Amazon S3 Bucket in your primary AWS region are replicated to additional AWS regions. In the primary region, you need a Amazon S3 Bucket and a custom KMS key used for encryption.
import * as s3 from '@aws-cdk/aws-s3'
import * as kms from '@aws-cdk/aws-kms'
const key = new kms.Key(this, 'Key')
const alias = key.addAlias('archive')
const bucket = new s3.Bucket(this, 'Bucket', {
bucketName: `${props.prefix}-archive`,
encryption: s3.BucketEncryption.KMS,
encryptionKey: alias,
blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
bucketKeyEnabled: true,
versioned: true,
removalPolicy: cdk.RemovalPolicy.RETAIN
})
To use S3 bucket replication, you need to create an IAM Role with the permissions to access data in S3 and use your KMS key:
import * as iam from '@aws-cdk/aws-iam'
const role = new iam.Role(this, 'ReplicationRole', {
assumedBy: new iam.ServicePrincipal('s3.amazonaws.com'),
path: '/service-role/'
});
role.addToPolicy(
new iam.PolicyStatement({
resources: [
bucket.bucketArn
],
actions: [
's3:GetReplicationConfiguration',
's3:ListBucket'
]
})
);
role.addToPolicy(
new iam.PolicyStatement({
resources: [
bucket.arnForObjects('*')
],
actions: [
's3:GetObjectVersion',
's3:GetObjectVersionAcl',
's3:GetObjectVersionForReplication',
's3:GetObjectVersionTagging'
]
})
);
role.addToPolicy(
new iam.PolicyStatement({
resources: [
key.keyArn
],
actions: [
'kms:Decrypt'
]
})
);
With all that in place, the next step is to create an Amazon S3 Bucket and KMS key in all regions you want to use for replication.
CloudFormation StackSet
To avoid creating individual CloudFormation stacks in every region you want to use for replication, you can use a CloudFormation StackSet to automate the regional deployments. Currently, the AWS Cloud Development Kit only supports the low-level access to CloudFormation StackSet resources:
import * as cdk from '@aws-cdk/core'
new cdk.CfnStackSet(this, "StackSet", {
stackSetName: `${props.prefix}-archive-replication`,
permissionModel: "SELF_MANAGED",
parameters: [
{
parameterKey: 'Prefix',
parameterValue: props.prefix
},
{
parameterKey: 'ReplicationRole',
parameterValue: role.roleArn
}
],
stackInstancesGroup: [
{
regions: props.replications,
deploymentTargets: {
accounts: [this.account],
},
},
],
templateBody:templateReplicationData,
});
The templateReplicationData is a CloudFormation template containing the Amazon S3 and KMS resources for every region. The parameter ReplicationRole is need to grant access to the regional KMS key for the IAM Role used for replication.
Parameters:
Prefix:
Type: String
ReplicationRole:
Type: String
Resources:
Key:
Type: AWS::KMS::Key
Properties:
KeyPolicy:
Version: 2012-10-17
Id: access-account
Statement:
- Sid: Enable IAM User Permissions
Effect: Allow
Principal:
AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
Action: kms:*
Resource: '*'
- Sid: Replication
Effect: Allow
Principal:
AWS: !Ref ReplicationRole
Action:
- kms:Encrypt
- kms:ReEncrypt*
- kms:GenerateDataKey*
- kms:DescribeKey
Resource: '*'
KeyAlias:
Type: AWS::KMS::Alias
Properties:
AliasName: alias/archive/replication
TargetKeyId: !Ref Key
Bucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
Properties:
BucketName: !Sub ${Prefix}-archive-replication-${AWS::Region}
AccessControl: Private
PublicAccessBlockConfiguration:
BlockPublicAcls: Yes
BlockPublicPolicy: Yes
IgnorePublicAcls: Yes
RestrictPublicBuckets: Yes
VersioningConfiguration:
Status: Enabled
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: Yes
ServerSideEncryptionByDefault:
SSEAlgorithm: aws:kms
KMSMasterKeyID: !Sub arn:aws:kms:${AWS::Region}:${AWS::AccountId}:${KeyAlias}
The AWS resources in the replication regions use a specific name pattern to reference them in the next configuration. For example, the regional resources for eu-north-1 have these names and IDs:
- KMS Key
arn:aws:kms:eu-north-1:1234:alias/archive/replication - S3 Bucket
arn:aws:s3:::prefix-archive-replication-eu-north-1
With this naming pattern, you can now extend the IAM Role used for replication in your primary region:
role.addToPolicy(
new iam.PolicyStatement({
resources: props.replications.map(
region => `arn:aws:kms:${region}:${this.account}:alias/archive/replication`
),
actions: [
'kms:Encrypt'
]
})
);
role.addToPolicy(
new iam.PolicyStatement({
resources: props.replications.map(
region => `arn:aws:s3:::${props.prefix}-archive-replication-${region}/*`
),
actions: [
's3:ReplicateDelete',
's3:ReplicateObject',
's3:ReplicateTags'
]
})
);
role.addToPolicy(
new iam.PolicyStatement({
resources: props.replications.map(
region => `arn:aws:s3:::${props.prefix}-archive-replication-${region}`
),
actions: [
's3:List*',
's3:GetBucketVersioning',
's3:PutBucketVersioning'
]
})
);
Without the naming schema, you would not be able to accomplish this. The IAM Role needs access to the S3 Bucket and KMS Key in every region, but you cannot reference objects in a CloudFormation StackSet using CloudFormation templates or the CDK.
Last, but not least, you need to configure the S3 Replication Configuration for the S3 bucket in your primary location. The AWS CloudFormation Development Kit has no higher-level construct yet, but you can still use the low-level objects to configure replication:
const cfnBucket = bucket.node.defaultChild as s3.CfnBucket;
cfnBucket.replicationConfiguration = {
role: role.roleArn,
rules: props.replications.map(
(region, index) => (
{
id: region,
destination: {
bucket: `arn:aws:s3:::${props.prefix}-archive-replication-${region}`,
encryptionConfiguration: {
replicaKmsKeyId: `arn:aws:kms:${region}:${this.account}:alias/archive/replication`
}
},
priority: index,
deleteMarkerReplication: {
status: 'Enabled'
},
filter: {
prefix: ''
},
sourceSelectionCriteria: {
sseKmsEncryptedObjects: {
status: 'Enabled'
}
},
status: 'Enabled'
}
)
)
}
Again, this only works with hard-coded names and the pattern of predefined names for S3 buckets and KMS keys.
Summary
This uses the AWS Cloud Development Kit to create an AWS CloudFormation template to create an AWS CloudFormation stack. The AWS CloudFormation stack creates an Amazon S3 bucket, an AWS Identity & Access Management role, an AWS Key Management Service key and an AWS CloudFormation StackSet. The AWS CloudFormation StackSet uses an AWS CloudFormation template to create an AWS CloudFormation stack in all AWS regions. The AWS CloudFormation stack in every region creates an Amazon S3 bucket and an AWS Key Management Service key. You’re welcome. 🤯
The complex orchestration is need to deploy all resources with a single npx cdk deploy command. Of course, there are other ways to achieve this, but if you want to only rely on AWS services, have no custom code, and use a single deployment state, use this!