AWS Multi-Account Deployment Strategy

Even though there is no one-size-fits-all approach when it comes to organising AWS Accounts, the strategy which I describe in this blog post should be a good start for many organisations. It has worked well for many of our customers at Labyrinth Labs. I also explore pros and cons of implementing a multi-account structure and outline what the adoption of multi-account structure looks like if you already have AWS accounts in place.

I assume you already know what AWS Multi-Account structure is. If you don’t or would like to refresh your knowledge, I recommend reading Establishing your best practice AWS environment document. AWS has published a great white-paper which takes a deep-dive into this topic. I recommend reading it if you want to see other AWS Multi-Account strategies or want to know more details about this topic.

What are the benefits of using multiple AWS Accounts (pros)?

Let’s go over the benefits of this setup so you get an idea whether implementing this into your AWS Cloud setup might be worth the effort.

Isolation

The most obvious benefit is the great isolation we put in place when we run workloads in multiple AWS Accounts. It goes without saying that development, test and production environments should be isolated. Even though it’s simple to isolate workloads within one AWS accounts on network level (multiple VPCs), it’s not that straightforward when it comes to non-VPC AWS Services such as IAM.

Security Boundaries

The AWS Multi-Account setup helps you with setting security boundaries per workload environment. For instance, you might want to allow public access on S3 buckets in development whereas you want to be confident there are no bucket policies in-use which allows public access in production. Although many security boundaries on workloads could be implemented within one AWS Account, it’s much easier to implement, maintain and scale them using AWS SCP policies which work on AWS Account level.

Simplified human access management

In case you embrace AWS Multi-Account approach with AWS SSO, the human access management to your AWS resources becomes more flexible and simpler to manage. You can easily grant developers full access (including IAM) to development and test environments whereas grant them no access to production whatsoever. If you wanted to do this within one AWS Account, you would need to make sure IAM policies attached to developers limit access to dev resources using resource tags or prefixes which is a cumbersome process if you compare it with AWS SSO Permission Sets attached to individual AWS Accounts.

AWS Service Quotas and API rate limits (throttling)

Since AWS Service Quotas and API rate limits apply on AWS Account level, you have the option to set different values on each AWS Account. It might make sense to have Lambda concurrency execution quota dedicated for production. If you have the workload environments in one AWS Account, development lambda functions excessive execution could use all the account quotas and cause throttling of production functions.

Limit Blast-Radius

The isolation also helps with limiting the impact of a misconfiguration or malicious actions. For example, a misconfiguration in dev AWS account has limited impact on production workload running in a separate AWS account.

Simplified cost management

A proper AWS tagging is required to generate useful cost reports. However, it is difficult to manage and force the AWS tag policies in some cases. For instance, a developer doing an experiment with a new AWS service does not want waste time thinking about organisation AWS tag policies. Multi-Accounts structure gives you another layer of filters you can use on untagged resources when generating cost reports or setting up AWS Budgets.

Are there any downsides (cons)?

◦Cross account access management is more complex than access management within one AWS Account. Ideally, we should not need to setup many cross account access policies because it breaks the isolation.

◦The process of splitting existing AWS Accounts might be very difficult or impossible without significant effort and downtime. In case we need to move an AWS resource from one account to the other, we need to recreate it. The earlier you setup AWS Multi-Accounts on your cloud adoption journey, the less effort it requires.

The AWS Multi-Account Structure Strategy

The strategy described in this section can be implemented as it is or modified according to your organisation’s needs. As mentioned in the beginning, there is no one-size-fits-all approach. Let me know whether this might or might not work for your organisation in the comments. The effort for the implementation and maintenance of this setup should be lowered with infrastructure as code tools like terraform. Implementation details and any terraform code examples are out of scope of this blog post.

◦AWS Organisation — AWS Service which consolidates AWS Accounts into a single manageable unit.

◦AWS SSO — Although we could set this up without AWS SSO, I highly recommend it because it simplifies user and access management. It not only centralises the user management, but also manages IAM Roles we need to create when setting up cross account access.

◦Service Control Policies — Allow you to set Security Boundaries on the accounts level.

◦OUs — Since AWS Accounts inherit SCP policies from the OU they belong to, it makes sense to group accounts into OUs that have common security and operational needs. For example, sandbox accounts might have lower security requirements than workload accounts.

◦Management and Member accounts — Management account is the one in which we create AWS Organisation itself, configure access management using AWS SSO and create OUs and SCP policies. Member accounts are the ones that host running workloads or specific services. The AWS billing of all accounts is consolidated to the management account.

◦Account’s baseline — Define a baseline configuration and apply it to each account individually. For example, configuration services such as Cloudtrail, AWS Config and GuardDuty.

◦Workload accounts — The accounts where you host workloads. You can split them per workload environment (dev, stage, prod) and group them into a single OU or multiple OUs per environment.

◦Audit account (Log archive) — Store all the logs you want to keep for audit purposes in a separate account. These logs should be considered as immutable and never changed. There should be very strict security policies set in place on this account. The workloads and tools that need to access these logs should use a cross account access using an IAM role with read-only permissions.

◦Sec account (Security Tooling) — All the security tools you want to use (AWS GuardDuty, AWS Config, AWS Firewall Manager, …). Other examples: AWS Athena for analysing Cloudtrail logs stored in audit account, 3rd party cloud security monitoring services and tools (SIEM).

◦Shared account — Tools and services which need access to workload accounts and are not required to be isolated in the workload environments. For example: Observability tools, CI/CD platform, Route53 DNS zones, … Another use-case for this account is a centralised network hub: VPN Connections, Transit Gateway.

◦Sandbox accounts — Sandbox environments where you can do experiments. These accounts shouldn’t have many security policies in place and shouldn’t have access to any company data. Resources created in these accounts are temporary. The difference between sandbox accounts and dev workload account is that services running in dev workload account have access to company data which is needed for service development. You can create one sandbox account for each team. Spending limits (AWS Budgets) should be set to prevent accidental high spend during the experiments. You could trigger AWS NUKE through a lambda function on a certain threshold.

Ideas for enhancing the strategy

◦Two AWS Organisations — If SCP policies start getting more complex, it’s better to have a development environment where we can create and test them. A separate development AWS Organisation may come in handy in this case.

◦Separate prod and non-prod CI/CD — In case the company security policies don’t allow having one centralised CI/CD platform, you can move it out from the Shared account to multiple accounts.

◦Separate prod and non prod Observability stack — The same reason as for the separation of prod and non-prod CI/CD.

Journey from single to multiple AWS Accounts

In case you like the idea of having multiple AWS Accounts and would like to start splitting your single AWS account, you should be aware of a few things which you might not anticipate:

◦It is not possible to just move existing resources between the accounts. In order to migrate a resource from one account to the other, you need to create it in the destination account and destroy it in the source account. This might be unacceptable for some production resources.

◦Due to the previous limitation, you most likely want to keep production workloads running in the current account. The current account will become the production workload account in the new AWS Multi-Accounts hierarchy.

◦You need to create a completely separate AWS account where you need to configure billing from scratch. That will become your management account where you create AWS Organisation and other member accounts. You invite the old account to the AWS Organisation.

◦As soon as the old account joins the new AWS Organisation, the generated billing of the old account will start accumulating in the consolidated billing of the AWS Organisation. You will end up paying two separate bills for that month in which you merge the old account into the AWS Organisation.The first bill includes spend before and the other includes spend after the merge.

◦AWS Cost explorer history of the old account resets which means you loose the history. You should export the cost explorer data if you want to keep the cost explorer history before the old account joins the organisation.

–Andrej Krnáč
Infrastructure & DevOps & Kubernetes Consultant at Labyrinth Labs