Person, Clothing, Pants, Indoors, Room, Crowd, Face, Chair, People, Audience

In Crowd — AWS on-demand elevated access

Author: Karthik Padmanabhan

Karthik has been with PageUp for over 5 years, having originally joining in a Technical Advisor role, he is now our Head of Architecture and has been a key contributor to our AWS architecture and strategy.



Some time ago at PageUp, we adopted AWS multi-account strategy. Today we have about 80 AWS accounts in PageUp.

The multi-account strategy enabled autonomy and ownership at PageUp, while like anything, it has trade-offs.

Even with a handful of accounts, managing elevated access was a big challenge for us. We did not have the luxury to have a person be responsible for this nor did we want centralised responsibility for it.

So, when it came to a multi-account strategy, we knew it would pay to be proactive and automated.

As you must have figured it out by now, this article discusses how at PageUp, we automated on-demand elevated access to AWS for our development teams.

Principles

We tried to stick to the “Principles of least privilege.”

  • Everyone has ReadOnly access to AWS Accounts they belong to by default.
  • Only provide elevated access to AWS accounts that are requested/needed.
  • Elevation of access is only for an exceptional situation, on-demand, peer-reviewed, auditable, time-bound and automated.

The last principle especially highlights the importance given in PageUp to encourage Continuous Deployment and Infrastructure as Code practices.

Intended experience

Elevated AWS access experience @PageUp

Most of the times developer require elevated access to AWS only when they are dealing with a production situation, so we wanted to ensure the experience is as straight forward and pain-free as possible.

In Crowd — The solution

AWS State Machine

AWS State Machine

Given the process requires some form of the state machine, our readily available choice was AWS Step function.

We had some experience already with AWS Landing Zone that we deployed not long ago — so the team was quite comfortable with it.

Organising AWS Accounts

Like I mentioned above, we have more than 80+ accounts; we use a combination of AWS Organisation Units & PageUp AWS Domains to organise them.

Domain-Vending-Machine

Each of PageUp AWS Domain has seven accounts. The creation, as well as management of these AWS accounts, are based on Continuous deployment system called Domain Vending Machine (It uses AWS’s Account Vending Concept — a subject of another day).

Example: HR Organisation is a big thing in PageUp, so is Screening (in the context of recruitment). They have their own PageUp AWS Domains.

IAM User & Group management

All our users, groups and their associated permissions reside in a Git repo. It allows for easy audit and change management. As you see below each user is associated with groups each representing a PageUp AWS Domain.

Example

users:  
  karthikp:    
    groups:      
      - SoftwareDevelopers       
      - RecruitmentDomain
      - CareersDomainterence:    
   
groups:      
      - SoftwareDevelopers       
      - RecruitmentDomain
      - CareersDomain
...


Groups such as “SoftwareDevelopers” provide access to standard IAM actions that are basic for all developers.

For every PageUp AWS Domain, we have two policies as highlighted below.

groups:
  Screening:    
    policy_file: Screening/Screening.j2    
  ScreeningMaintenance:    
    policy_file: Screening/ScreeningMaintenance.j2
...


The maintenance policy is used for elevation and will have PowerExecutionRole or similar associated with the group.

{# Default Policy sample #}
{    
  "Statement": [        {            
    "Sid": "AllowAssumeScreeningUserExecutionRole",
    "Effect": "Allow",
    "Action": [ "sts:AssumeRole" ],
    "Condition": {
         "Bool": {                   
            "aws:MultiFactorAuthPresent": "true"                
         }        
     },
     "Resource": [                
        {# screening_deploy #}
        "arn:aws:iam::xxx:role/ReadOnlyExecutionRole",
        {# screening_staging #},
        "arn:aws:iam::xxxx:role/ReadOnlyExecutionRole",
         ....{# Maintenance policy sample #}
{    
  "Statement": [        {            
    "Sid": "AllowAssumeScreeningUserExecutionRole",
    "Effect": "Allow",
    "Action": [ "sts:AssumeRole" ],
    "Condition": {
         "Bool": {                   
            "aws:MultiFactorAuthPresent": "true"                
         }        
     },
     "Resource": [                
        {# screening_deploy #}
        "arn:aws:iam::xxx:role/PowerUserExecutionRole",
        {# screening_staging #},
        "arn:aws:iam::xxxx:role/PowerUserExecutionRole",
         ....

Circling back to the workings

The system knows that every PageUp AWS domain has two policies, a default one and a maintenance policy. Once a request is approved, it attaches the user with the corresponding maintenance policy for a configured amount of time.

a simple explanation on the system  
a simple explanation on the system

It’s been good 18 months since we put in place and since then never bothered to touch it because

It’s one of those simple solutions that just works


Credits

The credits for designing and building this simple solution goes to my team of excellent developers current and past.

Terence Tham — Designed overall solution.
Colin Scott — Among several things, he named it as “In Crowd.”
Jack G — Contributed to “PageUp-AWS”

Recommended Jobs

Senior Software Developer

422588 Philippines Philippines Philippines Product Development
Using design thinking to deliver incremental value to our clients creates opportunities for us to split our customer-loved .NET app into event-sourced micro-services and continuously-delivered single-page applications. Come get your hands dirty as...

Product Manager

422593 Philippines Philippines Philippines Product Product Management
People are a company's greatest asset! We have an opportunity for a Product Manager to join our growing global Product Team. Customers are at the heart of what we do, so it's important that our teams can effectively and efficiently deliver custome...

Project Coordinator (Early Career or Graduate position)| Feb 2022 Start

422596 Melbourne Melbourne Australia Melbourne, Australia Customer Success Professional Services
Does your dream graduate or early career role have you planning and coordinating projects in a fast paced tech environment? Then you've certainly come to the right place!

Customer Success Manager

422589 Melbourne. Sydney Melbourne. Sydney Australia Melbourne, Australia. Sydney, Australia Customer Success Customer Success
Partner with PageUp's enterprise clients to drive customer success! Work in one of our other lovely states? We'd love to hear from you too! Remote / WFH opportunities available.

Software Developer

422590 Melbourne Melbourne Australia Melbourne, Australia Product Development
People are a company's greatest asset! Luckily we realised that early on and built an innovative SaaS platform to help clients (and ourselves of course!) recruit, develop, engage and support their people. Come get your hands dirty as we set out on...

Implementation Consultant

422597 Melbourne Melbourne Australia Melbourne, Australia Customer Success Professional Services
Drive customer success by managing the successful implementation of PageUp's HR Software solution to global clients.