Managing the AWS Cloud Secrets - The Best Possible Way

Imagine this: You are in the charge of running a busy online business with a large cloud infrastructure. Your application processes thousands of transactions and connects to databases, payment gateways, and third-party APIs. One day, you wake up to discover that a developer accidentally exposed the public key in the repository, leading to an unauthorised attempt on your infrastructure. This scenario isn’t uncommon that’s why proper cloud secrets management isn’t just best practice, it is important for business survival.

In cloud environments, managing secrets can sometimes be daunting. Your main goal should be to avoid accumulation of unprotected data and disorderliness in your secrets’ management, and setup. But the question is… how? The answer lies in a key component of your cloud infrastructure setup called “Cloud Secrets Management”.

Understanding Cloud Secrets Management

Firstly, let’s understand the concept of cloud secrets. What are cloud secrets? Cloud secrets are sensitive credentials your application needs to function for users. Sensitive credentials such as Database connection strings, API keys for payment processors, OAuth client secrets, SSL/TLS certificates, and Encryption keys. Loosing these secrets could cause a serious disaster for your application. Picture this as locking your house, and leaving the premises with your keys under your doormat, hoping for safety.

In the early days of software development, the golden standard was:

# config.env
DB_PASSWORD=myS3cureP@ssw0rd
STRIPE_API_KEY=sk_live_123456789
AWS_SECRET_KEY=AKIAIOSFODNN7EXAMPLE

This simply was,

• Store everything on plain text files.
• Move it to the application servers.
• Configure the text files in the servers for only authorised users and applications to access.

While this was safe in its own way, these credentials were still vulnerable and risky while in production because;

• Files can be committed accidentally into version control.
• No encryption at rest
• No automated rotation of credentials after a given time.

Imagine managing a large heap of environmental variables and secrets for a large infrastructure system all by yourself. Previously, you may have kept these codes in a plain notebook and they will be easy for you to understand initially, but becomes a daunting task as the application keeps running.

In modern times, secrets can be managed both at rest and in transit. Also, they integrate seamlessly with your CI/CD tools removing all chances of human error.

In an AWS cloud environment, managing your cloud secrets is not only about keeping them safe, but it’s also about giving them access to the right applications, and the right stakeholders without putting these secrets at risk.

AWS Secrets Manager

Let’s start with one of the big boys in the cloud secrets game: “AWS SECRETS MANAGER”. This is like a vault where you store your secrets, and AWS takes care of encrypting them and makes sure they are only accessible to those who need them. Here’s a practical example using Python:

import boto3
import json

def get_secret():
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name='us-west-2'
    )
    
    try:
        secret_value = client.get_secret_value(
            SecretId='prod/app/database'
        )
        return json.loads(secret_value['SecretString'])
    except Exception as e:
        raise e

# Usage
db_credentials = get_secret()
connection = mysql.connector.connect(
    host=db_credentials['host'],
    user=db_credentials['username'],
    password=db_credentials['password']
)

You also get built-in integration with other AWS tools. Sounds lovely. With AWS Secrets Manager, you don’t have to change your passwords every few months. AWS does it for you.

For example, a team managing an e-commerce micro-service architecture can use Secrets Manager to automatically rotate database credentials every 30 days.

When this rotation occurs, Secrets Manager will:

• Create a new Password
• Updates the database
• Tests the new credentials
• Makes the new secrets available to applications.

AWS SSM Parameter Store

Think of the AWS Systems Manager Parameter Store as a younger sibling to the AWS Secrets Manager. Solution Architects and Cloud Engineers prefer this option not only for storing secrets but also for storing configuration data. If you have database connection strings or other sensitive application settings, the Parameter is your best bet.

Here’s a practical example in Python.

import boto3

def get_parameter(param_name):
    ssm = boto3.client('ssm')
    response = ssm.get_parameter(
        Name=param_name,
        WithDecryption=True
    )
    return response\['Parameter'\]['Value']

# Usage
api_key = get_parameter('/myapp/prod/api-key')
database_url = get_parameter('/myapp/prod/database-url')

Just like the e-commerce micro-service in the previous section, engineers can use Parameter Store to manage configuration across different environments.

/myapp/prod/payment-gateway-url
/myapp/prod/redis-host
/myapp/staging/payment-gateway-url
/myapp/staging/redis-host

There are two options: Plain Text parameters, and Encrypted parameters. Plain text parameters are simply like having sticky notes on your workspace. These parameters are easy to read, but not secure. Encrypted parameters are the most secure option. The AWS SSM parameter store also integrates well with other AWS services.

S3 Secret Files

Finally, let’s talk about storing secrets on Amazon S3 buckets. S3 is great for storing large amounts of data, and secrets. But this is not a secure option. Here’s a practical example in Python.

import boto3
from botocore.exceptions import ClientError

def upload_secret_file(file_name, bucket, object_name=None):
    if object_name is None:
        object_name = file_name

    s3_client = boto3.client('s3')
    try:
        response = s3_client.upload_file(
            file_name, 
            bucket, 
            object_name,
            ExtraArgs={
                'ServerSideEncryption': 'aws:kms',
                'SSEKMSKeyId': 'arn:aws:kms:region:account:key/key-id'
            }
        )
    except ClientError as e:
        return False
    return True

When using S3, you need to ensure that all buckets are configured with access control, and kept private. Otherwise, your secrets will be exposed for public access. Although S3 is a great digital storage facility, you always need to lock the door and hide the key.

Also, it is best you combine your S3 buckets with encryption and IAM policies like the one below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DenyUnencryptedObjectUploads",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::your-bucket/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "aws:kms"
                }
            }
        }
    ]
}

Choosing The Appropriate Strategy

So which way should you choose?

For startups with a simple architecture, cost efficiency is often a primary concern alongside security. AWS Parameter store provides a solution for storing non-sensitive configuration data as plain text parameters, which keeps cost minimal. For sensitive information such as API keys, and database credentials, leveraging on Parameter Store’s SecureString type will provide you with but cost efficiency, and security.

For medium to large enterprises that require a robust approach to Cloud Secrets Management, AWS Secrets Manager is the best tool of choice here. This is mainly because of it’s automatic secret rotation capabilities. This helps when you deal with a lot of micro-services requiring multiple secrets.

AWS Secrets Manager integrates with CI/CD pipelines, and has the ability implement cross-account access for shared access to same credentials across multiple organisation accounts.

Never use production secrets in development environment to prevent security breaches, and difficulty in tracking down issues. Maintain a different set of credentials for development and production, and ensure they have limited permissions, and access.

Conclusion

Managing cloud secrets is an important part of application security. By following the examples in this article, you can create a solid secrets management strategy that also scales with your application.

Being a cloud engineer comes with a lot to deal with pertaining to handling application secrets in production. However, by using the right tools, techniques, and practices, you can manage your secrets wisely, and keep your data safe at all times.

The small monthly cost of implementing proper secrets management cannot be compared to the high cost of a security breach.