CI\CD for Secrets

All customers want to have their application highly available so their customers can reach their site anytime. In the AWS cloud, we use services such as Elastic Load Balancing and Auto Scaling for our applications when using server architecture.

Whether it runs on an EC2 instance, Elastic Beanstalk, ECS, or even EKS, in the end we'll have servers that will run our application. Recently we’ve discovered a problem: we wanted to have these applications highly available but couldn’t add more servers without solving it first. 

What to do with Secrets?

Even if we do solve this problem, we need to keep the secrets safe and well-encrypted so no one could access them.

In this case, the application is running on EC2 instances, behind a Load Balancer and the solution is customized for this architecture. The application looks for a 'secrets' file hosted on each server. 

Prerequisites for this Solution

•       Several EC2 instances behind a Load Balancer or serving the same application
•       Have all instances managed by AWS Systems Manager (SSM
•       Add to the Instance Role permissions to get Parameters

Solution Architecture

Solution 

For this solution, the decision was made to keep the secrets as a Secure String advanced parameter in the AWS System Manager Parameter Store, so the encryption is done by AWS KMS, and the permission to the file will be managed by AWS IAM.

After uploading the secrets to the Parameter Store, a CloudWatch Event was created. This event will listen to changes on the parameter and trigger an action if a change is made. The action is the System Manager Document customized for this task. The document will be consumed by the System Manager Run Command and will distribute the changes to all matching servers according to the appropriate Tags.

The event Policy:

{

 "source": [

"aws.ssm"

  ],

  "detail-type": [

"Parameter Store Change"

  ],

  "detail": {

"name": [

   "/dev/env.php"

],

"operation": [

   "Create",

   "Update"

]

  }

}

Event Configuration:

Using the Document: ReplaceEnvPhp with targets according to env (dev,staging,prod).

Parameters are: and the role is the default one.

The SSM Document is a custom one defined by:

---

schemaVersion: "2.2"

description: "Replace Secrets File"

parameters:

  Parameter:

type: "String"

description: "the parameter to replace"

  Region:

type: "String"

description: "the region to work on"

default: "us-east-1"

mainSteps:

- action: "aws:runShellScript"

  name: "replaceScript"

  inputs:

runCommand:

- "/usr/local/bin/aws ssm get-parameter --name --with-decryption --region | jq --raw-output ' .[] | .Value ' > ~/file.name"

- "sudo chmod 400 ~/file.name"

- "sudo cp /your/application/path /your/application/path/save/"

- "sudo mv -f ~/file.name /your/application/path"

To have an Instance managed by SSM:

1.     Install the SSM-agent on the instance: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-ssm-agent.html
2.     Create an SSM Instance Role https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-instance-profile.html