CI\CD for Secrets

All customers want to have their application highly available so their customers can reach their site anytime. In the AWS cloud, we use services such as Elastic Load Balancing, Auto Scaling for our applications when using server architecture. Whether it runs on an EC2, Elastic Beanstalk or ECS, or even EKS at the end we will have servers that will run our application. Recently I’ve discovered a problem. We wanted to have these applications highly available but couldn’t add more servers without solving it first. 

What do with Secrets?

Even if we do get this problem solved we need to keep the secrets safe and well encrypted so no one could access them.

In my case, the application is running on EC2 instances, behind a Load Balancer and the solution is customized for this architecture. The Application looks for a secrets file hosted on each server. 

Prerequisites for this Solution

•       Several EC2 instances behind a Load Balancer or serving the same application
•       Have all instances managed by SSM
•       Add to the Instance Role permissions to get Parameters

Solution Architecture

Solution 

For this solution, I’ve decided to keep the secret as a Secure String advanced parameter at AWS System Manager Parameter Store so the encryption is done by AWS KMS and the permission to the file will be managed by AWS IAM.

After uploading the secrets to the Parameter Store I’ve Created a CloudWatch Event event that will listen to changes on the parameter and will trigger an action if a change is made. The action is the System Manager Document which I’ve customized for this task. The document will be consumed by the System Manager Run Command and will distribute the changes to all matching servers according to the appropriate Tags.

The event Policy:

{

 "source": [

"aws.ssm"

  ],

  "detail-type": [

"Parameter Store Change"

  ],

  "detail": {

"name": [

   "/dev/env.php"

],

"operation": [

   "Create",

   "Update"

]

  }

}

Event Configuration:

Using the Document: ReplaceEnvPhp with targets according to env (dev,staging,prod).

Parameters are: and the role is the default one.

The SSM Document is a custom one defined by:

---

schemaVersion: "2.2"

description: "Replace Secrets File"

parameters:

  Parameter:

type: "String"

description: "the parameter to replace"

  Region:

type: "String"

description: "the region to work on"

default: "us-east-1"

mainSteps:

- action: "aws:runShellScript"

  name: "replaceScript"

  inputs:

runCommand:

- "/usr/local/bin/aws ssm get-parameter --name --with-decryption --region | jq --raw-output ' .[] | .Value ' > ~/file.name"

- "sudo chmod 400 ~/file.name"

- "sudo cp /your/application/path /your/application/path/save/"

- "sudo mv -f ~/file.name /your/application/path"

To have an Instance managed by SSM:

1.     Install the SSM-agent on the instance: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-ssm-agent.html
2.     Create an SSM Instance Role https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-instance-profile.html