All customers want to have their application highly available so their customers can reach their site anytime. In the AWS cloud, we use services such as Elastic Load Balancing, Auto Scaling for our applications when using server architecture. Whether it runs on an EC2, Elastic Beanstalk or ECS, or even EKS at the end we will have servers that will run our application. Recently I’ve discovered a problem. We wanted to have these applications highly available but couldn’t add more servers without solving it first.
Even if we do get this problem solved we need to keep the secrets safe and well encrypted so no one could access them.
In my case, the application is running on EC2 instances, behind a Load Balancer and the solution is customized for this architecture. The Application looks for a secrets file hosted on each server.
For this solution, I’ve decided to keep the secret as a Secure String advanced parameter at AWS System Manager Parameter Store so the encryption is done by AWS KMS and the permission to the file will be managed by AWS IAM.
After uploading the secrets to the Parameter Store I’ve Created a CloudWatch Event event that will listen to changes on the parameter and will trigger an action if a change is made. The action is the System Manager Document which I’ve customized for this task. The document will be consumed by the System Manager Run Command and will distribute the changes to all matching servers according to the appropriate Tags.
"Parameter Store Change"
Using the Document: ReplaceEnvPhp with targets according to env (dev,staging,prod).
Parameters are: and the role is the default one.
description: "Replace Secrets File"
description: "the parameter to replace"
description: "the region to work on"
- action: "aws:runShellScript"
- "/usr/local/bin/aws ssm get-parameter --name --with-decryption --region | jq --raw-output ' . | .Value ' > ~/file.name"
- "sudo chmod 400 ~/file.name"
- "sudo cp /your/application/path /your/application/path/save/"
- "sudo mv -f ~/file.name /your/application/path"
Don't use secrets in code and don't sync them at source control. Use parameter store or secrets manager to keep your secret.
Always rotate your secrets according to a known schedule. When you do, keep in mind to update them in all your servers