Using AWS Lambda, CloudWatch and Secrets Manager to schedule secure hits of a service endpoint

The problem

Let’s imagine we have a Service that has a health check endpoint. Upon hitting the endpoint our service checks if all our services are running correctly. We want this rule to run once every 24 hours. Obviously we can’t / don’t want to rely on someone to manually hit the endpoint once every 24 hours, so we need to automate the process. We also require the added caveat that only privileged users (permissioned admins etc.) can trigger the health check, so it doesn’t get spammed or attacked by anyone who is aware of what the endpoint might be. So only a logged in user has permission to trigger the health check. We could set something up like:

setInterval(() => hitHealthCheck(), 1000 x 60 x60x 24);

which will hit our healthCheck every 24 hours. But we have no control over the time, and we also have the added complication (particularly if your application is being hosted somewhere expensive to run) of having the application constantly ‘Hot’. We are assuming we can attach a token to our headers in our request to pass our permission checker, and that our api lives at api.myApp.com.

The solution

We can use AWS Cloudwatch to manage our scheduling of the health check, and we can host the function that is being used to call the health check inside an AWS Lambda function. The theory behind this is we can use the magic of Lambda functions to only spin up and down when they are needed to, we can use the schedule functionality of Cloudwatch to call our Lambda function for us, and we can also use the capability of AWS Secret Manager to ensure that the Lambda function does not expose usernames or passwords (very important!). The steps outlined below are an outline how to get this set up and running, quickly and (relatively) pain free! In this we assume you have a system which allows you to add a user with a specified username and password to your system, with the relevant privileges!

Adding a new username and password to the Secrets Manager

In Secrets Manager in the AWS console, we will need to add a new username and password. To do this select Store a new secret. Because we are storing some credentials to log in to your own system you need to select Other type of secrets and then add a key value pair for both username and password.

(please use a random generator to get your string values!)

Give it a name and a description (optional)

For this example we aren’t going to rotate our secret (although you probably should do if you can!) So select next, then store to store your new secret.

That’s it, our username and password are now stored successfully on our AWS Secret Manager (we will access them later).

Adding our user to the database

I’m afraid I can’t provide a lot of advice on how to do this for any particular system. The desired outcome of this step is to have a user in your own system that has a username and password as specified in your new secret, which has the privileges necessary to hit your endpoint.

Lambda Function

Now we are going to add our Lambda function inside AWS. Go to Lambda in the AWS console and select “Create Function”. Decide on a name (e.g. prod/healthCheck . From here, using our example you will need to keep with the following while creating:

1. Your Lambda is running on node and you haven’t messed with the Handler or Environment, just the code inside index.js
2. You may need to increase the timeout from 3 seconds to something higher
3. You will need to check that the Lambda role IAM permissions have access to Secrets Manager
4. If you are dealing with PROD you may need to be aware of any of your VPC restrictions.

The Lambda function contains a lot of the magic needed to tie everything together:

First we have our imports, and our setup to retrieve the information from the Secret Manager. These functions hit AWS and return a promise containing the username and password we have securely stored. We create them to be asynchronously called so we can await them going forward:

The next two functions are basic functions to make a https request to our endpoints, one to login, and one to fire the health check.

Our asynchronous login function

Our asynchronous health check function

Then you finally have the Lambda function itself, which AWS uses to fetch the secret, log in, and finally trigger our health check. If we have an error in our secret fetch we log it to our console.

Our AWS Lambda function

We can test this in AWS to make sure it works correctly manually using the TEST button. All going well you should have a log below stating that you logged in successfully and a response from your health check service.

Set up schedule using CloudWatch

The final step is to add a rule to CloudWatch to set our Lambda function to run every 24 hours

• Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
• In the navigation pane, choose Events, Create rule.
• For Event Source, do the following:
• Choose Schedule.
• We are using a Cron job and in this example setting it to run at 3am every morning
• For Targets, choose Add target, Lambda function.
• For Function, select the Lambda function that you created.
• Choose Configure details.

Configure our schedule

• For Rule definition, type a name and description for the rule.
• Choose Create rule.

Name our schedule

All finished!

Now you should be able to check your service logs and see that your health check is being triggered at 3am every morning!