AWS Lambda function integration with external service

Hello everyone, in this blog I will share how you can integrate you AWS lambda function with your application for asynchronous operation.

To explain this flow, I will share the workflow for following simple applications. You application wants to do heavy work (e.g., Process heavy excel file, Convert User uploaded video another format so that it can be consumed by another platform efficiently (browser, mobile, TV, Smartwatch etc.).

There are two ways to this.

1. Webhook (i.e., API endpoint to your backend service)
2. Queue Service/Message Broker

In this blog, I will explore option 1, as this is very simple to implement in your infrastructure.

To perform said operation, Lambda function has to do following operation after its triggered.

1. Convert assets to other format or do some heavy processing on Excel
2. Upload Assets to AWS S3/Insert data in Storage (NoSQL, SQL etc.)
3. Get Credentials to communicate with our API Endpoint & Share status with our original application, whether it was success or failure.

As you can guess from above, steps 1 and 2 are main functionality which will be handled by lambda function itself, but step 3 is where you will face problem. Problem in step 3 is how do you pass your credentials in lambda function.

Let’s understand how you can pass details in lambda function.

• Environment variables
• AWS Secret Manager

Environment variables are the easiest, but they face pose one problem is that whoever has access to lambda can see the credentials and abuse it. so, lets with go with Second option.

Let’s walk you through above architecture.

1. User uploads new video assets on AWS S3
2. AWS S3, Triggers Lambda (based on trigger configured in AWS S3)
3. Lambda invokes and reads credentials (e.g., username, password) from AWS Secret Manager (We are not integrating it CloudFormation and using API approach, as doing it with CloudFormation defeats the purpose of exposing credentials in environment.
4. AWS Lambda process its assets, by downloading S3 assets locally and converting them to other formats and uploads them to S3.
5. Lambda calls Auth service by using credentials received by AWS secret manager and receives the JWT token or Any Auth Token.
6. Next, Lambda calls the Web Application with appropriate payload using JWT token received in previous step.

Above approach looks good, but it becomes problematic when lambdas are running at scale or you have multiple lambda function (i.e., to convert assets, process CSV File, Process Excel File, Generate Reports etc.). Let’s discuss those in brief.

1. Caching JWT Tokens becomes difficult, Multiple JWT tokens needs to be created for each lambda function.
2. Each lambda function needs to have logic attached to get JWT Token and if there is any change, all lambda function code needs to be updated manually.
3. Each Lambda needs to be aware of Auth Service, In Above Diagram I have shown Auth and Web Service in same place but in real life it could be different.

To address above problem, we could modify our architecture to manage credentials at central location.

Let’s discuss the change in this architecture,

1. New Lambda function which handles the Auth flow with our Auth Service/Web Service. by pulling Credentials from AWS Secret Manager and storing JWT Token back in AWS Secret Manager. so that other lambda function don’t have to worry about Authentication.
2. AWS CloudWatch triggers this new lambda function at intervals. let’s say every 30 minutes to 24 hours depending on the expiry of JWT Tokens so that we also don’t have to generate long living JWT tokens.
3. Auth logic is removed from every other AWS lambda function.

As credentials are not exposed to other lambda function, you also get additional security major.

Let’s now discuss Pros/Cons of doing it will this way.

Pros

1. Secret credentials not exposed to unwanted users through environment variables.
2. Don’t have to call Auth API Multiple times to get authentication tokens (Token Caching), reducing load on your auth service/web service.
3. Centrally managed user credentials in AWS Secret Manager, so it would be easier to manage/rotate. and Access to it can managed through AWS Policy.
4. Lambda function is lean (don’t have to integrate authentication in every lambda)

Cons

1. Auth Lambda needs to be run initially before another lambda gets triggered so that JWT token is available.

Please do share your thoughts in comment sections.

Thanks for reading…