Home > Uncategorized > Running a scheduled task on #AWS – A workaround for the 15 minute #Lambda limit

Running a scheduled task on #AWS – A workaround for the 15 minute #Lambda limit

Let’s start with a bit of context. You want to run a process every day that could last 30 minutes, perhaps you’re importing data from an external source to a database. You can’t use Lambda, because the limit is 15 minutes, and it’s wasteful to use an EC2 instance because for 99% of the time the server will sit idle.

So, containers to the rescue. This is where you firstly write your code, and containerize it. Make sure it runs locally, as expected. Then you push the docker image to ECR. I followed most of the steps in this blog post:

https://towardsdatascience.com/deploying-a-docker-container-with-ecs-and-fargate-7b0cbc9cd608

The difference being, in my case, the application had a finite run time – i.e. do a task, then exit (not a web server).

Which means, that after I created my task definition, I then went to the Scheduled Task tab as shown below;

Where I could create a scheduled task that I can set to run every day, or minute, or whatever you need (the minium is 1 minute).

The container will run until it has finished it’s workload, so there is no 15 minute limit, like on AWS lambda.

Categories: Uncategorized
  1. No comments yet.
  1. No trackbacks yet.

Leave a comment