I was wondering if I could schedule simple bash scripts using AWS Fargate for some trivial batches operations.

To be completely honest, It is also an excuse to learn more about AWS Fargate, and to convert a legacy bash script based on EC2 Spot instance to a container world.

In this post, we will see how to schedule a bash script job once a day. To do so, we will deploy the corresponding AWS infrastructure (even if it’s serverless, yes :wink:) using Terraform.

The code repository is available here: serverless-jobs-using-fargate

TL;DR

  • git clone https://github.com/z0ph/serverless-jobs-using-fargate.git
  • Prepare your Docker image (using Dockerfile)
  • Adapt the variables.tf, and variables in Makefile to your needs
  • Run make plan
  • Run make deploy
  • Run make build-docker
  • Take a nap, enjoy :cocktail:

Rest

Deployment using Terraform

As a pre-requirement, you will need to edit variables.tf file and set your own values corresponding to your needs, with special focus on XXX :dizzy_face:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

variable "aws_region" {
  default = "eu-west-1"
  description = "AWS Region"
}

variable "env" {
  default     = "dev"
  description = "Environment"
}

variable "project" {
  default     = "no-project-name"
  description = "Project Name"
}

variable "description" {
  default     = "empty-project-description"
  description = "Project Description"
}

variable "artifacts_bucket" {
  default     = "no-artifact-bucket-defined"
  description = "Artifacts Bucket Name"
}

variable "ecs_event_role" {
  default     = "XXX"
  description = "IAM Role used for CloudWatch"
}

variable "ecs_taskexec_role" {
  default     = "XXX"
  description = "IAM Role used for Task Execution"
}

variable "subnets" {
  type        = list(string)
  default     = ["XXX"]
  description = "Subnets used for Fargate Containers"
}

variable "security_groups" {
  type        = list(string)
  default     = ["XXX"]
  description = "Security Groups used for Fargate"
}

variable "schedule" {
  default     = "rate(XXX hours)"
  description = "Schedule for your job"
}

variable "assign_public_ip" {
  default     = "false"
  description = "Set public IP on Fargate Container"
}

variable "ecs_cpu_units" {
  default     = "1024"
  description = "Container: Number of CPU Units"
}

variable "ecs_memory" {
  default     = "2048"
  description = "Container: Memory in MB"
}

variable "docker_image_arn" {
  default     = "XXXXXXXXXXXX.dkr.ecr.xxxxx.amazonaws.com/xxxx:lastone"
  description = "Arn of the Docker Image"
}

First, we will create an Elastic Container Registry (ECR) to host our newly built Docker image.

1
2
3
4
5
6
7

resource "aws_ecr_repository" "ecr" {
  name = "${var.project}-ecr-${var.env}"

  tags = {
    Project = "${var.project}"
  }
}

Second, you will find in ecs.tf the corresponding AWS ECS cluster, task definition json template file.

In this file, you’ll find the target CPU, memory, docker image, etc… of your container(s). All are using vars mapping, so you just need to update variables.tf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

resource "aws_ecs_cluster" "ecs_cluster" {
  name = "${var.project}_ecs_cluster_${var.env}"

  tags = {
    Project = "${var.project}"
  }
}

data "template_file" "task" {
  template = "${file("./automation/tf-fargate/tasks/task_definition.json")}"
  vars = {
    name = "${var.project}"
    aws_region = "${var.aws_region}"
    docker_image_arn = "${var.docker_image}"
  }
}

resource "aws_ecs_task_definition" "task_definition" {
  family                   = "${var.project}_task_definition_${var.env}"
  container_definitions    = "${data.template_file.task.rendered}"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "${var.ecs_cpu_units}"
  memory                   = "${var.ecs_memory}"
  execution_role_arn       = "${var.ecs_taskexec_role}"
  task_role_arn            = "${aws_iam_role.ecs_role.arn}"

  tags = {
    Project = "${var.project}"
  }
}

Third, the schedule logic will land in the cloudwatch.tf file because we are using CloudWatch Events. You can use either rate(X minutes|hours|days) OR cron(****)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

resource "aws_cloudwatch_event_rule" "cw_run_task" {
  name                = "${var.project}_run_task_${var.env}"
  description         = "Run ${var.project} on ${var.schedule}"
  schedule_expression = "${var.schedule}"
}

resource "aws_cloudwatch_event_target" "cw_event_target" {
  target_id = "${var.project}_event_target_${var.env}"
  arn = "${aws_ecs_cluster.ecs_cluster.arn}"
  rule = "${aws_cloudwatch_event_rule.cw_run_task.name}"
  role_arn = "${var.ecs_event_role}"
  ecs_target {
      launch_type           = "FARGATE"
      platform_version      = "LATEST"
      task_definition_arn   = "${aws_ecs_task_definition.task_definition.arn}"
      network_configuration {
        subnets             = "${var.subnets}"
        security_groups     = "${var.security_groups}"
        assign_public_ip    = "${var.assign_public_ip}"
      }
  }
}

Push your container image to ECR

I’m using custom alpine image with awscli pre-installed.

Using make build-docker it will run the following commands under the hood:

1
2
3
4

  $ aws ecr get-login --region $(AWS_REGION)
  $ docker build -t fargate-image .
  $ docker tag fargate-image:latest $(ECR)
  $ docker push $(ECR)

Tada

FinOps :money_with_wings:

For Amazon ECS, AWS Fargate pricing is calculated based on the vCPU and memory resources used from the time you start to download your container image (docker pull) until the Amazon ECS Task* terminates, rounded up to the nearest second. A minimum charge of 1 minute applies.

Pricing is per second with a 1-minute minimum. Duration is calculated from the time you start to download your container image (docker pull) until the Task terminates, rounded up to the nearest second.

Resource type Price (eu-west-1)
per vCPU per hour $0.04048
per GB per hour $0.004445

Pricing Example

Pricing for US East (N. Virginia) Region

For example, your service uses 10 ECS Tasks running for 1 hour (3600 seconds) every day for a month (30 days) where each ECS Task uses 0.25 vCPU and 1GB memory.

Monthly CPU charges

Total vCPU charges = # of Tasks x # vCPUs x price per CPU-second x CPU duration per day (seconds) x # of days Total vCPU charges = 10 x 0.25 x 0.000011244 x 3600 x 30 = $3.04

Monthly memory charges

Total memory charges = # of Tasks x memory in GB x price per GB x memory duration per day (seconds) x # of days Total memory charges = 10 x 1 x 0.000001235 x 3600 x 30 = $1.33

Monthly Fargate compute charges

Monthly Fargate compute charges = monthly CPU charges + monthly memory charges Monthly Fargate compute charges = $3.04 + $1.33 = $4.37

Source: https://aws.amazon.com/fargate/pricing/?nc=sn&loc=2

Ref:

That’s all folks!

zoph.