Installing the new Cloudwatch Agent for ECS

If you want to gather log files on your ECS hosts running a Amazon ECS-optimized AMI, then there’s instructions on how to install the Previous CloudWatch Logs Agent using cloud-init and some user data. The old Logs Agent is still supported but the new Unified CloudWatch Agent is recommended as it is touted to be faster but more importantly, allows easy collection of instance metrics.

To install it, simply launch an EC2 instance using one of the Amazon ECS AMI’s, and put this in the user-data section (expand the advanced configuration of the launch wizard). Once the EC2 instance is started up, just simply create an AMI from it, shutdown the EC2 instance and use this new AMI for you ECS clusters or for your Batch Compute Environments.

With a bit of modification, this script could be used to install and configure it on the launch of each ECS instance, but having a pre-baked AMI is a bit cleaner.

This script also increase the default docker container size to 100GB, using the ECS Agent startup options. If its not needed, just remove the section from the start of the first “—==BOUNDARY==” to the start of, but not including, the next “—==BOUNDARY==”.

It is also available at this gist.

Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version: 1.0

--==BOUNDARY==
MIME-Version: 1.0
Content-Type: text/cloud-boothook; charset="us-ascii"

#!/bin/bash +x
# optionally set ECS agent options (increase base container size)
cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=100G"' >> /etc/sysconfig/docker

--==BOUNDARY==
MIME-Version: 1.0
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
yum install -y unzip jq
mkdir -p /tmp/cw-agent-install
cd /tmp/cw-agent-install
curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip
unzip AmazonCloudWatchAgent.zip
./install.sh
rm -rf /tmp/cw-agent-install

# Inject the CloudWatch Logs configuration file contents
cat > /opt/aws/amazon-cloudwatch-agent/etc/config.json <<- EOF
{
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/dmesg",
            "log_group_name": "ecs/var/log/dmesg",
            "timezone": "UTC"
          },
          {
            "file_path": "/var/log/messages",
            "log_group_name": "ecs/var/log/messages",
            "timezone": "UTC",
            "timestamp_format": "%b %d %H:%M:%S"
          },
          {
            "file_path": "/var/log/docker",
            "log_group_name": "ecs/var/log/docker",
            "timezone": "UTC",
            "timestamp_format": "%Y-%m-%dT%H:%M:%S.%f"
          },
          {
            "file_path": "/var/log/ecs/ecs-init.log",
            "log_group_name": "ecs/var/log/ecs/ecs-init.log",
            "timezone": "UTC",
            "timestamp_format": "%Y-%m-%dT%H:%M:%SZ"
          },
          {
            "file_path": "/var/log/ecs/ecs-agent.log.*",
            "log_group_name": "ecs/var/log/ecs/ecs-agent.log",
            "timezone": "UTC",
            "timestamp_format": "%Y-%m-%dT%H:%M:%SZ"
          },
          {
            "file_path": "/var/log/ecs/audit.log.*",
            "log_group_name": "ecs/var/log/ecs/audit.log",
            "timezone": "UTC",
            "timestamp_format": "%Y-%m-%dT%H:%M:%SZ"
          }
        ]
      }
    },
    "log_stream_name": "{cluster}/{container_instance_id}"
  }
}
EOF

--==BOUNDARY==
MIME-Version: 1.0
Content-Type: text/upstart-job; charset="us-ascii"

#upstart-job
description "Configure and start CloudWatch Logs agent on Amazon ECS container instance"
author "Amazon Web Services"
start on started ecs

script
 exec 2>>/var/log/ecs/cloudwatch-logs-start.log
 set -x
 
 until curl -s http://localhost:51678/v1/metadata
 do
  sleep 1 
 done

 # Grab the cluster and container instance ARN from instance metadata
 cluster=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .Cluster')
 container_instance_id=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F/ '{print $2}' )
 
 # Replace the cluster name and container instance ID placeholders with the actual values
 sed -i -e "s/{cluster}/$cluster/g" /opt/aws/amazon-cloudwatch-agent/etc/config.json
 sed -i -e "s/{container_instance_id}/$container_instance_id/g" /opt/aws/amazon-cloudwatch-agent/etc/config.json
 
 /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/config.json -s
end script
--==BOUNDARY==--