Installing the new Cloudwatch Agent for ECS
If you want to gather log files on your ECS hosts running a Amazon ECS-optimized AMI, then there’s instructions on how to install the Previous CloudWatch Logs Agent using cloud-init and some user data. The old Logs Agent is still supported but the new Unified CloudWatch Agent is recommended as it is touted to be faster but more importantly, allows easy collection of instance metrics.
To install it, simply launch an EC2 instance using one of the Amazon ECS AMI’s, and put this in the user-data section (expand the advanced configuration of the launch wizard). Once the EC2 instance is started up, just simply create an AMI from it, shutdown the EC2 instance and use this new AMI for you ECS clusters or for your Batch Compute Environments.
With a bit of modification, this script could be used to install and configure it on the launch of each ECS instance, but having a pre-baked AMI is a bit cleaner.
This script also increase the default docker container size to 100GB, using the ECS Agent startup options. If its not needed, just remove the section from the start of the first “—==BOUNDARY==” to the start of, but not including, the next “—==BOUNDARY==”.
It is also available at this gist.
Content-Type: multipart/mixed; boundary="==BOUNDARY==" MIME-Version: 1.0 --==BOUNDARY== MIME-Version: 1.0 Content-Type: text/cloud-boothook; charset="us-ascii" #!/bin/bash +x # optionally set ECS agent options (increase base container size) cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=100G"' >> /etc/sysconfig/docker --==BOUNDARY== MIME-Version: 1.0 Content-Type: text/x-shellscript; charset="us-ascii" #!/bin/bash yum install -y unzip jq mkdir -p /tmp/cw-agent-install cd /tmp/cw-agent-install curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip unzip AmazonCloudWatchAgent.zip ./install.sh rm -rf /tmp/cw-agent-install # Inject the CloudWatch Logs configuration file contents cat > /opt/aws/amazon-cloudwatch-agent/etc/config.json <<- EOF { "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "/var/log/dmesg", "log_group_name": "ecs/var/log/dmesg", "timezone": "UTC" }, { "file_path": "/var/log/messages", "log_group_name": "ecs/var/log/messages", "timezone": "UTC", "timestamp_format": "%b %d %H:%M:%S" }, { "file_path": "/var/log/docker", "log_group_name": "ecs/var/log/docker", "timezone": "UTC", "timestamp_format": "%Y-%m-%dT%H:%M:%S.%f" }, { "file_path": "/var/log/ecs/ecs-init.log", "log_group_name": "ecs/var/log/ecs/ecs-init.log", "timezone": "UTC", "timestamp_format": "%Y-%m-%dT%H:%M:%SZ" }, { "file_path": "/var/log/ecs/ecs-agent.log.*", "log_group_name": "ecs/var/log/ecs/ecs-agent.log", "timezone": "UTC", "timestamp_format": "%Y-%m-%dT%H:%M:%SZ" }, { "file_path": "/var/log/ecs/audit.log.*", "log_group_name": "ecs/var/log/ecs/audit.log", "timezone": "UTC", "timestamp_format": "%Y-%m-%dT%H:%M:%SZ" } ] } }, "log_stream_name": "{cluster}/{container_instance_id}" } } EOF --==BOUNDARY== MIME-Version: 1.0 Content-Type: text/upstart-job; charset="us-ascii" #upstart-job description "Configure and start CloudWatch Logs agent on Amazon ECS container instance" author "Amazon Web Services" start on started ecs script exec 2>>/var/log/ecs/cloudwatch-logs-start.log set -x until curl -s http://localhost:51678/v1/metadata do sleep 1 done # Grab the cluster and container instance ARN from instance metadata cluster=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .Cluster') container_instance_id=$(curl -s http://localhost:51678/v1/metadata | jq -r '. | .ContainerInstanceArn' | awk -F/ '{print $2}' ) # Replace the cluster name and container instance ID placeholders with the actual values sed -i -e "s/{cluster}/$cluster/g" /opt/aws/amazon-cloudwatch-agent/etc/config.json sed -i -e "s/{container_instance_id}/$container_instance_id/g" /opt/aws/amazon-cloudwatch-agent/etc/config.json /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/config.json -s end script --==BOUNDARY==--
Comments
Post a Comment