Use Instance Store With AWS Elastic Container Storage

Many EC2 instance types come with instance attached storage (Instance Store) which can provide a fast local storage that is faster than using an EBS volume. If your using the Amazon ECS-optimized AMI (Amazon Linux 1), it’s instance storage is a secondary EBS volume that is used for storing docker containers and volumes. If your launching it on an EC2 with instance store, it is ignored and only the one EBS volume is used.

Update July 3, 2019: Added details for Amazon Linux 2

Amazon Linux 1: Amazon ECS-optimized AMI

The Amazon Linux 1 based version of the ECS AMI uses the Device Mapper storage driver for container storage, which uses a thin-pool volume (part of LVM).
Here is a simplistic cloud-init script that detects the attached SSD and NVMe SSD’s and adds them to the LVM volume group. Just launch your EC2 instance with the following user data or download the script from this gist if you’ve got a more complex init script already.

Note: I have not done thorough performance testing but have noticed a slight increase in IO transfer times when configured this way on the newer generation (i.e. c5d, r5d etc) types.

EC2 User Data

#cloud-boothook
set -ex


# find ephemeral devices using the metadata service
# note: this will block loading until the metadata service is ready
find_ephemeral_devices() {
    echo "Querying metadata instance store for ephemeral volumes" <&2
    local DEVICE_NOT_FOUND=0
    for d in $(curl -s "http://169.254.169.254/latest/meta-data/block-device-mapping/" | grep ephemeral); do
       NAME=$(curl -s "http://169.254.169.254/latest/meta-data/block-device-mapping/$d")
       DEVICE_NAME="/dev/$NAME"
       echo "Detected ephemeral device $d corresponding to $DEVICE_NAME" <&2
       # if block device (-b)
       if [[ -b "$DEVICE_NAME" ]]; then
          echo "$DEVICE_NAME"
          DEVICE_NOT_FOUND=0
       else
          echo "Couldn't find device $DEVICE_NAME" <&2
          DEVICE_NOT_FOUND=1
       fi
    done
    return ${DEVICE_NOT_FOUND}
}

# on nvme instances the device names don't match what AWS reports, so just filter by model name
find_nvme_ephemeral() {
  lsblk -o KNAME,MODEL | grep "Amazon EC2 NVMe Instance Storage" | awk '{ print "/dev/"$1 }'
}

METADATA_DEVICES=$(find_ephemeral_devices)
NVME_DEVICES=$(find_nvme_ephemeral)

DEVS=/dev/xvdcz
[[ ! -z "$NVME_DEVICES" ]] && DEVS="$NVME_DEVICES $DEVS"
[[ ! -z "$METADATA_DEVICES" ]] && DEVS="$METADATA_DEVICES $DEVS"

echo "Updating docker storage devices to '$DEVS'"
sed -i -e "s;DEVS=.*;DEVS=\"${DEVS}\";g"  /etc/sysconfig/docker-storage-setup

Amazon Linux 2: Amazon ECS-optimized Amazon Linux 2 AMI

The setup for the Amazon Linux 2 is a little different as it is configured to use a different storage driver, it instead uses the overlay2 storage driver which stores files directly on the underlying filesystem. The AMI is configured to use just the root EBS, if you require more storage then you just launch with or extend out the root EBS. As a result of this, adding extra volumes isn’t as simple. A somewhat inconsistent approach is to mount part of the docker storage system onto the Instance Storage if it exists and to use the root EBS if not.

The docker documentation mentions:

Writing into a container’s writable layer requires a storage driver to manage the filesystem. The storage driver provides a union filesystem, using the Linux kernel. This extra abstraction reduces performance as compared to using data volumes, which write directly to the host filesystem.

Implying that using volumes can be more performant when doing IO to disk. Following this guidance, this script creates a logical volume (stripped across all available devices to increase throughput) mounted at the location where docker stores it’s volumes: /var/lib/docker/volumes

Here is the basic user data to bootstrap the storage setup, with the main work done by docker-amzn-2-add-ephemeral.sh. This code is available on the same gist, or downloadable from the s3 location:
s3://devgrok-blog-files/amazon-ecs/ecs-init-scripts/

EC2 User Data

Content-Type: multipart/mixed; boundary="==BOUNDARY=="
MIME-Version: 1.0

--==BOUNDARY==
MIME-Version: 1.0
Content-Type: text/cloud-config

mounts:
 - [ ephemeral0 ]


--==BOUNDARY==
MIME-Version: 1.0
Content-Type: text/cloud-boothook; charset="us-ascii"

#!/bin/bash +x
# an early running script to setup scripts that get triggered after the ECS init service is started
#
# Write the bootstrap script to /opt/devgrok and run it - so output definitely logged
mkdir -p /opt/devgrok
cat > /opt/devgrok/bootstrap-devgrok-init.sh <<- 'EOF'
#!/bin/bash -x
mkdir -p /opt/devgrok/ecs-init-scripts/
yum install -y aws-cli
aws s3 sync s3://devgrok-blog-files/amazon-ecs/ecs-init-scripts/ /opt/devgrok/ecs-init-scripts/
chmod +x /opt/devgrok/ecs-init-scripts/*.sh

# setup aws cloudwatch
/opt/devgrok/ecs-init-scripts/install-cw-agent.sh
[[ -e "/etc/systemd/system/bootstrap-awslogs.service" ]] || \
  cp /opt/devgrok/ecs-init-scripts/systemd-bootstrap-awslogs.service /etc/systemd/system/bootstrap-awslogs.service

# setup storage
/opt/devgrok/ecs-init-scripts/docker-amzn-2-add-ephemeral.sh

EOF


chmod +x /opt/devgrok/bootstrap-devgrok-init.sh
/opt/devgrok/bootstrap-devgrok-init.sh 2>&1 | tee -a /var/log/bootstrap-devgrok-init.log


--==BOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/sh
# this runs later in the boot process
systemctl daemon-reload
systemctl enable bootstrap-awslogs.service
# the no-block is important as the ecs-init service may be dependant on the cloud-init boot process finishing first
systemctl start bootstrap-awslogs.service --no-block

--==BOUNDARY==--

Comments

Post a Comment