Posts

Use Instance Store With AWS Elastic Container Storage

Many EC2 instance types come with instance attached storage (Instance Store) which can provide a fast local storage that is faster than using an EBS volume. If your using the Amazon ECS-optimized AMI (Amazon Linux 1), it’s instance storage is a secondary EBS volume that is used for storing docker containers and volumes. If your launching it on an EC2 with instance store, it is ignored and only the one EBS volume is used. So here is a simplistic cloud-init script that detects the attached SSD and NVMe SSD’s and adds them to the LVM volume group.
Note: I have not done thorough performance testing but have noticed a slight increase in IO times when configured this way on the newer generation (i.e. c5d, r5d etc) types.Just launch your EC2 instance with the following user data or download the script from this gist if you’ve got a more complex init script already.EC2 User Data#cloud-boothook set -ex # find ephemeral devices using the metadata service # note: this will block loading until t…

Create a Private Microservice Using an Application Load Balancer

Image
Previously if you wanted to create an REST API powered by a lambda you only had one choice: API Gateway. This has a few limitations notably they’re always public so you need to use IAM or similar to lock it down and you can only use a custom domain name once globally, meaning no duplicating the implementations across multiple accounts with the same host endpoint. AWS recently announced another way to create a RESTful endpoint for Lambda’s: Application Load Balancers.

Using an async iterator on Node.js + S3

There isn't support for async iterators (for await...of) in Node.js v8.9 which is AWS Lambda's runtime. It's shame, as it’s a great feature that allows you to iterate over an iterable that returns as result asynchronously, i.e. retrieving another page from a database, using a compact for loop that feels synchronous but under the covers is actually done asynchronously.
Which means, if you want to use a library that written specifically to use it (e.g. Amazon DynamoDB QueryPaginator), you have to use an even more verbose syntax. However with a bit of re-purposing you can use a generator function that returns a Promise and if you await each promise given in the loop it will behave like an async iterator.

Auto partition secondary EBS on CentOS 7

If you add an additional blank EBS volume to a CentOS 7 EC2 instance, it won’t auto-partition in the same way as the primary volume gets resized on launch. Additionally if your baking an image you will certainly encounter problems with it not automounting on different instance types and even manually trying to mount it can give misleading error messages.

Using a S3 Hive Metastore with EMR

When configuring Hive to use EMRFS (i.e. s3://) instead of using the implied HDFS cluster storage of the metastore, which is vital if you want to have a persistent metastore that can survive clusters being destroyed and recreated; you might encounter this message: Access Denied (Service: Amazon S3; Status Code: 403;...).

Resizing root partition on CentOS 6 in the cloud

If your unfortunate enough to still be using CentOS 6 in AWS than more than likely be stumped why the root partition isn’t resizing like most other distros you launch.

Installing the new Cloudwatch Agent for ECS

If you want to gather log files on your ECS hosts running a Amazon ECS-optimized AMI, then there’s instructions on how to install the Previous CloudWatch Logs Agent using cloud-init and some user data. The old Logs Agent is still supported but the new Unified CloudWatch Agent is recommended as it is touted to be faster but more importantly, allows easy collection of instance metrics. To install it, simply launch an EC2 instance using one of the Amazon ECS AMI’s, and put this in the user-data section (expand the advanced configuration of the launch wizard). Once the EC2 instance is started up, just simply create an AMI from it, shutdown the EC2 instance and use this new AMI for you ECS clusters or for your Batch Compute Environments. With a bit of modification, this script could be used to install and configure it on the launch of each ECS instance, but having a pre-baked AMI is a bit cleaner.
This script also increase the default docker container size to 100GB, using the ECS Agent st…