Create a Private Microservice Using an Application Load Balancer


Previously if you wanted to create an REST API powered by a lambda you only had one choice: API Gateway. This has a few limitations notably they’re always public so you need to use IAM or similar to lock it down and you can only use a custom domain name once globally, meaning no duplicating the implementations across multiple accounts with the same host endpoint. AWS recently announced another way to create a RESTful endpoint for Lambda’s: Application Load Balancers.

Last month, at the company I work for, we finished our migration from HipChat to Slack. Part of which included migrated a lot of bespoke applications that no one was overly familiar with, to deal with these we essentially had 2 choices:
  1. Replace the HipChat calls with Calls to Slack – there would be a fair amount of work with plenty of unknowns, one of which was the remaining lifetime of the application
  2. Redirect the calls to HipChat instead to an adaptor/proxy endpoint that converts the message and calls the Slack API
Option 2 seemed like the simplest option whui Option 1 would have been the desired end state if we wanted to keep the applications long term. My initial solution was a serverless designing using API Gateway and Lambdas, modifying all the hardcoded references of "api.hipchat.com" to "hipchat-proxy.private.example.com". To secure this endpoint to stop anyone from spamming into our Slack account, I was going to create a whitelist of HipChat tokens (essentially mimicking how HipChat was secured). We had multiple AWS accounts and I wanted each account to have a version of the Lambda so would deploy a CloudFormation stack in each, creating a custom domain name mapping to the API Gateway. I quickly hit a snag, which is:

You can only use host name once globally for API Gateway custom domain names

I’ve been told that API Gateway & Custom Domain Names are implemented by AWS using CloudFront, which explains where the restriction comes from: CloudFront only allows one domain name to be used globally, regardless whether its same or different accounts. I thought that was the end to this simplistic design until I discovered Application Load Balancers and Lambda integration…

Using Lambda and LoadBalancers

In the scenario described above, the legacy applications can have the api endpoint configured to a domain name and then each AWS account would use a private dns entry to point to it’s application load balancer. Since ALB’s can be either public or private, you can create a private one that only has a private IP address and is inaccessible from outside your VPC. If you trust the servers running in it, you can leverage this the way of securing your lambda by using security groups. This will limit access to EC2 instances that are included in the security group attached to the ALB.

Below is a very simple implementation of the HipChat proxy and is meant for illustrative purposes. For better security, the slack token should be stored in AWS Security Manager – the lambda would call it to get the token and not store it locally or in the templates. Of course to fully secure it, you should add token filtering to whitelist only tokens that you know to be relayed to Slack.

There isn’t support yet in a lot of frameworks (including AWS’ SAM cli), so a fair bit of configuration has to be done manually. I patched a router package to make it work (see requirements.txt), using it allows for a nice mapping of URL paths to functions.
To setup the lambda:
  1. First package and deploy the stack (see build script).
  2. Next create an internal application load, during the wizard you can configure the lamdba as the default target group or you can use the linked template to create an ALB
  3. If the lambda wasn’t linked in the previous step, go to the AWS console, navigate to Lambdas and select the lambda just deployed, then follow the steps on the AWS blog
  4. Make any modifications necessary to the security group used to create the ALB in step 2, to allow access for your EC2 etc, to access it on port 80 (or 443 if your using HTTPS). A note on HTTPS and using ACM, because the dns record is private, you’ll need to be able to create DNS records in the parent domain in order to validate your right to create certificates for that domain.
  5. Sign onto your host a run curl –v http://chat.private.example.com (replacing example.com with your domain).

What are private hosted zones?

AWS Route53 lets you create Private Hosted Zones, a feature which allows you to create a DNS subdomain that doesn’t resolve publicly outside your VPC. This gives you the ability to define DNS records which are not only private but are tied to specific VPC(s).

If you define private.example.com as a private hosted zone,  and have no entries for private.example.com or it’s subdomain in the public hosted zone example.com, then any outside request for those domains won’t resolve. As the private zone is linked to a VPC, only those VPC’s know about it and hence can resolve them. It also means is that if you have multiple VPC’s (which includes multiple accounts), then each one can resolve these DNS records to different IP addresses etc.

Download the code

You can download the serverless template, proxy python script and a build script HERE.

The lambda

This is the implementation of hipchat_proxy.py :
import json
import logging
import os

from lambdarest import lambda_handler
from slackclient import SlackClient

logger = logging.getLogger()
logger.setLevel(logging.INFO)

sc = SlackClient(os.environ['SLACK_TOKEN'])


def generic_handler(event, room_id_or_name):
    hipchat_body = json.loads(event['body'])
    msg_text = hipchat_body['message']
    try:
        resp = sc.api_call("chat.postMessage", channel="test_room", text=msg_text)
        if resp['ok']:
            logger.info("Success calling slack")
            return {"statusCode": 204, "body": ""}
        else:
            logger.error("Slack response %s", resp)
            return {"statusCode": 500, "body": "Slack response: " + resp}

    except Exception as err:
        logger.error("Caught error calling slack", err)
        return {"statusCode": 500, "body": "" + err}


@lambda_handler.handle("post", path="/v2/<room_id_or_name>/notification")
def room_notification(event, room_id_or_name):
    return generic_handler(event, room_id_or_name)


the_handler = lambda_handler

template.yaml

This is just a basic deployment as SAM doesn’t yet support application load balancers as an event source. Since I wanted to keep the LoadBalancer separate from the Lambda stack I just opted to manually create it but if you want to add it to your stack this template can be added or used directly.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  SlackToken:
    Type: String
    Description: "the token used to call slack"

Resources:
  HipchatFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: hipchat_proxy/router.the_handler
      Runtime: python3.6
      CodeUri: ./
      Description: 'back end for batch rest api store'
      Timeout: 5
      Environment:
        Variables:
          SLACK_TOKEN: !Ref SlackToken

Limitations of API Gateways

Private:
Public:
  • You can only using a domain name once, globally
  • It is publicly accessible, using sigv4 Authorization headers for IAM auth is not an option in some cases

Comments

  1. https://github.com/trustpilot/python-lambdarest
    has been updated to support Application Load Balancer:

    ```
    from lambdarest import create_lambda_handler

    lambda_handler = create_lambda_handler(application_load_balancer=True)

    @lambda_handler.handle("get", path="/foo//")
    def my_own_get(event, id):
    return {"my-id": id}
    ```

    pip install lambdarest>=5.4.0

    ReplyDelete
  2. Thanks for the informative blog post, we also have a relevant site SQL Server Load Rest Api

    ReplyDelete

Post a Comment