July 2016 Update: this was a worthwhile experiment, but I've retired this approach for provisioning my own CloudFormation stacks in favour of a better tool: fab_aws.

CloudFormation is one of my very favourite aspects of working within the AWS ecosystem. Without proselytizing too extensively about its benefits, it allows for automated provisioning of entire stacks of AWS infrastructure; not just EC2 instances and S3 buckets, but also their respective IAM roles and bucket policies, the VPC they belong to, network configuration such as security groups, and NACLs. The implications for disaster recovery, environmental separation, repeatability, etc, are profound.

A common criticism levied against CloudFormation, though, is the unwieldiness of its syntactic approach to actually declaring the infrastructure. CloudFormation stacks are specified in a JSON-based template file. There are two glaring problems in that statement: (1) JSON and (2) file singular. JSON is a great data-interchange format, but it can be problematic for configuration: commenting is not supported, and it can be inordinately difficult to unpack and modify complex structures given their characteristic rightward drift and towers of curly braces. More often than not, CloudFormation templates become absurd JSON monoliths, and, at that, monoliths that lack the benefit of exposition of the author's intent.

I've started working around this problem with a variant of a solution I use for this very website¹: static generation! Rather than write JSON directly, we can use static generation tooling to produce the final CloudFormation template artifact. This way, we can centralize parameters, make comments, extract components into their own files, and run arbitrary code at generation time. Oh, and perfectly format the resulting JSON with jsbeautifier while we're at it.

Cloudformable

I've built a git repo that I'm calling Cloudformable that represents a starting point for new CloudFormation definitions. It relies on grunt, a task runner for the nodejs ecosystem, as well as assemble for static generation and handlebars for templating.

File Structure

A Cloudformable project looks like this:

$ tree
├── Gruntfile.js
├── cloudformation-stack.hbs  # The entry point.
├── data                      # Data to be injected.
│   └── about.yml
├── helpers                   # Arbitrary code definitions.
│   └── git.js
├── package.json
└── partials                  # Template fragment extractions.
    ├── outputs.hbs
    ├── parameters.hbs
    ├── resources
    │   ├── lambda-execution-policy.hbs
    │   ├── lambda-execution-role.hbs
    │   ├── lambda-function.hbs
    │   ├── lambda-permission.hbs
    │   ├── s3-bucket-policy.hbs
    │   └── s3-bucket.hbs
    └── resources.hbs

Entry point: cloudformation-stack.hbs

The top level cloudformation-stack.hbs file is the starting point for preprocessing:

{{!-- cloudformation-stack.hbs --}}

{{!-- This is a comment! --}}
{{!-- Comments aren't valid in JSON, but they are in Handlebars! --}}
{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Description": "{{ about.description }}",

  "Metadata": {
    "CommitHash": "{{ commitHash }}",
    "CommitComment": "{{ commitComment }}"
  },

  "Parameters": {{> parameters }},

  {{!-- Unused for now; uncomment and add a partial to enable. --}}
  {{!-- "Mappings": {{> mappings --}}
  {{!-- "Conditions": {{> conditions --}}

  "Resources": {{> resources }},

  "Outputs": {{> outputs }}
}

The standard CloudFormation keys (e.g. Resources, Outputs, etc) are plainly identifiable, but their actual definition is extracted into partials; the {{> }} handlebars operators render those partials. Moreover, we can extract data (e.g. {{ about.description }}) to the data definition, as well arbitrary code that runs at generation time².

Partial: parameters.hbs

Here is the definition of parameters.hbs:

{{!-- parameters.hbs --}}
{
  "MemorySize": {
    "Type": "Number",
    {{!-- defaultMemory: defined in lambda-function.hbs --}}
    "Default": {{ lambda-function.defaultMemory }},
    "Description": "Memory size for Lambda function"
  }
}

Aside from being easier to understand, we can take advantage of handlebar variables to render values from elsewhere. In this case, the Lambda function's default memory parameter can reside with the Lambda function definition: leading to parametric locality, in other words.

Rendering

A simple grunt command renders the final template file (and jsbeautifies the resulting JSON file):

$ grunt render
Running "assemble:site" (assemble) task
Assembling _output/cloudformation-stack.template OK
>> 1 pages assembled.

Running "jsbeautifier:files" (jsbeautifier) task
Beautified 1 files, changed 1 files...OK

Done, without errors.

A sample of the output, in all its final rendered and beautified glory, can be found here.

Use

Feel free to use this on your own project. The repo README.md contains more details, but it's really as simple as this:

git clone https://github.com/dliggat/cloudformable
cd cloudformable
npm install
grunt render

I originally set out to release this as a grunt plugin, but it became clear that the right model is to treat this entire repo as a starting point for other CloudFormation initiatives.

Addendum: Why Node?

Putting it mildly, I can't claim to be a tremendous fan of JavaScript as a general purpose language. However, the advantages of node for tasks like this are increasingly difficult to deny. The npm ecosystem is enormous, and grows by the day. Also, node seems to have quietly acquired a top-tier status within the AWS world: it is the original (and only one of three so far) AWS Lambda language³, and is the only modern⁴ language option for the AWS IoT SDK. Moreover, most of the worst ills of JavaScript go away with ES6, so the future's looking bright, and therefore now strikes me as a great time to build some strong fundamentals in this realm.