cdk integ-runner - a deeper dive

Introduction

If you build CDK constructs, you know the feeling: your unit tests are green, the synthesized template looks right, and then you deploy and something behaves differently in the real account. Unit tests check that a resource is configured the way you expect. They can't tell you whether your Lambda, your DynamoDB table, and the IAM role actually work together once CloudFormation has created them.

That gap is exactly what integ-runner and the integ-tests library are for. There's already a good introduction on dev.to that covers the theory, so I won't repeat it. In this post, I want to share the parts that tripped me up when adding real integration tests to my open-source constructs: the things that aren't in the docs and why, as the title hints, the tooling is not yet perfect in some respects.

Let's get started.

A quick primer

integ-runner looks for files matching integ.*.ts in your test directory. Each file is a self-contained CDK app that defines a stack under test plus its assertions, using the integ-tests library. Two things happen when you run it:

• Snapshot tests synth the app and diff the template against the committed *.snapshot/ directory. No AWS calls, fast, catches unintended template changes.
• Assertion tests deploy the stack for real, call your resources (a Lambda, an API, an AWS SDK call), and check the response.

A brand-new test always fails the snapshot step because there's nothing to compare against. You deploy and record it with:

integ-runner --directory ./integ-tests --parallel-regions us-east-1 --update-on-failed

Upon success, the snapshot is written and committed. One detail worth knowing: by default the update workflow deploys the previous snapshot first and then applies your change as a stack update, so it also catches breakage that only shows up on an update rather than a fresh deploy. Keep that in mind; it comes back to bite us in the CI section.

These tests deploy real infrastructure, so they are slow. A single pass across three of my tests looks like this:

Test Results:
- integ.ubuntu          872.404s
- integ.custom-domain  1039.632s
- integ.al2023          881.885s
 
Tests: 3 passed, 3 total

14 to 17 minutes per test. That price tag is why every one of the following gotchas actually hurt. Each one is a 15-minute round trip to find out you got it wrong.

The challenges & solutions

Granting IAM permissions to the function under test

Your assertion often invokes a Lambda that performs the real work, in my case, sending an email via SES. That function needs its own permissions. The way that works is to attach them to the function itself via initialPolicy:

const myfunc = new NodejsFunction(stackUnderTest, 'my-func-handler', {
  entry: path.join(__dirname, 'functions', 'my-func-handler.ts'),
  runtime: lambda.Runtime.NODEJS_18_X,
  timeout: Duration.minutes(5),
  // works
  initialPolicy: [
    new iam.PolicyStatement({
      actions: ['ses:SendEmail'],
      resources: ['*'],
    }),
  ],
});
 
const myAssertion = integ.assertions
  .invokeFunction({
    functionName: myfunc.functionName,
    logType: LogType.TAIL,
    invocationType: InvocationType.REQUEST_RESPONSE, // run it synchronously
    payload: JSON.stringify({ k1: 'v1', k2: 'v2' }),
  })
  .expect(
    ExpectedResult.objectLike({
      // Note: the result is wrapped in a Payload object. Keep that in mind while asserting.
      Payload: { resultCode: 200 },
    }),
  );

What does not work is reaching for the assertion provider's role instead:

// gives a 403: the provider role is not the function's role
// myAssertion.provider.addToRolePolicy({
//   Effect: 'Allow',
//   Action: ['ses:SendEmail'],
//   Resource: ['*'],
// });

The provider is the internal Lambda that the assertion framework uses to call your function; it is not the function under test. Grant the permission where the work happens, on the function, not on the provider.

And even with the correct policy, the first invocation can still return a 403. That's IAM being eventually consistent: CloudFormation creates the role and invokes the Lambda faster than the permissions propagate across AWS's distributed systems, so the handler needs to retry authorization errors instead of failing on the first one. If your assertion flakes only on a fresh account or a fresh deploy, this is usually why.

The assertion result is double-wrapped

invokeFunction returns the Lambda's payload wrapped in a Payload field, and the value is the JSON-serialized return value. That serialization step is easy to miss. Say your handler just returns the string OK. The obvious assertion looks right:

.expect(ExpectedResult.objectLike({ Payload: 'OK' })); // does not work

But the run fails, and the output shows why:

{
  "ExecutedVersion": "$LATEST",
  "LogResult": "<base64-encoded logs>",
  "Payload": "\"OK\"",
  "StatusCode": 200
}
!! Expected OK but received "OK"

The returned string OK came back as "OK", with the quotes, because it's the JSON encoding of the string. So the expected value has to be encoded the same way:

.expect(ExpectedResult.objectLike({ Payload: '"OK"' })); // works

Expected OK but received "OK" is a confusing error message when you don't know the payload is JSON. Now you do.

Changing an assertion doesn't always re-run the test

This one cost me a while. I updated the expected value in an assertion, re-ran the runner, and it reported a pass without ever redeploying and re-checking. integ-runner decides what to run from the snapshot diff, and a change to an assertion's expected value doesn't always move the synthesized template enough to be picked up, so the deployed assertion is never re-executed. That means an incorrect assertion can keep passing on a stale snapshot (aws-cdk#25191, still open at the time of writing, and reproduced by others as recently as 2.148.0-alpha.0).

The workaround is to force it:

integ-runner --directory ./integ-tests --parallel-regions us-east-1 --update-on-failed --force

--force reruns the integration test even when the snapshot comparison says nothing changed. It's slower, you pay the full deploy every time, but it's the only way to be sure your assertion change is actually exercised.

Don't return large objects from assertions

At some point, an assertion started failing with a message that has nothing to do with my test logic:

IntegTest/DefaultTest/DeployAssert failed:
Response object is too long.

The reason: the assertion result travels back to integ-runner through a CloudFormation custom resource, and CloudFormation caps that response object. The maintainer who closed aws-cdk#24490 pegs the limit at 4096 bytes and confirms it's a CloudFormation limit, not a bug in the construct.

Two habits keep you under it. First, don't log or return large blobs from custom resources. A full HTTP response body will blow the limit on its own:

// No logs due to error 'Response object is too long'
// log({ message: 'Fetching document', url })
const response = await fetch(url);

Return the minimum your assertion needs to check:

return 'OK';

Second, when you do need to assert on a big response, assert on a single value with assertAtPath() instead of comparing the whole object. That's the workaround the maintainer recommends, and it keeps the payload small.

Asserting on a URL-encoded policy document

Here's a subtle one. I wanted to assert on the trust policy of an IAM role, a GitHub OIDC provider setup like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:fooOrg/fooRepo:*"
        }
      }
    }
  ]
}

CloudFormation and the AWS SDK hand that policy document back to you URL-encoded, as one long string. So your expected value has to be encoded the exact same way, and encodeURIComponent alone isn't enough, because it leaves * untouched while AWS encodes it as %2A. I ended up with a tiny script to build the expected string and prove it round-trips before wiring it into the assertion:

var expected = '%7B%22Version%22%3A%222012-10-17%22%2C%22Statement%22%3A%5B%7B%22Effect%22%3A%22Allow%22%2C%22Principal%22%3A%7B%22Federated%22%3A%22arn%3Aaws%3Aiam%3A%3A123456789012%3Aoidc-provider%2Ftoken.actions.githubusercontent.com%22%7D%2C%22Action%22%3A%22sts%3AAssumeRoleWithWebIdentity%22%2C%22Condition%22%3A%7B%22StringEquals%22%3A%7B%22token.actions.githubusercontent.com%3Aaud%22%3A%22sts.amazonaws.com%22%7D%2C%22StringLike%22%3A%7B%22token.actions.githubusercontent.com%3Asub%22%3A%22repo%3AfooOrg%2FfooRepo%3A%2A%22%7D%7D%7D%5D%7D';
 
var plain = `
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                },
                "StringLike": {
                    "token.actions.githubusercontent.com:sub": "repo:fooOrg/fooRepo:*"
                }
            }
        }
    ]
}
`.replace(/\s+/g, '').trim();
 
// encodeURIComponent leaves '*' alone; AWS encodes it as %2A, so patch it
var encoded = encodeURIComponent(plain).replace(/\*/g, '%2A');
console.log(encoded === expected); // true

If your policy assertions fail for no obvious reason, check the encoding of *, : and the whitespace first.

Running integ tests in GitHub Actions

Running these locally is fine. Running them in CI for an open-source project is where it gets interesting, because integration tests need real AWS credentials, and you cannot hand real credentials to code from a pull request you haven't read.

That's the classic "pwn request" problem: a workflow triggered by a fork PR runs attacker-controlled code, and if that workflow has access to your secrets, the secrets are gone. GitHub's own guidance is to require approval for fork workflow runs. Two building blocks make this workable:

• OIDC to AWS instead of long-lived access keys, so there's no static secret to steal in the first place (AWS's write-up is a good start, and ranthebuilder's post is a great end-to-end example).
• Maintainer-gated runs for anything from a fork. This is exactly how the aws-cdk repo does it: a PR submits a build request, and a maintainer has to approve it before the CodeBuild integ pipeline runs against real AWS.

The main merge-base error

The first time I ran the update workflow in CI, it blew up like this:

Running test integ-tests/integ.rootmail.ts in eu-west-2
fatal: Not a valid object name main
Could not checkout snapshot directory integ.rootmail-custom-santa.ts.snapshot using these commands:
git merge-base HEAD main && git checkout {merge-base} -- integ.rootmail-custom-santa.ts.snapshot
error: Error: Command exited with status 128

Remember the update workflow deploys the previous snapshot first. To find it, integ-runner runs git merge-base HEAD main to locate the common ancestor and check out the old snapshot from there. In a default actions/checkout, the history is shallow and main isn't fetched as a local ref, so merge-base can't resolve main, and the whole thing exits with code 128. The fix is to give CI the history it needs: a full checkout with fetch-depth: 0, so main actually exists locally. You can watch it fail and then go green across two of my runs (the failing one and the follow-up). The whole CI setup landed in awscdk-rootmail#221.

Once it runs, it looks like this, with real deploys in parallel across regions:

One more thing, from a CDK maintainer when I asked about this in the CDK Slack:

It's not enough to check that the test works on brand-new infrastructure (deploying from nothing). You also need to check whether it deploys to the previous state (a deploy update over an existing stack). I don't think there's really a way to catch all update permutations. So this always has to be a risk-based approach.

You cannot test every upgrade path from every past version, so pick the transitions that matter and accept that the rest is risk-based. It's also why integ-runner bothers to deploy the old snapshot before your change in the first place.

What's next

There's a second half to this that deserves its own post: locking these constructs down to least privilege (working backwards from the IAM Access Analyzer's Last Accessed data, CloudTrail Lake, and permission boundaries) and keeping them clean with cdk-nag. I hit enough sharp edges there, like a role with AdministratorAccess that was still missing s3:ListBucketVersions, or nag suppressions that only work by resource path, that it's a topic on its own. More on that in the follow-up.

Conclusion

integ-runner is the best tool we have for testing that CDK constructs actually behave once they're deployed, and I wouldn't ship a construct library without it. But it has sharp edges, and most of them cost you a 15-minute deploy to discover. Here's the short version to save you that time:

What I learned: when an integration test fails, first ask whether the test harness is wrong before you touch the code under test. With integ-runner, more often than I'd like, it's the harness.

I hope this saves you a few of those 15-minute round trips. Happy testing!