CI/CD for the cdk rootmail construct

We've all been there: your snapshot tests are green, cdk synth is happy, and you ship the construct. Then a user opens an issue two weeks later because the thing never actually worked in a real account. For awscdk-rootmail that risk is real, because most of what the construct does only happens after a real deployment: a subdomain gets delegated, SES starts receiving mail, DKIM propagates, a Lambda parses an incoming email and opens an OpsItem. None of that shows up in a synth.

In this post I want to share how I set up CI/CD for rootmail so that every change is continuously tested against real AWS, not just a snapshot, but a full deploy-send-receive loop in the regions where SES email receiving is actually available.

What rootmail does (and why a snapshot isn't enough)

Quick recap for anyone new here. rootmail is an opinionated CDK construct that gives you a single mailbox for all the AWS account root user emails in your organization. It's the cdk adaption of the superwerker rootmail feature. You own a domain, the construct carves out a subdomain like aws.mycompany.test, wires up SES email receiving, stores every incoming mail in S3, and raises a Systems Manager OpsItem so nothing gets lost.

The problem for testing is that almost all of the interesting behaviour is runtime behaviour:

• The subdomain NS records have to be delegated and actually propagate, and the custom resource polls every 10 seconds and can take 10-15 minutes to go green.
• SES email receiving only works in a handful of regions, so "it deploys" has to mean "it deploys there".
• An email sent to root+<id>@aws.mycompany.test must be received, parsed, written to S3, and turned into an OpsItem.

A jest snapshot proves the CloudFormation template didn't change by accident. It proves nothing about whether an email actually arrives. That gap is the whole reason this pipeline exists.

The problem

So the goal is simple to state and annoying to build: every pull request should prove, against real AWS, that rootmail still receives and processes an email end to end, and it should do that without me babysitting a console.

Three things make that harder than a normal integ test:

1. It's slow. A single run deploys a stack, waits for DNS, sends a mail, and tears everything down. That's ~20 minutes of real infrastructure, not milliseconds.
2. It's regional. SES receiving isn't everywhere, so the test (and later the published template) has to target the SES-capable regions specifically.
3. It needs credentials. Real deploys need an AWS role, and a pull request from a fork must never get access to it.

The solution

I ended up with two layers of end-to-end testing on top of the normal unit/snapshot tests, plus a projen-generated pipeline that runs them and publishes the artifact.

Layer 1: integ-tests with the CDK integ-runner

The first layer uses @aws-cdk/integ-tests-alpha and the integ-runner. It deploys the real stack, runs assertions against real AWS APIs, and then destroys everything. The test is the stack under test plus a set of assertions.

The heart of it is the assertion chain in integ-tests/integ.rootmail-filter-subject.ts: first check that the hosted-zone parameter landed in SSM, then invoke a Lambda that sends a real test email and expect a clean "OK" back.

const rootmail = new Rootmail(stackUnderTest, 'testRootmail', {
  domain: testDomain,
  subdomain: testSubdomain,
  // tests took on average 10-15 minutes, but we leave some buffer
  totalTimeToWireDNS: Duration.minutes(20),
  wireDNSToHostedZoneID: hostedZoneId,
  customSesReceiveFunction: customSesReceiveFunction,
  filteredEmailSubjects: ['filter-me-subject', 'i-want-to-be-filtered'],
});
 
// 1. the NS records for the subdomain made it into SSM
const getHostedZoneParametersAssertion = integ.assertions
  .awsApiCall('SSM', 'getParameter', { Name: rootmail.hostedZoneParameterName })
  .expect(ExpectedResult.objectLike({
    Parameter: { Name: rootmail.hostedZoneParameterName, Type: 'StringList' },
  }));
 
// 2. send a real email through SES and expect the handler to return "OK"
const sendTestEmailAssertion = integ.assertions
  .invokeFunction({
    functionName: sendEmailHandler.functionName,
    invocationType: InvocationType.REQUEST_RESPONSE, // run it synchronously
    payload: JSON.stringify({
      id, text: message,
      sourceMail: `test@${fullDomain}`,
      toMail: `root+${id}@${fullDomain}`,
    }),
  }).expect(ExpectedResult.objectLike({ Payload: '"OK"' }));
 
// chain them: parameters first, then the email round-trip
getHostedZoneParametersAssertion.next(sendTestEmailAssertion);

The send-email-handler itself is deliberately dumb: it just calls ses.sendEmail(...) and returns "OK" or "NOK". That single call kicks off the whole receive -> parse -> store-in-S3 -> OpsItem flow that the construct is actually responsible for.

There are two scenarios living side by side in integ-tests/:

• integ.rootmail-filter-subject.ts verifies the filteredEmailSubjects feature.
• integ.rootmail-custom-santa.ts verifies you can pass your own customSesReceiveFunction.

Running them locally is the projen integ-test target:

# dry-run first to see if the change is destructive
npx projen integ-test -- --dry-run
# then for real; --force when the runner doesn't detect custom-resource changes
npx projen integ-test -- --force

Layer 2: testing the published CloudFormation template

The integ-tests prove the source works. But plenty of people don't consume rootmail as a cdk construct; they grab the synthesized CloudFormation template. So the second layer tests the published artifact the way a real user would: create a stack straight from the template URL and wait for it to finish.

aws cloudformation create-stack \
  --stack-name rootmail-cfn-test \
  --template-url https://mvc-dev-releases.s3.eu-central-1.amazonaws.com/rootmail/${RELEASE_VERSION}/awscdk-rootmail.template.json \
  --parameters ParameterKey=Domain,ParameterValue=rootmail-test.mavogel.xyz \
               ParameterKey=Subdomain,ParameterValue=zft23-eu-central-1 \
               ParameterKey=WireDNSToHostedZoneID,ParameterValue=<hosted-zone-id> \
  --capabilities CAPABILITY_AUTO_EXPAND CAPABILITY_IAM CAPABILITY_NAMED_IAM \
  --disable-rollback --region eu-central-1
 
# then poll describe-stacks until *_COMPLETE or *_FAILED, up to ~10 minutes

My rule of thumb here: if users can deploy it two ways, test it two ways. So the template gets exercised twice: once with the WireDNSToHostedZoneID given, and once without it (adding the hosted zone by hand afterwards), because those are two genuinely different code paths in the custom resource.

Wiring it into CI/CD

The pipeline is projen-generated, so the workflows are defined in .projenrc.ts and not hand-edited YAML. On top of the standard build -> awslint -> self-mutation -> package-js flow, I added custom publish jobs:

• release_s3_dev in build.yml: on every pull request, synth the template, publish the assets, and push both a JSON and a (via cfn-flip) YAML template to the mvc-dev-releases bucket, versioned as 0.0.0-<branch>-<timestamp>-<sha>.
• release_s3 in release.yml: on release, the same thing to mvc-prod-releases.

Two design decisions worth calling out:

Multi-region publishing. Because SES receiving is regional, the template and assets go to a fixed list of SES-capable regions so a user can always pull from a bucket near them:

const releaseRegions = [
  'us-east-1', 'eu-west-1', 'us-west-2',
  // regions that got SES receiving in 2023-09
  'eu-central-1', 'us-east-2', 'ca-central-1',
  'ap-northeast-1', 'ap-southeast-1', 'ap-southeast-2',
].join(',');

Forks never touch secrets. The publish job only runs when the build didn't self-mutate and the PR comes from the same repo, using OIDC role assumption (DEV_RELEASE_ROLE / PROD_RELEASE_ROLE) instead of long-lived keys:

if: "!(needs.build.outputs.self_mutation_happened) && !(github.event.pull_request.head.repo.full_name != github.repository)"

The public buckets these jobs publish into aren't magic; they come from an earlier building block, the s3-cdk-assets-bootstrap construct, and I wrote up the release workflow itself in more detail here.

The testing layers at a glance

Each layer catches a different class of failure, which is exactly why I keep all of them:

Conclusion

Snapshot tests tell you what your template looks like. They don't tell you whether an email ever arrives. For a construct like rootmail, where the whole value is in the runtime behaviour, the only test I trust is the one that deploys into a real account, sends a real mail, and checks it came out the other end.

If you're building a CDK construct that does real work at deploy time, my advice is to invest early in integ-tests-alpha and let CI run it on every PR. Yes, it's slow and it costs a few cents in real infrastructure. It's still cheaper than a user finding the bug for you.

The code is all in the open:

• awscdk-rootmail
• the released construct

Feel free to provide feedback in the form of issues or pull requests. There is also a CONTRIBUTING.md in the repository, which explains it in more detail.