Case study

How this page gets here

This page is not hosted on a platform. It is a Docker image, built from this repository by a GitHub Actions runner, stored in Amazon ECR, pulled onto an EC2 instance and served by Nginx. Everything below is the real pipeline, and every claim here can be checked against the source or the AWS console.

The path a commit takes

  1. Push to main. The commit SHA becomes the version number of the artefact. There is no latest tag anywhere, so “what is running?” always has an exact answer.
  2. Static checks (~5 s). Required files present, HTML tags balanced, health-endpoint contract intact. This gate touches nothing in AWS, so a typo fails here instead of after a two-minute build.
  3. Build, then smoke test. The image is built under a local name and started. Five routes are fetched from it, and /healthz must return exactly ok. Only an image that has already served traffic is allowed near the registry.
  4. Push to ECR. Retagged with the full commit SHA. Tag immutability is enabled, so a tag can never be repointed at different bytes — which is what makes a rollback to a tag mean something.
  5. Deploy through Systems Manager. The runner holds ssm:SendCommand against one instance. The script runs as root on the instance, pulls the image, removes the old container, starts the new one and polls /healthz.
  6. Verify from the internet. The endpoint is fetched from outside AWS, because a check that runs inside the machine cannot detect a broken security group.
  7. Guard. A final job fails the run unless the deploy actually executed. GitHub will otherwise report a skipped job as a green run.

When the new version is unhealthy

The deploy script polls /healthz ten times, two seconds apart. If the container never answers, it restarts the previous image, polls again, and then exits non-zero:

ROLLING BACK to .../devops-cicd-demo:<previous-sha>
still unhealthy - rolling back to previous image
healthy after 2 attempt(s)

The site stays up. The pipeline still fails. That split is deliberate: a rollback is a recovery, not a pass, and a run that reports green after recovering teaches everyone to ignore red.

This was proven by deploying a deliberately broken image to the running instance — not in a staging copy of the pipeline.

Why there is no SSH

The obvious design is a GitHub runner SSH-ing into the instance. It cannot be made safe here: GitHub publishes 5,622 IPv4 runner ranges and a security group accepts 60 inbound rules, so whitelisting the runner is arithmetically impossible. The only alternative would be port 22 open to the world. Systems Manager removes the need for the port entirely, and adds an audit trail: every command that runs on the instance is recorded with its output.

Security posture

ControlHow it is implemented
No credentials on the serverEC2 instance role, pull-only, one ECR repository
No inbound SSHPort 22 closed; verified from outside: 22 open=false
Least privilege (CI)Push and pull on one repository, plus ssm:SendCommand on one instance
No secrets in GitGitHub encrypted secrets; keys and env files ignored
Image integrityTag immutability, scan on push, lifecycle policy for untagged images
One public surfacePort 80 only. The deploy path does not use it.

What broke, and what it taught

Four consecutive deploys failed with what looked like a payload problem. The real cause was a --comment argument of 105 characters against a documented limit of 100 — rejected before it ever reached the machine.

It stayed hidden for four runs because my diagnostic hid its own message: I built a GitHub error annotation from the first line of the AWS error, the CLI prefixes its output with a blank line, and GitHub silently discards an annotation with an empty message. The instrument built to reveal the failure was invisible.

The fix was four characters. The lesson was larger: when a diagnostic shows nothing, suspect the diagnostic. The full write-up, including the false-green run that skipped a deploy while reporting success, is in the repository under docs/.

What this is not