Case study
How this page gets here
This page is not hosted on a platform. It is a Docker image, built from this repository by a GitHub Actions runner, stored in Amazon ECR, pulled onto an EC2 instance and served by Nginx. Everything below is the real pipeline, and every claim here can be checked against the source or the AWS console.
The path a commit takes
-
Push to
main. The commit SHA becomes the version number of the artefact. There is nolatesttag anywhere, so “what is running?” always has an exact answer. - Static checks (~5 s). Required files present, HTML tags balanced, health-endpoint contract intact. This gate touches nothing in AWS, so a typo fails here instead of after a two-minute build.
-
Build, then smoke test.
The image is built under a local name and started. Five routes are fetched
from it, and
/healthzmust return exactlyok. Only an image that has already served traffic is allowed near the registry. - Push to ECR. Retagged with the full commit SHA. Tag immutability is enabled, so a tag can never be repointed at different bytes — which is what makes a rollback to a tag mean something.
-
Deploy through Systems Manager.
The runner holds
ssm:SendCommandagainst one instance. The script runs as root on the instance, pulls the image, removes the old container, starts the new one and polls/healthz. - Verify from the internet. The endpoint is fetched from outside AWS, because a check that runs inside the machine cannot detect a broken security group.
- Guard. A final job fails the run unless the deploy actually executed. GitHub will otherwise report a skipped job as a green run.
When the new version is unhealthy
The deploy script polls /healthz ten times, two seconds apart.
If the container never answers, it restarts the previous image, polls again,
and then exits non-zero:
ROLLING BACK to .../devops-cicd-demo:<previous-sha>
still unhealthy - rolling back to previous image
healthy after 2 attempt(s)
The site stays up. The pipeline still fails. That split is deliberate: a rollback is a recovery, not a pass, and a run that reports green after recovering teaches everyone to ignore red.
This was proven by deploying a deliberately broken image to the running instance — not in a staging copy of the pipeline.
Why there is no SSH
The obvious design is a GitHub runner SSH-ing into the instance. It cannot be made safe here: GitHub publishes 5,622 IPv4 runner ranges and a security group accepts 60 inbound rules, so whitelisting the runner is arithmetically impossible. The only alternative would be port 22 open to the world. Systems Manager removes the need for the port entirely, and adds an audit trail: every command that runs on the instance is recorded with its output.
Security posture
| Control | How it is implemented |
|---|---|
| No credentials on the server | EC2 instance role, pull-only, one ECR repository |
| No inbound SSH | Port 22 closed; verified from outside: 22 open=false |
| Least privilege (CI) | Push and pull on one repository, plus ssm:SendCommand on one instance |
| No secrets in Git | GitHub encrypted secrets; keys and env files ignored |
| Image integrity | Tag immutability, scan on push, lifecycle policy for untagged images |
| One public surface | Port 80 only. The deploy path does not use it. |
What broke, and what it taught
Four consecutive deploys failed with what looked like a payload problem. The
real cause was a --comment argument of 105 characters against a
documented limit of 100 — rejected before it ever reached the machine.
It stayed hidden for four runs because my diagnostic hid its own message: I built a GitHub error annotation from the first line of the AWS error, the CLI prefixes its output with a blank line, and GitHub silently discards an annotation with an empty message. The instrument built to reveal the failure was invisible.
The fix was four characters. The lesson was larger: when a diagnostic shows
nothing, suspect the diagnostic. The full write-up, including the false-green
run that skipped a deploy while reporting success, is in the repository under
docs/.
What this is not
- Not zero-downtime: one instance, so the container swap leaves a brief gap.
- Not autoscaling: a load balancer and a second instance would be the next step.
- Not monitored: no alerting yet, and the omission is documented rather than hidden.
- Not OIDC: the account plan blocks workload identity federation, so a scoped key is used and disclosed.
Source, workflow and phase-by-phase records on GitHub Back to the work