How We Ship: The Daily Release Train

Every weekday at eleven in the morning, a scheduler fires and each of our eleven services deploys whatever is on its main branch to staging. A day later the same commit goes to production, unless something has happened in between to say it should not. Nobody presses a button. Nobody asks whether the release is ready. If a change is on main, it is on the train.

That is a long way from where we were at the start of the year. This post is about how we got here: what the old pipeline cost us, how the daily release train works, the parts of it we only understood once it was live, and what we would tell another team about to make the same move.

Three Branches Tied to Three Environments

Every service had three permanent branches, develop, release, and main, and each one was wired to an environment. A push to develop built and deployed to dev. A push to release built and deployed to staging. A push to main built and deployed to production. The branch was the environment, and promoting a change meant moving it from one branch to the next.

In practice that meant three or four pull requests to get one change into production. You merged to develop, then a PR from develop to release, then one from release to main. A bot kept the branches in step with cascading PRs in both directions. Merge to develop and it opened the PR up to release; merge to release and it opened the one up to main; each of those waited for a person to approve it. Land a hotfix on main and it opened PRs back down through release and develop, merging them itself when it could. When it could not, someone got a Slack alert and went to resolve a conflict before unrelated work could move.

Tying branches to environments produced most of our pain, and it produced it in three ways.

Releases were big and they were late. Nothing flowed to production on its own, so changes piled up on release while QA worked through them. If QA was testing one feature and ten others were waiting behind it, all eleven waited. If QA found a bug, someone hotfixed staging, QA tested again, and two more days went by with the same ten features still sitting there. Eventually the pile got large enough that we shipped it as a batch and hoped. Before we did, someone had to ask each feature's owner whether theirs was ready, across time zones and around people out of office. There was no cadence, so nobody could tell you whether something would be live today, this week, or later.

Every merge was a deploy. A build and a deploy were one event, roughly thirty minutes end to end, and every PR merged into any of the three branches triggered one. A dependency bump got the same treatment as a feature someone was actively testing.

Rollback meant a rebuild. There was no way to put back the previous release without going through the pipeline again: tag an older commit, wait for the image to build, deploy it. Ten minutes for a small service, forty for our Rails monolith. And a hotfix merged straight to main cascaded back down through the other branches, which was often not what you wanted and was where a good share of the merge conflicts came from.

None of this was anyone's mistake. It is the natural shape of gitflow with a bot attached, and it had worked at a smaller team size. It was clear it would not work at the size we were becoming.

One Branch, One Schedule

The model we run today came out of a lot of whiteboarding with Prateesh, Asi, and Hammad, who shaped the design with me from the first sketch, and the first thing it does is separate two events the old pipeline had fused: merging code and deploying it.

Each service has one permanent branch, main. Feature branches come off it and go back into it by squash merge, with one approving review and green CI. Merging deploys nothing. It puts the change on the next train.

The train is a schedule. At 11 AM, Monday to Friday, every service builds the head of main and deploys it to staging. After a successful staging deploy it records that commit as the day's pin. The production tick, Monday to Thursday, reads the most recent pin that has soaked for at least twenty-four hours, checks out that exact commit, and deploys it. Friday has no production tick, so Friday's pin soaks over the weekend and ships Monday. The best case for a merge is about a day from main to production, if it lands just before a tick. The worst is about five days, for a merge just after Wednesday's tick, whose pin sits on staging over the weekend. Either way you know which it is the moment you merge.

Two decisions in that design mattered more than they looked.

The pin is a record, not a pointer at staging. Engineers can still dispatch a branch to staging by hand when dev is not enough, and that would be a problem if production simply promoted whatever was on staging. It does not. The train's staging deploy writes a pin, a manual staging deploy does not, and production reads only pins. The two paths keep separate ledgers.

The same commit goes to both environments, but not the same image. We still build once per environment, with each environment's configuration pulled from Parameter Store at build time and baked into the container. We considered building once and promoting the artifact, and decided the refactor was not worth it at our deploy volume: every service's entrypoint, its IAM, and the way Next.js handles public environment variables would all have changed. What we test on staging is the exact source that ships, not the exact bytes. We wrote that tradeoff down and accepted it.

Dev is still there for branch testing, deployed by manual dispatch. The train does not touch it.

Feature Flags Are What Make a Train Possible

A single branch on a fixed schedule has an obvious failure mode. If my change and yours are both on main, and yours is broken, mine cannot ship without it. In the old world you would have held your PR off release. In the new one there is nothing to hold it off.

The answer is that shipping code and releasing a feature are separate decisions. A feature that is not ready to be seen ships dark behind a flag, default off, and is switched on later without a deploy. A risky change carries a kill switch, so turning it off does not mean rolling the whole train back. Large features land in pieces, each one inert until the flag flips.

Hammad built that flagging infrastructure, and it had to be in place before the train could carry real traffic. It is the first thing I would tell another team to do. Without it the train stops at the first unready PR, and you are back to asking ten people whether they are ready.

What Guards the Promotion

The design on paper had a soak and a pin. The version in production has a good deal more, and every addition came from something that happened.

Some of it we saw coming. A hotfix deployed straight to production on a Tuesday afternoon would, without a check, be undone by Wednesday's tick, which promotes Tuesday morning's pin from before the fix existed. So the production job runs an ancestry check first: if the pinned commit is already an ancestor of what production is running, there is nothing to do, and the tick skips.

Some of it we did not. An engineer deployed a hotfix from a branch, then squash-merged that branch to main. A squash merge creates a new commit with no git ancestry back to the branch, so the check saw production running a commit that was not an ancestor of anything and concluded it was safe to overwrite. We caught it before the tick fired, but only because someone was watching. The guard now also asks GitHub which pull request a commit came from, which survives a squash, and every manual production deploy is marked as such in its deployment record, so the train holds until a pin newer than the hotfix has soaked.

Some we learned in the first week. We went live just before a holiday weekend. We had a rule against Friday production deploys and nothing at all about holidays, so the train was perfectly willing to ship to production on a day when nobody was around to watch it. There is now a holiday calendar checked in beside the workflow, and the train skips production on the holiday and on the day before it, for the same reason it skips Friday.

And one we had exactly backwards. The first version made the production job depend on the staging job, which in GitHub Actions means production only runs if staging succeeded. That sounds like safety. It is the opposite: a failed staging deploy today says nothing about yesterday's pin, which soaked for a day and passed everything, and blocking it only delays known-good code. Now a staging failure holds up tomorrow's promotion, because there is no new pin, and leaves today's alone. The only things that stop production outright are a failed golden test, where a service has them, and a person pausing the train.

The clock moved too. GitHub's scheduled workflows fired late, sometimes by a lot, and a twenty-four hour soak measured against a drifting clock is not twenty-four hours. The tick now comes from an EventBridge schedule that invokes a small Lambda, which dispatches the workflow. It fires when it says it will.

Hotfixes and Rollbacks Are Ordinary Now

A hotfix is a pull request like any other, merged to main, followed by one command that deploys that commit to production without waiting for the train. It is announced loudly in Slack because it bypasses the soak, and the guard makes sure the next tick does not undo it. The next pin that contains the fix soaks in the normal way and the train picks up where it left off.

Rollback is a separate workflow that repoints the service at an image already in the registry. No build. Thirty to sixty seconds, for every service, including the one whose build takes forty minutes. Under the old model that same rollback was a forty minute rebuild. Every deploy notification links to it. Beneath that, the ECS deployment circuit breaker reverts a deploy that cannot pass its health checks before it takes any traffic.

The Train Is the Gate, and the Tests Decide

The soak is time for humans to notice. The tests are what notice on their own. On our marketplace frontend, the train runs a small golden-path suite after the staging deploy, the flows that make us money, in under six minutes. If any of them fail, the train pauses itself and production does not deploy until someone clears it. After a production deploy a smoke suite checks the critical pages. A larger regression suite runs nightly and is advisory: it can give someone a bad morning but it cannot stop the train, and it is also where new tests prove they are stable before being promoted to the golden set. Every test creates its own data and cleans up after itself, so none of them depends on the state of staging. The move from a large, slow Selenium suite to this arrangement is its own story, and someone else will tell it.

Rolling It Out

We did not switch everything at once. The first cutover was our smallest active service, chosen because it gets few commits and a mistake there would inconvenience nobody. That is where the reusable workflows were built and where the first month of edge cases showed up. Then the two highest-volume repos, the marketplace frontend and the Rails monolith, in parallel, because that is where the cascade hurt most and where the change paid off fastest. Then the rest.

Each cutover was the same set of steps, which we wrote down after the first one: lock the two branches that were going away, protect main, replace the workflow files in one PR, delete the cascade, create the schedule. Most of the effort was not in the workflows. It was in making sure people knew what was changing and why, and in taking the feedback that came back. A fair amount of the final design came from that feedback.

Visibility came late and should have come early. Within weeks of the first cutover the most common question in Slack was some version of "where is my PR". I answered it by hand for a while, then we built a dashboard that shows, for every service, what is on each environment, what will ship at the next tick, and whether the train is paused or being held by a hotfix. Give it a PR number and it tells you the day that change reaches staging and the day it reaches production. If I were doing this again the dashboard would go live with the pilot.

What We Learned

Start with the people, not the pipeline. The cascade model was not just a set of workflows, it was how everyone thought about shipping. The train changed that, and the change stuck because we spent the time explaining it, took objections seriously, and adjusted. The pipeline work was the smaller half.

Decide with evidence. Before we built anything we measured build times per service, counted how many PRs a change needed, and diffed the three Dockerfiles each repo carried, which in one case differed by a single whitespace character. Those facts made the case, and they also told us what to fix first.

Everything that made the train safe was learned live. The pin, the soak, and the schedule were designed. The squash-merge hole, the holiday, the coupled jobs, and the drifting clock were all discovered by running it. Starting on a low-traffic service is what made those discoveries cheap.

Batching is a feature. We do not deploy on every merge and we are not sure we ever want to. Every deploy runs the full suite against staging and then production, and done per PR most of that spend would go to verifying dependency bumps. A daily batch with a soak gives us most of the value of continuous deployment at a fraction of the cost, and for now it is a better fit for where our tests and flags are.

What's Next

Twice a day. Two of our services generate enough change that one tick a day is a real wait, so we are trialling a second daily window on one service now, with the soak becoming the gap between windows rather than a fixed day. That service's schedule is now data rather than workflow code, so moving a window is a configuration change instead of a change to eleven repositories.

Ephemeral development environments. One shared dev environment is a traffic jam whenever two features are in flight, and the cure is a full stack per branch that tears itself down when the branch is gone.

The train was never the destination. It was the step that got us shipping every day, safely, with a rollback that takes a minute. Where we go from here is a question of how much further we want to shorten the distance between merging a change and seeing it in production.