From Prototype to Production

The payments prototype I had a hand in had a demo that worked perfectly. A demo is a path you walked ahead of time, on inputs you chose, in front of people you wanted to impress. Real traffic did none of that.

Validation started as a stack of if-statements in one function. Fine for the room. The first real failures were encodings we did not test, blank fields we assumed required, and a job that half-finished and left the data in a state nobody designed for. The function did not survive contact with any of those.

Error handling stops being a footnote

A prototype is allowed to throw. You see the stack trace, you fix it, you rerun. In production the exception is a user staring at a spinner, or a payment that landed in a half-written row. The work is no longer “make it succeed”. It is “decide what happens on every way it can fail”, and on that path that turned out to be most of the code.

Observability is a feature, not a chore

You cannot fix what you cannot see. When something went wrong after the rewrite, the logs, the metrics, and the traces were the entire investigation. Adding them after the fact means reconstructing what you should have recorded, and it costs many times what building it in would have. I have paid that tax. It is not worth it.

The data was never the shape we designed for

The prototype ran on data shaped the way we imagined data. Real users produced encodings we did not test, fields left blank that we assumed were required, volumes an order of magnitude off the guess. The edge cases waved off in design review became the top of the support queue.

The rewrite is normal

The production version split into separate stages: validation, then enrichment, then recovery, each one testable and observable on its own. The rewrite took longer than writing the prototype had. The right ratio, not a failure. The prototype’s job was to find out what to build. The production version’s job is to survive being right. Researchers and clinicians taught me the same lesson from the other side: they do not walk the path you rehearsed.

A prototype answers whether this can work. Production answers what happens when it does not.