Rootstrap Blog

If you postpone technical debt it’ll bite you

An exciting contract

Back in 2013, an agency hired our company to create a game for Microsoft. They’d showcase their new and shiny Surface tablet and use the game in conferences around the US, like the big San Diego Comic-Con International event.

Our WebGL 3D game

Our client said, “The backend is done. We only want you guys to build a 3D UI with WebGL.” It was a complex multiplayer game. We worked hard for months and created an outstanding game UI with state-of-the-art technology.

A rotten core

Unfortunately, the backend we inherited and weren’t allowed to modify had one of the worst PHP code we’d ever seen. It was a humongous monolithic piece of code with no tests, terrible database design, and terrible SQL generated by terrible practices. For example, we could easily do SQL injection on almost every web service. Anyway, it was not about the language, but about the engineering practices. Never blame a programming language for your lack of skills.

We insisted that we had to rebuild the whole thing. But our client stayed firm: “It’s already built. We won’t spend money on it again. Please, focus on building the frontend.” And even worse: “The company that built it guaranteed that it will scale.”

Collapse

Months of work collapsing

The game went live, and it seemed to work fine for a while. But one day, everything collapsed. We thought it was a DDoS attack, but it was a Tsunami of real users. Microsoft posted a link to our game on both Twitter and Facebook, using their official account. That simple act created chaos. Sixty thousand unique users per hour hit a platform running on top of a backend that could barely handle 10 concurrent users.

The client knew about this marketing move, but they didn’t warn our team. It’s extremely important to keep your team updated.

Fix attempts

The backend totally collapsed, and we didn’t have many options. We scaled to the most massive databases and servers on AWS and spent a ton of money per hour trying to handle the load. But nothing worked. Signup collapsed, and no one could pass through the registration process.

We were clear with our client: “We told you it wouldn’t scale.” You always need to tell the painful truth, even when your client ignores you.

A real solution — overhaul

The client understood they’d messed up big time. It was their most important contract ever. So they asked us to fix it. We went back to the planning phase and did real engineering.

We rebuilt the whole backend in three weeks and made it incredibly simpler. We changed the UI to fit our new, well-designed API. We used caches and created an auto-scalable environment. Server costs were a fraction of what they had been. We did load tests and simulations. The platform could handle an even larger number of users if needed.

Lessons learned

But do you know what? It was too late. Even though some users kept coming, the game never got anywhere near the kind of traffic it had in the beginning. Our client lost Microsoft. And we lost the project.

This was before we had mandatory code audits before inheriting existing projects. Nowadays we wouldn’t start any work without making sure that we were able to build a high-quality backend.

We learned a harsh lesson. Start with a solid foundation even if that means spending more money than you’d planned. Don’t build on top of something you know is bad, but you think you can fix it later with no consequences. Ideally, just say “no.”

Never ever underestimate technical debt in your work. Even if it’s not yours, it will come back and bite you.

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.