We’re builders. And we’re fortunate to talk with lots of other builders each week. One thing we know from these conversations is that builders are constantly learning - often by swapping war stories about things that didn’t go exactly according to plan. Whether it’s something unexpected - or unexpectedly hard - each usually offers at least one hard-won lesson.

This week we spotlight Rick Hensbergen, the CEO & Founder of Esther.ai, and invite you to benefit from his experience building an AI-powered automated testing platform.

Esther.ai - Lessons from a GenAI SaaS Builder

About Esther.ai -

During my 25 years in the software development world I’ve observed development getting faster and faster, while testing has become more and more of a bottleneck. There are many reasons for this, but the solution isn’t bigger test teams or better testers. Our market research confirmed my own experience with our testing clients that most want more test automation earlier in the life cycle. But there simply aren’t enough testers with automation skills. There are over 27 million software developers in the world and about 4.5 million testers. But only a tiny fraction of those testers can automate. So we created Esther.ai to fill the gap by automating the entire testing process: design, build, execute and report. Today, Esther.ai executes 500,000 test cases an hour - fully automated with the output available in different formats and through APIs back to tools like JIRA.

The worst part?

As the founder of a bootstrapped startup, the continuous stress of being able to fund the next step has been the most challenging part. Every change costs time and money. That isn’t surprising. But what has surprised me is that the changes we’re having to make aren’t optional. My entire career has revolved around systematically testing changes we’ve made to software. When you build on top of these fast evolving LLMs, things break even when we don’t make changes. And they break without warning - so it’s always a fire drill to fix. And not even a drill really, because these things happen in Production.

I’ll give you 2 examples from just this quarter. In January, OpenAI deprecated the model we were using. This caused over a week’s worth of unplanned work. This month they made a change to the new model that broke everything again.

I’m from the Netherlands - a quarter of which is built on polder - which is basically swamp land. To build a house in the Netherlands, you need to pour concrete pilings that go down 15 to 20 meters before they hit a solid base.

I now realize that the way we built Esther.ai by connecting directly to OpenAI APIs is like building a house in the Netherlands without those concrete pilings. I see Lamatic as a way to fix this problem which is why we’re planning to migrate.

Top 3 take aways -

1) More Modular & More Managed Services
Things are changing so fast. There are a lot of things I’ve learned in the last 12 months that aren’t even relevant now. And I’m sure that will be just as true in 12 months. My development philosophy now is to focus on modularity and eliminating custom code wherever possible.

And it isn’t just about LLM integrations. For instance, a year ago we started out on MySQL. Then we had a major project to move to MongoDB - because MySQL didn’t scale well. We did it all again when we moved to Postgres - because MongoDB had become too slow. Each of these migrations was costly and time-consuming.

So, of course, one thing I’d do differently is to build on top of the Lamatic’s managed platform. Esther.ai currently supports 6 production test types. I’m convinced that if we had started on Lamatic’s platform, we’d have all 42 test types on our roadmap live right now.

2) More Market Research
The second thing I’d do differently is to gather more market insight earlier. In retrospect, I relied too heavily on my own experience to direct our product roadmap.

I’ve learned that this is a really common mistake which I’m working right now to correct by re-engaging potential clients to explore their challenges, frustrations, current solutions and hopes for the future more deeply. Not only will this help us get our product roadmap right, it will also create a level of understanding and engagement that will help us bring on new clients.

3) Sell, Sell, Sell
Finally, I’d focus way more energy on selling. And getting our product into the hands of early adopters sooner.

Looking back at the last 12 months, I can see my experience in the banking industry conditioned me to see failed test conditions as something to avoid. What I’ve learned is that for an entrepreneur, these failures are the fastest way to learn and iterate.