In this edition
GenAI development is all about speed
I spend a lot of time working with SaaS product teams. Many are working really hard to modernize their applications with GenAI, but are struggling to achieve their expected results. Others are having trouble getting started and worry they should be doing more.
Regardless of where you are, these 4 simple steps will help you pick up the pace to get in front of the competition and stay there.
1. Assume AI Can Do It
As “product people” we know the road to success is to work backward from our user’s ideal experience. But with new technologies, it’s often hard to know what’s possible. Avoid the temptation to begin by exploring what is and isn’t possible.
Instead, assume that AI can give you what you need (it’s really only a matter of time until it can). This simplifying assumption forces you to design your application and to modularize the AI piece (as an API call). It clarifies your ideal solution and will save you time experimenting with technologies, techniques and data that are changing as fast as you can experiment.
While complications will inevitably be introduced to address short-comings and corner cases, these are secondary measures that you can cut away over time (as the technologies, techniques and your data improve).
2. Adopt a “Buy Bias”
It’s still early days in this post AI world. GenAI is following the same arc all new technologies follow (whether the combustion engine or the internet). This arc starts with the early pioneers handcrafting components that over time evolve into building blocks.
It’s important to adopt these components. Doing so instantly adds new capabilities to your current and future offerings. More importantly, it allows you to focus more intently on the parts of your product that are true differentiators. Devoting resources to building things you can offload, especially when they’re evolving quickly, adds velocity-killing workload and tech debt.
3. Design your GenAI Workflow
After clearly defining the payload your application requires, it’s time to design a GenAI workflow that reliably generates it. To do this well, you need two things: a subject matter expert and a solid understanding of how Large Language Models (LLMs) work.
Start by generating “test data” - a representative assortment of prompts and answers your subject matter expert considers to be “correct”. Understand and document “why” each answer is correct (i.e., the context).
LLMs use probabilistic inference (also known as statistical inference) to compare each prompt to a vast collection of prior observations. The answer given by the LLM is a function of just two things … the conversion of the prompt into its semantic meaning (itself a function of the embedding model and how well the prompt is written) and the observations (i.e., data) that were used to train the LLM.
Evaluate how reliably the LLM responds to your test prompts with correct answers. If your expert doesn’t like a response, work to understand why. Turn this understanding into GenAI workflow “nodes” that mirror your expert’s thought process and which can be optimized for accuracy and consistency.
As you do this, it’s helpful to know that an LLM’s probabilistic inference process mirrors inductive reasoning (i.e., fitting a single observation to a pattern of similar observations with known answers). It is unlike rule-based deductive reasoning (which involves applying known rules to derive a factual conclusion). That said, you can add rules (sometimes referred to as guard rails) by including them in your prompt.
If your prompt engineering and chaining fail to achieve the required accuracy and consistency, you can experiment further by implementing Retrieval Augmented Generation (RAG) and/or by fine-tuning your model with additional observations that illustrate the answers your expert would provide to similar prompts.
4. Streamline DevOps for GenAI
Last month we described how the traditional DevOps framework must be adjusted to work well for LLM-connected applications. Following this framework will help you deliver and iterate your GenAI product features much faster.
Your Feedback
We produce this newsletter for you! Let us know what you’d like to see more (or less) of. Join our Slack community (click below) or respond to this email with your thoughts.
