Token efficiencies: what enterprises need to know about AI use

There is an under-recognized risk associated with AI use in enterprise that has nothing to do with compliance, legal, or hallucinations. That risk is token use and massive cost overruns. There are several prevailing trends happening in the market right now that could potentially put most organizations at risk when it comes to frontier model use. In this brief post, I'll break down a few of those trends and how you can use efficient infrastructure software like Adaly to mitigate those risks. 

1. Subsidized tokens are going away. Current frontier models are massively subsidized by the huge investments that venture firms and other independent entities invest. These capital requirements of the frontier models, eventually need to shift from investor subsidies to positive unit economics. This means price increases.
2. Context is only getting bigger. The tendency in enterprise is to focus on massive data mobilization to make those vast troves of signals available to different applications. This originated with on-premise data centers for companies and quickly moved to cloud compute and data warehouses and lakes. The problem with a strategy for a probabilistic technology like deep learning is that more context is not always better. In fact, more context often leads to imprecise predictions from these systems. Beyond the imprecision of deep learning systems as they relate to more data, forcing AI systems to sift through more context creates massive token usage, both for inference and training (when applicable).
3. We're shifting from human use of AI systems to agent use. Through workflow automation using deep learning systems like those with the frontier model, companies' velocity and scale of data use and transport are exploding. Without the right human filters and discretion, agentic functions are sending more and more data to and from these frontier models, racking up massive costs. There are already numerous public examples in the market of agentic use blowing through IT budgets in 1/12th the time. 

One of the approaches that can help to address these risks is through efficient understanding of data retrieval and use. By only mobilizing the specific data that's required in real time and passing that context to the models for use, the efficiencies that can be gained here are material. Adaly uses an atomic understanding of what data is being requested and selects only the data needed to answer a particular business question, making it available for use in frontier models. This software infrastructure layer is critical for enterprises who are looking to mitigate the risks associated with massive cost overruns. While it doesn't address the subsidies mentioned above, it definitely addressed the ability to focus on the appropriate context and make agentic use highly efficient through proper selection and retrieval.     

As enterprises continue to invest in artificial intelligence, it's essential to understand the token economy. Without proper infrastructure, the path to positive returns on investments associated with AI will be very, very limited. With Adaly, we hope to address this enterprise risk head-on.