AI functions in a nutshell
In early 2023, we started to see the emergence of “AI functions”, such as this post by Databricks. Although the exact definition is still being crystallized, we wanted to explain the high-level concept of AI functions and the motivation behind them.
In Python, functions are fundamental building blocks of software that can be characterized as 1) formally-defined (in input, output, and behavior), 2) modular, 3) easy to implement and invoke, 4) having variable time and memory complexity.
Large language models (LLMs) on the other hand are 1) ill-defined (prompt wrangling is required and outputs can be very different); 2) particularistic (high inter-model variability); 3) difficult to call in-code; 4) constant execution latency and memory consumption. However, they are incredibly powerful and can perform difficult tasks without requiring direct implementation details from the application developer i.e. classifying 1000s of emails; generating poems; extracting key insights from a Tweet.
AI functions are a way of combining the structure and definition of classic functions with the power and versatility of LLMs, enabling developers to create AI-native software. In practice, an AI function is the regularization of input and output to an LLM, so that such a function can be called in code and behave just like a normal function, but without any code supplied by the developer.
In another framing, AI functions are the interface between Software 1.0 and Software 2.0 as coined by Andrej Karpathy (had to reference one of the OGs :))
If that was a lot, don’t worry. In this article, we’ll break down the differences between AI functions and traditional ones, discuss some examples of real-world applications, and how we aim to make them better.
AI functions vs. traditional functions
The key differences between AI functions and traditional functions can be summarized below:
From the above, you might be wondering - wait, if an AI function has constant memory usage and running time, why don’t we just use them instead of traditional functions if we want to build predictable applications?
Well, it's because they are eye-wateringly expensive to run. The LLM for AI functions costs about 1-2 FLOPs per token per parameter. This means that to classify a single sentence using GPT4 as a backend into ‘good’ or ‘bad’ would roughly use all of the computational power of an iPhone 14 (or 1 million times the power of the computer that got humans onto the moon, or the most powerful computer in the world in 2003 - 12 Teraflops. All to process a single sentence).
In addition to sheer cost, they also behave as blackboxes, which means they can behave in unexpected ways, creating a challenge for integrating them into your codebase, which is highly particular about what it expects. Javascript, for example, is commonly derided for being bad at this - giving rise to the new language of Typescript to address these flaws.
In order for AI functions to be incorporated into applications, these challenges must be addressed. AI functions must: 1) generate typed outputs from typed inputs; 2) execute predictably; and 3) run fast and cheap (<1 second in most cases - similar to serverless functions like AWS Lambda).
These challenges are exactly why we started working on Tanuki. We wanted to make it easy for any developer to create and call AI functions that are:
- Well-typed
- Reliable in performance
- Fast
- Cheap
- Modular
- Testable
We’ll release an article going into detail of how we accomplish the above. If you’re curious and impatient, feel free to dig into the library here, which includes the technical details on how we achieved the above.
Use-cases for AI functions in the real world
If you’ve read the above and thought, “cool, so what?”, here are some examples of AI functions being used within real companies and applications:
Classifier for customer support prioritization
A Series C company in San Francisco had trouble resolving customer issues across email and chat and needed a way to classify each issue by urgency. Initially, keywords didn’t work and GPT4 was unreliable. Therefore, a backend engineer used Tanuki to create a classifier to label tickets as “high”, “medium”, or “low” with several assertions to align the AI function.
Extracting Salesforce data from sales calls
A Series B company in New York wanted to automatically update Salesforce fields after every call so their sales team always knew what deals were progressing. Instead of building an entire LLM model from scratch, their engineers wrote an AI function that took call transcripts as inputs and extracted the key fields they needed.
To-do list builder
A freelance developer wanted to build an app that allowed users to turn their voice recordings and text into a structured todo list with 1) the action item, 2) deadline, 3) priority, and 4) participants involved. As users can be verbose and rambled, it was difficult to use traditional functions with all of the potential edge cases. This where using an AI function to retrieve the correct components in well-typed outputs allowed them to be fed into their todo list app in seconds.
What’s next
We’re still early in the development of AI functions and can’t wait to see how they evolve to serve additional use-cases and work with other tools.
At Tanuki, we’re currently working to improve reliability and make AI functions more powerful, cheaper, and faster. If you’re curious or want to contribute, please join us in our Discord!
Sources
Introducing AI Functions: Integrating Large Language Models with Databricks SQL