Baseten's cofounders are (from left) Amir Haghighat, Tuhin Srivastava, Phil Howes and Pankaj Gupta.
SLCA StudiosCompanies have spent billions on computing costs to train artificial intelligence models, but the costs are starting to shift to when the AI is actually used — say, the compute ChatGPT uses to answer a question in real time. That process, called “inference,” accounted for 40% of Nvidia’s computing revenue last year, the company shared in its latest earnings report. “If Nvidia’s business is 90% training and 10% inference, you could argue that AI is still in research,” CEO Jensen Huang said in a recent Wired interview. Having so much computing power dedicated to inference, he said, shows that “AI is finally making it.”
A flock of startups are in turn rushing to help companies to do inference. The latest to raise major funds is Baseten, which announced Monday a $40 million Series B investment led by IVP and Spark Capital, with participation from existing backers including Greylock, Conviction Partners and angel investor Lachy Groom. The round valued the San Francisco-based startup at more than $200 million, according to a source with direct knowledge of the deal.
Baseten helps customers take open-source or customized AI models that have already been trained, and launch them into applications for public use. Customers can bypass having to set up the backend infrastructure by instead deploying their models on Baseten’s cloud. And Baseten’s software automatically adjusts the amount of computing resources a company is paying for depending on how much traffic is coming to an AI model at a given time. It can also help broker access to GPUs, which are in scarce supply, if a customer needs them.
From its founding in 2019 through the end of 2022, Baseten was one in a sea of startups selling software that made it easier to build AI-powered apps. But they didn’t gain much traction, said CEO Tuhin Srivastava, who cofounded the company with Amir Haghighat, Philip Howes and Pankaj Gupta. Their efforts amounted to revenues of “basically zero,” Srivastava told Forbes.
“2023 made us realize that inference was actually the biggest market here, so we basically chopped everything and refocused solely on it,” he said.
Specifically, Srivastava is referring to last year’s mass excitement around OpenAI's ChatGPT. “The reason why ChatGPT is so good is because it doesn’t feel clunky,” he argued. “Turns out OpenAI has dozens and dozens of people working on inference internally” — making ChatGPT’s response times faster and eliminating issues that cause downtime.
“We are riding on this wave of open-source models getting very good over the last 12 months.”
Baseten’s decision to focus on inference catapulted revenue into the mid single-digit millions. IVP partners Somesh Dash and Shravan Narayen told Forbes they were compelled to invest after seeing this growth occur without a sales and marketing team. Some of the capital injection will go into building up that function, said Srivastava, who envisions close to doubling the current 25-person headcount by the end of the year. Some of the new funding will also be used to buy more GPUs and to optimize the company’s software.
Already, he said, about 20 “big logos” and tens of thousands of developers are paying for the product, which bills based on the amount of computing power used. Patreon, the membership platform where consumers pay creators for exclusive access to videos and other content, uses Baseten to manage its deployment of OpenAI’s open-source speech recognition model Whisper, which auto-generates subtitles.
“We are riding on this wave of open-source models getting very good over the last 12 months,” Srivastava said. But the current boom could also pose a risk factor later on. Baseten’s product depends on customers choosing to use customizable open-source models, despite the extra legwork needed to manage the infrastructure. If off-the-shelf closed-source solutions like OpenAI’s GPT, Anthropic’s Claude or Google’s Gemini come to dominate the market, Baseten could largely be moot. But Srivastava foresees a future where developers will use both open and closed models. Although closed-source services may boast better performance, factors like cost, speed and data privacy will continue to draw startups to open-source alternatives, he said.
Another risk comes in the competition that is heating up among AI infrastructure providers. Spark Capital’s Will Reed, who is joining the board as part of the raise, told Forbes that in his observation, Baseten was winning an “unbelievably disproportionate share” of customers, even though it’s not the cheapest option out there. For example, it powers all the machine learning features in the video and podcast editing app Descript, another startup Reed backed. Srivastava is betting customers will pick Baseten over rivals due to performance — it had zero minutes of downtime in 2023 across the AI apps it powers.
Investors are increasingly backing companies focused on inference: Modal Labs, which received funding from Redpoint and Amplify Partners; Fireworks AI, which is backed by Benchmark and Sequoia; and Replicate, which last year raised $40 million from investors including Andreessen Horowitz. Unicorn startups like Together AI and Anyscale are also entering the space, and Srivastava expects Amazon or Microsoft could be next: “Realistically you have to think about the hyperscalers as well because I think they do see that this is a competitive market.”
