In recent years, leading artificial intelligence labs and startups have released AI software designed for tasks of ever-growing complexity, including solving PhD-level math problems, reasoning through complex questions step-by-step, and using tools like web browsers to carry out intricate tasks.

The role of AI engineers in making that happen is well-documented—and often well-compensated. But less publicized is the role of a growing army of freelance experts, from physicists and mathematicians to photographers and art critics, enlisted by companies specialized in AI training, itself a multibillion-dollar industry. Those companies say human wisdom is essential to create sample problems, solutions, and grading rubrics that help AI improve its performance in a wide range of fields.

“As long as AI matters, humans will matter,” says Aakash Sabharwal, vice president of engineering at AI training company Scale AI.

Scale AI recently made the news when Meta announced plans to invest $14.3 billion in the company—and hired away its then-CEO Alexandr Wang to lead a new “Superintelligence” lab focused on AI research. But the company remains a top player in the field, recruiting expert AI trainers in a wide variety of subjects and building digital environments Sabharwal compares to “flight simulators for AI,” where humans can help machines learn everything from sending business emails to writing code.

“Way more PhDs”

The modern AI training industry grew out of earlier work to create labeled training data teaching computers to identify objects in photos or spot social media posts in need of moderation. “The early days of how people thought of this industry was what you’d call commodity labeling, like cat/dog, cat/dog, cat/dog,” says Matt Fitzpatrick, CEO of AI training company Invisible Technologies.

More recently, as generative AI models became available, human workers helped steer the software to correctly answer questions about topics like high-school level mathematics and communicate with virtual fluency in a variety of languages. Companies like Scale and Invisible have also built relationships with big businesses to help them fine-tune AI technology that can deliver insights based on their own needs and internal data.

And now, as leaders of AI companies regularly boast of their chatbots’ prowess at tackling advanced math and science problems, human experts are working behind the scenes to test their limits and push their knowledge levels forward.

“You’ve seen a real change in the seniority and expertise set of the expert pools,” says Fitzpatrick. “Way more PhDs, way more master’s [degrees].”

Hyper-specificity

Exactly what training firms provide to AI companies varies from task to task. It can include a mix of AI prompts and ideal answers, rubrics for evaluating AI responses, and corrections to the AI’s current best attempts. Trust is also an implicit part of the product: As Holger Mueller, a principal analyst and vice president at Constellation Research, points out, it has taken some time for big businesses to trust AI companies with their own data—including for fine-tuning purposes. And many AI training companies decline to publicly share a list of clients, citing confidentiality, with even training workers often not told exactly which company’s AI they’re working to improve.

Another big part of what training firms deliver is access to vetted pools of experts, and the promise that they can produce training data in even obscure areas on short notice, which is critical given the AI industry’s pace of growth. It’s not unusual for a client to expect Invisible to line up 50 experts in, say, computational biology overnight, with the expectation they’ll deliver usable training data within a week, Fitzpatrick says.

Despite reports that some AI companies have begun directly hiring experts themselves to train their systems—OpenAI has reportedly hired more than 100 former bankers from top Wall Street institutions to help teach its systems to do at least entry-level financial analysis—Fitzpatrick and other training company leaders say the specialized nature of their work, involving managing both technically sophisticated training platforms and large numbers of human workers, generally makes it hard for AI labs to do themselves.

“The vast majority of our work is hyper-specific experts for short sprints at a time,” he says. “It’s a complicated thing to do in house.”

That means that Invisible, which announced a $100-million funding round in September, along with its competitors, have all also devoted time to building robust recruitment and evaluation pipelines for human experts—often complete with AI tools of their own to speedily screen and onboard those experts and assess their progress. And clients are likely doing their own assessments of the data they get back. It’s not unusual for AI companies to solicit training data from multiple companies and compare the results, Fitzpatrick says.

Intellectual curiosity

The market for freelance experts with the time and knowledge necessary to train AI in obscure fields is, itself, naturally competitive, with training company executives boasting of their expert contractors’ credentials the way college presidents might brag about a new class of elite undergrads.

One AI training company, Mercor, currently has listings posted seeking “recreation workers” at $60 to $80 per hour, a “bilingual Spanish marketing expert” at $20 to $60 per hour, “legal experts” at $90 to $120 per hour, and Ireland-based “general practitioners” in medicine at $160 to $185 per hour, among numerous other listings. And in many areas of knowledge, the bar steadily rises to get assigned to projects, according to Mercor product manager Osvald Nitski.

“Software engineers are now required to have either experience in some niche programming language or incredibly strong scores on competitive coding challenges,” Nitski says. “We’re now sometimes sourcing named individuals, because the bar that needs to be met is so high that there are limited number of people in the world who actually meet it.”

Mercor, which on October 27 announced a $350 million Series C round at a $10-billion valuation, says it pays more than $1.5 million per day to its experts, with average pay above $85 per hour. More than 30,000 experts are signed to Mercor’s platform, according to the company.

And while the pay no doubt motivates experts, many of whom are working full-time in their fields, to enter the AI training arena, some are also motivated by intellectual curiosity and the desire to help hone software they hope can one day assist in their work or tackle outstanding problems.

“A harder problem”

Alice Chiao, an emergency medicine physician who serves as an expert for Mercor, says she hopes that AI can automate some of the drudgery, like charting and scribing, of medicine and thus help doctors better connect with their patients. She says her AI training work includes asking the systems to answer medical questions that may have stumped her in her practice—the kind of puzzling scenarios that pop up when real-world patients differ from textbook examples.

“We input these things and try to see where a model might fail,” she says. “And then we create an ideal response—you know, based on this finding, I would have ranked this differential diagnosis higher.”

Chiao emphasizes that she doesn’t anticipate the AI she trains replacing her or her fellow physicians. Rather, she sees the technology’s assistance helping to restore a level of human interaction that’s often disappeared from medical practice. “I do not think that training the AI is training a replacement,” she says. “I think that it has the significant potential to enhance the patient-physician relationship, which has eroded to a point where most physicians are not happy with the quality of patient-physician conversation and dialog that they get anymore.”

Another AI training company, Micro1, focuses on talent in finance, medicine, law, and engineering fields, says founder and CEO Ali Ansari. Average rates paid by the company hover around $100 an hour, though it varies between subject areas, with about 70% of experts making between $70 and $210 per hour, he says.

Micro1, which in September announced a $35 million Series A round at a $500-million valuation, also operates an AI recruiter that can vet potential candidates and even help share job listings on platforms like LinkedIn. Finding talent is a critical part of its operations. Part of the company’s goal, Ansari says, is to make sure that in-demand experts will not only perform well at a certain training task but have a good time doing it.

“We want to be able to predict how much an expert will enjoy a certain job as well, which is, in fact, actually a harder problem,” he says.

A social calling

One expert that works with Micro1 is Mark Esposito, a professor of economics and public policy at Harvard University. He now serves as Micro1’s chief economist but began his association with the company by training AI to answer policy questions. That’s something he sees as important in ensuring AI doesn’t give misguided advice to users looking to make important decisions.

“You don’t want any policymaker to be dealing with information that is grossly incorrect,” he says. “So that’s why I think there’s a bit of a social calling for this, in making sure that you’re really training models ethically, because they might really help people make a decision in the real world.”

Edwin Chen, CEO of AI training company Surge AI, speaks to a similar sort of calling, saying he’s dreamed of helping craft artificial general intelligence (AGI)—essentially, truly thinking machines—since he was a child. “Instead of playing the startup game, we’re a lot closer to a research lab,” he says. “And the only thing that matters to us is whether we succeed in building AGI.”

Still, the company recently boasted it’s making more than $1 billion in annual revenue, and Chen says pay rates for some of the experts it works with can reach as high as $500 per hour, with the company website citing contributions by Supreme Court litigators, Oxford linguists, Navy SEALs, and Olympic athletes.

The proportion of specialized experts among the AI training workforce has grown over time, but people with more general knowledge still contribute as well, Chen says. That’s unlikely to change, he adds, since AI tools do need steady training on even basic problems.

And even if AI continues to do better with hard problems and take on more roles, it’ll still need human guidance, as standards for its performance will continue to grow. This means AI is unlikely to make its human teachers obsolete any time soon.

“As they get used more and more, the capabilities increase, the applications increase, and so it’s no longer okay for models to be at 80% accuracy—you need to be at 99.99999% or whatever it is,” Chen says. “And at the same time, as the models get smarter and smarter, you always need humans to steer and align them.”

Historically, some AI training companies have faced complaints from less-expert training contractors of unpredictable hours, difficulties getting paid, and other issues familiar to gig economy workers. Others, including Scale AI, have since said they’ve taken steps to address complaints or otherwise emphasized their commitment to fair pay.

And in general, work in the AI training field seems likely to grow as long as businesses continue to invest in deploying AI, as training companies expand into enterprise work and even work with robotics companies to help AI understand how to move about in the physical world.