A key challenge is figuring out who will make and enforce the rules of the road. “The most important thing is to have a clear understanding of who is going to take the ball and run with it,” says Tom Davenport, distinguished professor of IT and management at Babson College. “Is it the chief information officer, the chief technology officer, the chief data and analytics officer? Many approaches can work.”

Davenport favors an AI steering committee approach, often with the CIO or CTO leading a group that may also include the chief security officer, a data officer, a risk officer, and representatives from legal and HR. Many organizations also create an AI Center of Excellence, bringing together business leaders and data scientists to determine the best use cases for AI and how to get maximum value.

John Castelly, chief ethics and compliance officer at ServiceNow, agrees that AI governance is a multidisciplinary affair. “It’s most definitely a team sport, and that’s because you can’t afford a bottleneck,” he says. “The speed of innovation requires movement. It requires collaboration, and it requires buy-in. You can’t have one person that’s responsible for understanding and knowing all about the development and the deployment of your AI or GenAI strategy.”

No matter what organizational structure is chosen for governance, it’s essential that it oversees all aspects of scaling, says Isaac Sacolick, founder and president of StarCIO, a consulting company. “If you don’t have a defined process, you’ll probably end up with a shadow AI problem,” where—as with shadow IT—individuals throughout the company take matters into their own hands and start using AI tools without central oversight. “That’s not just inefficient. It has real risk to it.”

Troublingly, more than half of workers admit to using generative AI tools from outside the organization and hiding it from their managers.

“We don’t want to open up a bunch of tools, throw [them] at the organization, and just say, ‘Have at it,’” cautions Sacolick. “Governance policies determine how much structure or how much flexibility you provide to employees to use these LLMs for different processes. Some organizations are going to give access to everybody. Some are going to be more selective.”