It takes fewer than five minutes into an interview with Tom Siebel before the tech billionaire begins to raise the alarm about artificial intelligence’s many risks.
“The way war is being reinvented, all of these new technologies are highly dependent on AI,” says Siebel, CEO and founder of enterprise AI software company C3.ai, in response to just the second question from Fortune.
The company provides AI applications to oil and gas companies ranging from Shell to Baker Hughes, as well as the U.S. defense and intelligence communities. But C3.ai won’t do business with nations that aren’t allies of democratic states, including China and Russia, because Siebel says “they will misuse these technologies in ways that we can’t possibly imagine.”
MIT Sloan Management Review and Boston Consulting Group recently assembled a panel of AI academics and practitioners and the final question they asked was: “As the business community becomes more aware of AI’s risks, companies are making adequate investments in responsible artificial intelligence.” Eleven out of 13 were reluctant to agree.
AI is changing how humans work, socialize, create, and experience the world. But a lot can go wrong. Bias in AI is when decisions are made that are systematically unfair to various groups of people. Critics fear bias can especially harm marginalized groups. Hallucinations perceive patterns that are imperceptible to humans and create inaccurate outputs. And drift is when large language models behave in unpredictable ways and require recalibration.
Americans are worried. A Pew Research Center survey this summer found that 52% were more concerned than excited about the increased use of AI. In the workplace, Americans oppose AI use in making final hiring decisions by a 71% to 7% margin. Women tend to view AI more negatively than men.
Yet it isn’t all doom and gloom. Half of organizations say risk factors are a critical consideration when evaluating new uses of AI. A large share of Americans do not believe the use of AI in the workplace will have a major impact on them personally.
“You should never have as a goal automating away a bunch of workers,” says Steve Mills, BCG’s chief AI ethics officer. “It is about how you pair people with AI to make their job better, allow them to do more, and take advantage of human creativity and ingenuity.”
AI works best when there’s human oversight. Many companies say their employees stringently review the AI models they create or use and that processes are put into place to ensure privacy and data security. Tech giants publicly share their responsible AI ethos to ease worries about the quickly evolving tech.
“I’m optimistic that AI will be a collaborative tool that workers will use,” says Todd Mobley, who represents employers in litigation for DLA Piper. “But it should be an ongoing, iterative process with discussions, training, and testing of the tools to ensure they are being used for the appropriate reasons. And that the tool is not creating unintended consequences.”
“We think AI can be very powerful for automating a lot of tasks, even automating decisions, but at some point you want a human in the loop to validate how decisions are being made,” says Rob Thomas, senior vice president of software and chief commercial officer at IBM.
Thomas stresses that there must be transparency to how AI is being built and where the data comes from. Regulation should oversee the use cases for AI, but not the technology’s development. And governance is critical to understand how models are performing.
To that end, before the end of this year, IBM will make watsonx.governance [sic] available to help businesses monitor and manage their AI activities, and employ software automation to mitigate risk, manage regulatory requirements, and address ethical concerns. “Our intent is to deliver software and capabilities that enables any company to deliver trust in AI,” says Thomas.
In September, German software giant SAP debuted a new generative AI copilot called Joule, which is being embedded in applications ranging from supply chain, to finance, to procurement. Users of some SAP products can ask a question or propose a problem and receive AI answers drawn from business data across SAP’s portfolio and third-party sources.
Thomas Saueressig, who runs SAP’s product engineering team and ethical AI development efforts, says it is critical to acknowledge that bias does exist in large language models and that SAP puts resources into mitigation efforts to ensure Joule’s prompts avoid bias. Saueressig says it is “absolutely essential” that AI development is human centered. “We believe it is a duet, not a duel.”
Every SAP employee has signed the company’s AI ethics policy since the beginning of 2022. “We have a very clear focus on the value of data protection and privacy,” says Saueressig.
Tony Habash thinks AI will drastically change how therapists practice psychology. As chief information officer at the American Psychological Association, Habash sees beneficial uses ranging from AI-powered note-taking, to providing treatment indicators for the therapist to use to improve care. There’s also potential for AI to advance medical research and make healthcare more accessible by lowering costs.
“We think the biggest change ahead of us is the human-machine relationship,” says Habash. Historically, humans had to learn programming languages, like Java, to tell a machine what to do. “And then we wake up and the machine is speaking our language with generative AI.” Habash says this raises ethical questions about how humans can create the confidence, best practices, and guidelines for the machine-human interaction.
“A clinician that is working with an AI system will need to clearly understand how it is working, what it is built for, and how they use AI to improve the quality of health care service and ensure the well-being of the patient,” says Sunil Senan, senior vice president and global head of data, analytics, and AI at Infosys.
There are ways AI can help humans behave more humanely. Take the example of Match Group’s “Are You Sure?” feature. Using AI, Match’s Tinder dating app is able to identify phrases that could be problematic and flag to the sender that the message they are about to send could be offensive. Similarly, “Does This Bother You?” is a prompt message receivers can use to flag harmful language to Match.
These features have reduced harassment on the app, and Match has expanded “Are You Sure?” to 18 languages today. “The people who saw this prompt and changed their message were less likely to be reported,” says Rory Kozoll, senior vice president of central platform and technologies for Match Group. “These models are really impressive at their ability to understand nuance and language.”
When Microsoft’s venture capital fund M12 looks to invest, it looks for startups that can deploy technology responsibly. To that end, M12 invested in Inworld, which creates AI-powered virtual characters in role-playing games that aren’t controlled by humans. These virtual conversations could go off the rails and become toxic, but Inworld sets strict guardrails about what can be said, allowing it to work with family-friendly clients like Disney.
“I’m always going to feel better sleeping at night knowing that the companies we’ve invested in have a clear line of sight of the use of data, data that goes into the model, the data the model is trained on, and then the ultimate use of the model will abide by guidelines and guardrails that are legal and commercially sound,” says Michael Stewart, a partner at M12.
Ally Financial, the largest all-digital bank in the U.S., uses AI in underwriting and chatbots. When deploying AI, Ally experiments with internal customers and always has a human involved. Sathish Muthukrishnan, Ally Financial’s chief information, data, and digital officer, says the AI tech used in chatbots must talk exactly like an associate would engage with a customer.
“AI models are learning how to avoid bias, and it is our responsibility to teach that,” says Muthukrishnan.
Some AI startups are emerging that aim to establish more trust in the technology. In September, Armilla AI debuted warranty coverage for AI products with insurers like Swiss Re to give customers third-party verification that the AI they are using is fair and secure. “A model works really well because it is biased,” explains Karthik Ramakrishnan, cofounder and CEO of Armilla AI. Bias, he explains, isn’t a negative word because AI must be trained on data to think a certain way.
“But what we are worried about is how the model treats different demographics and situations,” says Ramakrishnan.
Credo AI is an AI governance offering that automates AI oversight, risk mitigation, and regulatory compliance. The software provides accountability and oversight to a client’s entire tech stack, helping clearly define who is reviewing systems; is there an ethics board and who is part of it; and how are systems audited and approved.
“Enterprise leaders don’t even know where AI is actually used within their own organization,” says Credo AI CEO and founder Navrina Singh. “They have records of models but they don’t have records of applications.”
At C3.ai, Siebel remains worried about AI landing in the wrong hands. Privacy is his biggest concern. Siebel says society should be aligned that AI shouldn’t misuse private information, propagate social health hazards, interfere with democratic processes, or be used for sensitive military applications without civilian oversight. As for bias, that’s a more difficult nut to crack.
“We have hundreds of years of history in Western civilization,” says Siebel. “There’s nothing but bias. Be it gender, be it national origin, be it race. Tell me how we are going to fix that? It’s unfixable.”
