Navigating the New Risks and Regulatory Challenges of GenAI

The rapid rise of generative AI, including large language models (LLMs) such as OpenAI’s ChatGPT/GPT-4, is creating new risks and regulatory challenges for business. Although it is still early days, companies cannot afford to delay developing policies and practices regarding the use of these technologies.

What new risks do these technologies pose, particularly as their users may rely on them for health, legal, or other professional services as well as business decisions? And how can both the developers building these tools and the companies using them assess and mitigate the risks? In this article, we provide some guidance.

Proprietary Information Leaks

Earlier this year, Samsung discovered that its employees had accidentally shared confidential data with ChatGPT — meaning that the company’s proprietary information could be used to further train OpenAI’s model and potentially be revealed to other users. Similarly, with some clever prompt engineering, users convinced Microsoft’s AI-powered chatbot to share information meant to be kept secret. Despite current safeguards in place, it’s clear that LLMs have the potential to pose substantial risk with respect to confidential or sensitive information that passes through these systems.

Addressing this risk requires joint efforts by both users and developers of generative AI tools. For example, guidelines for the prompts employees use as inputs to generative AI tools residing outside the company boundaries must be considered, and tools that alert employees when they are about to send a prompt that may include sensitive company information to a third-party generative AI system may need to be deployed.

Moreover, when a company fine-tunes existing foundation models — that is, core AI models currently mostly available from big tech companies that can be adapted for multiple downstream applications — with their own data, it should double down on data governance and prioritize visibility into the quality and clarity of the provenance of any data used, especially if the company works with external providers to do that fine-tuning.

Companies may also want to consider creating a so-called “sandbox” to allow employees to explore the capabilities of generative AI tools without sharing their prompts or the data with the developers. Harvard University, for example, set up such an environment where users can easily switch between different LLMs through a single interface without having prompts or any data inputs being shared with the LLM vendors. Such approaches are not without trade-offs: For example, not sharing such information with the developers could potentially limit how well the generative AI can be fine-tuned for the specific needs of the company.

Developers of these technologies also need to undertake careful due diligence with respect to both the data and the data providers used to train these AI models. In some cases, this may mean training models only on well-defined data sources and always carefully reviewing the provenance of the data that underlies a given tool or that passes through a tool during its usage.

For example, Getty recently shared plans to develop generative AI tools that would be trained on fully licensed content, enabling the company to ensure that the content creators who own the rights to the images used to train these models are able to get royalties from artificially generated images. Perhaps more importantly, the Getty approach promises those using the system protection from lawsuits claiming copyright infringement on the output of the generative AI system due to the provenance of its training data.

Inaccurate or Harmful Outputs

Generative AI is trained on a given data set, and there’s no easy way to trace back the source of an output to a specific input or to instruct the model to “forget” any problematic, sensitive, or illegal data on which it may have been trained (although exciting new research on how to “dissect” AI models to control their behavior is ongoing). As a result, these tools run the risk of creating outputs that are inaccurate or otherwise harmful, potentially at a substantial cost: When Google’s AI chatbot made a basic mistake in its first demo, the company’s valuation dropped by more than $100 billion. Hallucinations, misleading content, and other factual errors that make their way into LLM outputs range from amusing mistakes to costly and downright costly or even dangerous misinformation.

In response, developers and users alike need to implement monitoring and feedback processes to ensure the quality of the outputs generated by these technologies and to continuously improve them. In addition, companies also need to monitor the quality of the final work produced when employees use these technologies. While use of, say, LLMs can significantly improve the quality of the work — as recent research by a team of people from Harvard, MIT, the University of Pennsylvania, and the Boston Consulting Group shows — quality may actually deteriorate for some tasks. What if, for example, ChatGPT leads to worse employee performance or hurts the quality of decisions and services?

To ensure proper governance, including rigorous monitoring and continuous improvement processes, companies will also have to decide what level of internal transparency is best — or necessary — when using these technologies. There is a spectrum of approaches business leaders can take: from explicitly not constraining (and even encouraging) their employees to use generative AI to defining guidelines for usage (which may be suggestive and unenforceable) to setting up more-heavy-handed processes to detect and regulate usage.

As a side note, a hands-off approach isn’t necessarily a bad idea. In some applications, it may make sense to focus on the quality of the output rather than on exactly how that output was produced. We do not regulate the use of calculators, slide rules, treatises, and other tools; instead, we monitor the quality of the work accomplished with those tools.

Similarly, if it proves possible to put fact-checking or other systems in place to ensure that outputs are accurate, free from hallucinations, and avoid other pitfalls of AI-generated content, then there may not be as strong a need for employees to disclose that an AI tool was used in the course of their work. There are also contextual factors at play: In some applications, occasional errors may be acceptable, whereas in others, there may be limited or no margin for error. The level of risk can determine whether generative AI can be used in specific business cases.

Another consideration that will inform what approach to take is technical limitations. For example, many education leaders have expressed concern that it will be very difficult to detect cheating by students using ChatGPT to write essays or complete assignments. While new tools and training programs are emerging to help people detect AI-generated content, there are many applications in which enforcement of restrictions on the use of generative AI remains a challenge.

In fields like these, it may be especially important to either intentionally take a more hands-off approach (when the risk of harmful outputs is minimal) or to complement technical solutions with other forms of trust signaling and quality control such as certification or audits by reputable third parties (in situations where it’s more important to avoid certain types of harmful content).

Potential New Liabilities

The potential for harmful or inaccurate content in turn drives a whole host of new liability risks when using generative AI tools in business. As tools like GPT-4 demonstrate the ability to pass professional exams and perform certain tasks on par with humans in fields such as law and medicine, they’re increasingly likely to be incorporated into real-world applications. And while this certainly creates new opportunities, it also creates new risk as companies may be held liable for any harmful content or unsafe decisions these tools help them make.

For example, ChatGPT has been shown to be effective at producing first drafts of basic legal documents such as contracts, wills, and complaints. But risks of errors can increase when, say, a lawyer uses ChatGPT to draft a will for a client but does not notice that ChatGPT’s output includes provisions that are barred in the client’s state — meaning that the will won’t be enforceable. Or, say, if a lawyer uses ChatGPT to draft a complaint for a contract dispute and it hallucinates details about the case that are not true. In situations like these, lawyers can be subject to sanctions, including disbarment, and their firm may be subject to legal action.

Similarly, LLMs can be used to help physicians diagnose patients or to help patients learn about medical issues. Patient-facing mental health related chatbots are also hitting the market. But what happens if the chatbot gets it wrong and a patient suffers as a result? Normally, if doctors make mistakes, they can be sued for medical malpractice. But it’s less clear today whether the technology or health care provider would be held legally liable in a case of AI-driven malpractice (such as due to the use of a chatbot) or, for that matter, whether medical malpractice insurers would pay out in the case of such a lawsuit. This is particularly challenging with generative AI as its outputs are not easily traced back to specific data or data providers.

Liability in the medical context is further complicated by the notion of a “standard of care.” In medicine, malpractice is defined as deviation from what a reasonably skilled, competent, and educated medical provider would have done under the same treatment circumstances. Today, that definition would deem excessive reliance on an LLM as a problematic deviation from the standard of care. But there may come a time when the standard of care changes to incorporate some amount of (responsible) generative AI usage, potentially creating legal risk associated with choosing not to use these tools alongside the risks that come with using them.

Professionals and companies will need to consider a number of difficult questions that may arise due to reliance on the output generated by these technologies. What happens if potential licensure issues arise when non-professionals use LLMs to generate professional documents? Some uses of generative AI in law might be considered to be the unauthorized practice of law, leading to sanctions. What role do the users versus makers of generative AI have in policing the boundaries of how these models are used? What about intermediate users such as legal aid services, who make such tools available to clients and/or train those clients on how to best use the software?

To be sure, as there are no cut-and-dry answers, professionals and organizations would be wise to consult legal experts and carefully determine the best way to mitigate liability risk in their unique business environment. Laws are also likely to evolve as these technologies become more widely used, and as new risks — and new legal cases — are identified. But no matter what, providers and users of these technologies need to consider all of these new complex liability issues and either avoid them entirely or take proper insurance and/or risk-mitigation measures.

Regulatory Risks

The speed of innovation is so fast that generative AI applications can violate digital regulations as soon as those come into effect. For example, LLMs and foundation models are already exposing and testing the limits of regulations including the EU’s Digital Services Act, which was recently adopted to ensure trust and safety online, as well as the EU AI Act proposal.

Clearly, regulations governing the use of AI are still evolving. But as these laws expand to encompass new generative AI tools, companies relying on the large-scale generation and sharing of AI outputs may face new regulatory hurdles. For example, on the IP front, many litigations are already pending related to copyright concerns from artists and creators whose content has been used to train these models. Firms using tools built on questionably-sourced data may find themselves unwittingly in violation of copyright and other regulations as these legal structures mature.

In light of this complex and rapidly-changing regulatory landscape, companies should be vigilant about adopting the appropriate protocols and safeguards to ensure that these technologies are used effectively, responsibly, and legally. Any decisions about the use of such technologies — including when, by whom, how, and for what purpose they can be used — need to be made dynamically and at a meaningful level of granularity.

For example, many recent EU laws follow a risk-based approach, imposing increasing constraints on the conditions under which AI can be used depending on the potential risks it can engender. Companies may need to consider similar risk-based approaches, not only taking into the account the risks regulators consider, such as safety or impact on people’s livelihood, but also potential quality, operational, reputation, and other business risks. Moreover, protocols such as reliance on pre-release red-teams and ex-post content moderation can further help anticipate misuse of these tools.

Of course, as regulations evolve, new requirements are likely to arise. In particular, one area in which new laws are still being developed is external disclosure that AI is being used, in line with earlier privacy frameworks such as the EU’s General Data Protection Regulation (GDPR). If your customer service portal uses an LLM-powered chatbot, are you obliged to tell your customers? If the consulting advice you’ve prepared was informed by input from an LLM, do your clients need to know?

The approach to informed consent taken in the medical and legal fields may offer some guidance for businesses in industries where disclosure isn’t yet a legal requirement. In layman’s terms, the legal requirement for physicians is that they must disclose anything that could reasonably influence a patient’s decision to accept a recommended medical procedure. For example, some courts have held that physicians are legally required to tell their patients if a procedure will be performed by a substitute surgeon, since this could reasonably lead patients to change their mind about moving forward with the surgery.

For companies looking to adopt a similar approach, it may make sense to follow a similar principle. For example, if you know that an average consumer’s decision to buy your product or service would be influenced by the knowledge that it uses AI that may pose safety dangers, significant economic costs, or other risks to its users, you should consider disclosing that usage.

For example, the mental health app Koko found that while ChatGPT helped its volunteers write messages faster, the messages were less effective when people knew they were talking to a bot. As a result, the company decided to stop using ChatGPT. But in other cases, the knowledge that generative AI is involved may not have any impact on customers’ willingness to use a product, so there may be less of an ethical or legal imperative to disclose. Companies will need to ensure they stay on top of new regulations as they emerge, but they can prepare themselves today by following the spirit of other existing regulations related to informed consent and other consumer protections.

Competitive Pressure to Get on the Bandwagon

Despite these many risks, as generative AI becomes increasingly commonplace in a wide range of industries, opting not to use it may become increasingly untenable. If using an LLM can save lawyers several billable hours of work, their clients are likely to pressure them to do so — even if the systems in place to monitor these tools’ outputs are still fairly limited.

Indeed, some legal experts are already advocating that law schools should teach students how to use LLMs, arguing that these tools are likely to become an unavoidable component of the legal profession. In many industries, pressure to cut costs and stay competitive may push professionals to adopt these tools before they truly are ready, with insufficient structures to mitigate any substantial risks they may create. As such, companies will increasingly need to consider how to balance trade-offs between the potentially questionable quality of AI tools’ decisions or outputs and the competitive advantages associated with the speed, efficiency, and scale they enable.

The tradeoffs involved in using AI — such as those between explainability and accuracy, or privacy and security — are not new. One of the key tradeoffs of powerful tools like generative AI is between quality and speed. For example, these technologies have the potential to help meaningfully address the public’s enormous unmet civil legal needs. A significant number of middle-income Americans receive no meaningful assistance when facing important civil legal issues such as child custody, debt collection, eviction, and foreclosure. Providing individuals with access to LLMs — for instance to help them draft contracts or wills — could give them a head start before they make use of a limited-time legal aid provider through the “lawyer for a day” programs some courthouses offer. But what is gained in terms of speed and scale may be lost in terms of quality. It may also create externalities: For example, judges may now face seemingly plausible but actually “hallucinatory” legal documents.

Balancing tradeoffs is never easy. Users and developers of these technologies need to decide on minimum quality standards that need to be ensured when generative AI tools are in the hands of people with less training, without excessively sacrificing speed and efficiency. Companies may also need to include proper quality controls and management of breaches of quality standards, possibly in a gradual manner (i.e., potentially being less strict for early proposals and designs and stricter for final products). At the minimum, executives need to determine the relative importance of speed and scale versus quality for each use of generative AI.

Is your business ready for generative AI?

Currently, generative AI is still mostly used when visiting a particular website and offering a prompt in the case of a chat-based LLM or providing a seed image or prompt or both in the case of image-based generative AI. But as major generative AI companies are rapidly pushing towards much more full-scale integration into existing familiar products, we may not be far off from a moment where generative AI will be ubiquitous as, say, how predictive text has become in sending a text message from a mobile phone.

While we are still in the early stages, it is the essential moment to develop an organizational strategy for dealing with generative AI. Executives need to understand both the potential applications of these innovations as well as the new risks they can introduce and adopt tools, processes, and practices that can give their organizations a head start in managing these risks.

Copyright 2023 Harvard Business School Publishing Corporation. Distributed by The New York Times Syndicate.

Explore AAPL Membership benefits.