As a reader of the ISACA® Journal, you might have naturally noticed some cognitive dissonance happening-at least in one very specific area. What I mean is, on one hand, generative artificial intelligence (AI) is everywhere in the mainstream media and seemingly on everyone’s lips. Tools such as OpenAI’s ChatGPT, Google’s Bard, GitHub’s Copilot and others are everywhere. They have been covered extensively in the news and media, they have had a tremendous impact on sectors such as education (e.g., as students find new use cases for how AI can help them with their work), publication (e.g., as publications and authors use them to generate content), marketing and numerous other areas. Yet, professional guidance for practitioners-particularly trust practitioners-has not been abundant in the trade media.
It seems as though there is a veritable seismic shift happening on the technology landscape, yet many of us in the trenches are left wondering how to address it-and there seems to be relatively little guidance available to help us. Why is that?
There are two things happening, I think. One is that questions about risk often take more work to answer than questions about usage-in other words, it is easier to understand how to use something than it is to evaluate how risky it is to use. For example, compare what you need to know to answer the question, "How do I drive this car?" vs. what you need to know to answer the question, "Is this car safe to operate?" For usage, there is a set of critical information items: the rules of the road, how to operate the specific vehicle’s controls, etc. For risk, however, you need to know the answers to most (if not all) of the usage-related questions plus many other things, such as road conditions (and by extension the planned driving route); the vehicle maintenance history; the weather forecast; and the condition of the vehicle’s safety features, such as brakes, seatbelts and airbags. This is to name just a few.
Given this, I think it is natural that we would see usage emerge before any detailed analysis of risk-and ways to address those risk areas-becomes well known. Anyone who remembers the rise of virtualization, mobile, cloud or even (to go very far back in history) desktop computing can recognize the pattern where usage is already well under way before the full risk picture is known.
The second thing happening is that generally accepted safe practices for usage of these tools is taking time to emerge. If you are in the business of building these tools, there is some very exciting work happening. For example, the Open Worldwide Application Security Project (OWASP) is working on a Large Language Model (LLM) Top 10,1 the US National Institute of Standards and Technology (NIST) has draft guidance out (NIST AI 100-2e2023),2 and the UK National Cyber Security Centre (NCSC) has authored a set of principles about securing AI.3 The point is, if you are a developer or integrator of AI (particularly LLMs) there are multiple sets of robust (though nascent) standards and guidance out there. But what if you are not? What if you are just someone who wants to make sure their organization is protected when business units, individual teams, or users themselves use these tools in novel and unexpected ways? There is quite a bit less to go on here-less advice from peers, less guidance from authorities, and so forth.
In just the past year or so, we have seen integration of AI functionality into search engines; business applications such as sales tools, collaboration, and messaging platforms; and numerous other places.
It is worth exploring some things that practitioners might keep in mind as they evaluate where they adopt, how they might already be using these tools without the knowledge of trust practitioners, and what options they might consider in response. This discussion does not go into the nitty gritty of how AI generally and LLMs specifically are developed or how they operate. Interesting though these details are, they can be a bit of a distraction from the impacts to an organization’s digital trust. Instead, this discussion focuses on the emerging areas where the presence of these tools impacts an organization’s digital trust posture and offers some suggestions for what might be done in response.
Risk Considerations
A few quick caveats:
- The focus here is on LLM generative chatbot-style applications (compared to, for example, image generation).
- What is covered is not intended to be exhaustive. This is not intended to be a full and complete list of every possible thing that you would need to consider in your organization or every potential risk area.
- Although there are organization-specific factors such as business context, regulatory considerations, risk tolerance and numerous other factors that play a role in what does or does not pose risk to an organization, as with any set of potential risk areas, only some are probable. There are any number of less probable situations that could arise in a particular situation.
- Addressed here are only those areas that are likely to occur, that impact a large segment of organizations (i.e., that are close to being universally applicable), and that impact digital trust.
- Discretion and good sense must be used in evaluating what may or may not apply to an organization and which areas of risk apply to that organization based on context and unique organizational factors.
- Remember that this discussion is only looking at the usage side of the equation (i.e., by end users). If an organization is a developer of an AI-based tool, an integrator of those tools, or otherwise making use of AI, there are, of course, numerous factors to consider beyond what is provided here.
Caveats out of the way, there are several potential risk areas that are worthy of consideration.
Exposure of Intellectual Property
Perhaps the biggest elephant in the room and the issue that many are most concerned about is exposure of sensitive data. This can and does happen. Samsung, for example, recently banned the use of AI chatbots due to the exposure of proprietary intellectual property by engineers and staff using these tools.4 On multiple occasions, employees shared source code for error checking or code optimization purposes, and in another instance, an employee shared the details of a meeting to help with creation of a presentation.5 Because the tool in question uses submissions from users to help train the model further, this means that intellectual property becomes available beyond the business need to know. It is unknown how extensive this problem is; however, some data, such as research from the vendor community, suggest that the sharing of confidential intellectual property might have already been done by more than 4 percent of the workforce.6
Shadow Adoption
On the surface, a complete ban might seem like a viable option given the size of this trend (imagine, for example, the chaos that would ensue if more than 4 percent of users put sensitive information at risk via some other method, such as installing malware). It is made more challenging in practice, though, as a result of the second consideration: the adoption dynamics-specifically, the potential for shadow adoption. Not only are there multiple services and tools that users might wish to utilize in support of their daily work (a situation that always tends to compound shadow usage), but there is also a rapid- fire series of integrations underway. In just the past year or so, we have seen integration of AI functionality into search engines; business applications such as sales tools, collaboration, and messaging platforms; and numerous other places. This means that trying to technically enforce usage restrictions can be hard to do (especially when a service you already use opens up chatbot access via an integration on its side). And reliably detecting when and if data or information is exposed is likewise difficult for the same reasons.
Because LLMs provide the text that is statistically most likely to occur in any given circumstance with no awareness of how valid that text might be, it is sometimes blatantly erroneous with very little to indicate that this is the case.
Reliability and Provenance
The last consideration to be cited here relates to the reliability and provenance of the information obtained. It is concerning that users often place higher confidence in the reliability of the responses they get from these tools. Called "hallucinations," LLMs often provide inaccurate information or completely nonexistent "facts." Because LLMs provide the text that is statistically most likely to occur in any given circumstance with no awareness of how valid that text might be, it is sometimes blatantly erroneous with very little to indicate that this is the case. Many might be familiar with the reporting around attorney Steven A. Schwartz, who used ChatGPT to help prepare an official legal document. The LLM cited precedents that were entirely fabricated as "hallucinated" by the model. The judge in the case said, "six of the submitted cases appear to be bogus judicial decisions with bogus quotes and bogus internal citations."
Putting accuracy aside, though, provenance is also potentially at issue. Because the model consumes human-produced content exemplars and uses that as the basis for its responses, the ideas and concepts it relays (definitionally) were originated by others. Those ideas and concepts are then provided to users without attribution. If that sounds close to the plagiarism line to you, you are not alone in thinking so. OpenAI, the maker of ChatGPT, is already being sued7 related to its use of data (per the lawsuit, using "stolen private information") in the training of the model.
Addressing these Concerns
All of these areas are certainly ones about which practitioners tasked with ensuring digital trust in our organizations might be concerned. However, controls in this arena are still emerging, and generally accepted standards around how to minimize risk while still ensuring that users get value from these tools (and remain competitive) are also not yet fully baked. So what then can we do?
One option is, like Samsung, to limit usage until the impacts can be better and more thoroughly understood. However, for reasons described herein, this can be technically challenging to enact given the rush to integrate LLM functionality into existing tools (even search engines and the like), the hype in the marketplace about their utility (and, thereby, the increased desire on the part of users to make use of them), and the number of different choices and offerings that users can select from (with new ones being introduced every day).
Because it can be challenging to directly restrict access for users (not to mention that these tools can be valuable in the workplace when used with discretion), one might consider alternative approaches. For example, one might consider awareness training designed to highlight to users the potential privacy, security, assurance and other digital trust impacts associated with these tools along with guidance about how they can be safely used. For example, one might consider training users on not sending sensitive, regulated or other critical information to these services. Users might be instructed on the limitations of these tools, such as AI "hallucinations" (misinformation) and other limitations on what the tools can deliver. Users might be counseled on the need to verify LLM output to ensure that output from these tools is not someone else’s work being repurposed in a nonattributed way.
It also goes without saying that it can be advantageous to know where these tools are being used and for what use cases to understand the surface area of potential exposure. Organizations might, for example, maintain a record of usage when conducting activities such as business impact analysis (BIA), when performing assessments and/or audits of specific business areas and so on. Technical monitoring tools might be leveraged if organizations have them (e.g., HTTP forward proxy logs, network traffic logs, shadow IT monitoring tools) to look for areas of usage and enable follow-up.
LLMs and other AI tools have tremendous promise and potential. But like any new thing, care and forethought are required to ensure that organizations optimize risk while maximizing value.
Endnotes
1 The Open Worldwide Application Security Project (OWASP), OWASP Top 10 for Large Language Model Applications, USA, https://owasp.org/www-project-top-10-for-large-language-model-applications/
2 Oprea, A.; A. Vassilev; Adversarial Machine Learning, National Institute of Standards and Technology, USA, 2023, https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-2e2023.ipd.pdf
3 National Cyber Security Centre, Principles for the Security of Machine Learning, United Kingdom, 2022, https://www.ncsc.gov.uk/collection/machine-learning
4 Mauran, C.; "Whoops, Samsung Workers Accidentally Leaked Trade Secrets Via ChatGPT," Mashable, 6 April 2023, https://mashable.com/article/samsung-chatgpt-leak-details
5 Staff, "Concerns Turned Into Reality... As Soon as Samsung Electronics Unlocks ChatGPT, 'Misuse’ Continues," The Economist (Korea), 30 March 2023, https://economist.co.kr/article/view/ecn202303300057?s=31
6 Lemos, R.; "Employees Are Feeding Sensitive Biz Data to ChatGPT, Raising Security Fears," Dark Reading, 7 March 2023, https://www.darkreading.com/risk/employees-feeding-sensitive-business-data-chatgpt-raising-security-fears
7 Cerullo, M.; "ChatGPT Maker OpenAI Sued for Allegedly Using 'Stolen Private Information,’" CBS News, 30 June 2023, https://www.cbsnews.com/news/chatgpt-open-ai-lawuit-stolen-private-information/
ED MOYLE | CISSP
Is currently director of Software and Systems Security for Drake Software. In his years in information security, Moyle has held numerous positions including director of thought leadership and research for ISACA®, application security principal for Adaptive Biotechnologies, senior security strategist with Savvis, senior manager with CTG, and vice president and information security officer for Merrill Lynch Investment Managers. Moyle is co-author of Cryptographic Libraries for Developers and Practical Cybersecurity Architecture and a frequent contributor to the information security industry as an author, public speaker and analyst.