Blog

Navigating three privacy pitfalls of AI adoption

Trustworthy AI extends well beyond just privacy, spanning security and ethical considerations as well. But in order to get AI right, you need to first get data privacy right. Learn more about three common privacy pitfalls in AI adoption, and how you can avoid them

Bex Evans
Senior Product Marketing Manager
June 10, 2024

3 privacy pitfalls in AI adoption
Purpose limitation in the context of AI
Proportionality and balance in the context of AI
Business continuity and future-proofing data

Navigating responsible AI: A privacy professional's guide

Download our white paper to learn more about the role of privacy professionals in responsible AI

Download now

Gartner® predicts that “By 2026, more than 80% of enterprises will have used generative artificial intelligence (GenAI) application programming interfaces (APIs) or models, and/or deployed GenAI-enabled applications in production environments, up from less than 5% in 2023”¹

At the same time, according to Gartner Hype Cycle Methodology, “Interest wanes as experiments and implementations fail to deliver. Producers of the technology shake out or fail. Investments continue only if the surviving providers improve their products to the satisfaction of early adopters.”²

Research from Forrester identifies data privacy and security concerns as the top barrier to generative AI adoption. The promotion of trustworthy AI extends well beyond the privacy domain, spanning security and ethical considerations as well. But in order to get AI right, you first need to get data privacy right.

3 privacy pitfalls in AI adoption

That’s because AI is an amplifier of existing privacy gaps – a single misconfigured access point can get exponentially more problematic when its exposed to an AI system. When navigating the role of data privacy for AI systems, there are three privacy pitfalls to be aware of:

Purpose limitation: AI incentivizes secondary uses of data, which means using data for a different purpose than it was originally intended for.
Proportionality: AI requires robust datasets to ensure accuracy, fairness, and quality which may conflict with data minimization practices
Business continuity: Data comes in, but it doesn’t go out – if you don’t put the right data in then someday that system won’t be available for you.

Here, we’ll explore each pitfall in more detail and look at some common practices for addressing the risks associated with it.

Purpose limitation in the context of AI

A common scenario that illustrates the importance of responsible data usage is the collection of birth dates for identity verification. In contexts such as two-factor authentication or banking, verifying a person's identity is crucial. This process often involves collecting sensitive information – like their birth date.

However, possessing this data for identity verification does not automatically grant permission to use it for other purposes. For instance, if a marketing team wants to use birth dates to send out birthday promotions, they must first obtain explicit consent from the individuals involved. Without such consent, using birth dates for marketing is a violation of data privacy principles.

Challenges with generative AI and consent

The advent of large language models (LLMs) and generative AI adds another layer of complexity to this issue.

Providing clear and detailed information upfront is essential for obtaining informed consent, meaning individuals must understand in plain language how their data will be used to provide informed consent.

One significant challenge organizations face is striking a balance between offering enough context and avoiding overwhelming individuals with lengthy terms and conditions that they might – and often do – simply skip. Effective communication in the audience's language is vital to ensure that consent is truly informed and not just a formality.

Proportionality and balancing data robustness and minimization in the context of AI

Consider the use of a resume scanning tool designed to streamline hiring practices. Historically, organizations might exclude sensitive information such as race, gender, and ethnicity from resumes to minimize privacy risks and reduce overall risk if the application were to experience a breach. However, excluding these data points can also prevent the identification and mitigation of bias within the hiring process.

Bias can persist even when sensitive data is omitted, as other indirect factors might contribute to biased outcomes. To accurately analyze and ensure fair representation in a dataset, it’s necessary to document and record sensitive data points. This allows for proactive monitoring of the system for fairness and the detection of potential biases.

A common challenge faced by organizations today is that they may not have collected race, gender, or ethnicity information initially due to privacy concerns or the potential discomfort it might cause applicants. Consequently, they lack the data needed to perform thorough fairness assessments.

Privacy-enhancing technologies (PETs)

To address these challenges, privacy-enhancing technologies (PETs) can be employed. PETs such as differential privacy, synthetic data, homomorphic encryption, and multi-party computation help protect sensitive inputs while enabling the necessary analysis. These technologies allow for the safeguarding of individual privacy during data processing and model training.

However, it is important to recognize that there is no one-size-fits-all solution. The choice of PET depends on the specific use case and the infrastructure in place. In many scenarios, a combination of multiple PETs may be required to adequately protect privacy while maintaining the utility of the data.

Business continuity and future-proofing data in the context of AI

For consent to be truly lawful, it must be freely given and able to be withdrawn at any time. This principle poses a significant challenge when a consumer requests the deletion of their data from a business-critical system. If these processes rely on AI systems trained on personal data, the removal of such data can disrupt business continuity.

AI models, much like human brains, can’t simply forget information once it has been learned. The only solution is to roll back to a previous version of the model, trained before the data in question was included, and then retrain the model without it.

This necessitates robust documentation on model versioning, dataset versioning, and detailed tracking of data categories and identifiers to ensure the data can be accurately removed.

Data governance and model retraining

The complexities associated with enforcing data governance and model retraining highlight the importance of thorough documentation and precise version control. This involves maintaining detailed logs of model versions, datasets, and the identifiers used to track individual data points. When a data subject revokes consent, these records allow for the targeted rollback and retraining of models.

Retrieval-augmented generation (RAG)

Given these data governance challenges, there is a compelling case for employing retrieval-augmented generation (RAG). RAG involves retrieving facts from an external knowledge base to ground LLMs in the most accurate and up-to-date information. This approach offers several benefits:

Flexibility: External data sources used in RAG can be updated without the need to retrain the entire model.
Accuracy: By retrieving information from trusted sources, RAG enhances factual accuracy and significantly reduces the chances of AI-generated hallucinations.
Accessibility: RAG lowers the barrier to adopting customizable generative AI services as it doesn’t require advanced data science techniques.
Specificity: RAG is well-suited for retrieving specific information to provide accurate answers.
Cost: Using external knowledge as a prompt input is more cost-effective than fine-tuning the entire LLM.

By utilizing RAG, organizations can maintain control over the data input at the point of prompt rather than continuously retraining models. This approach helps ensure business continuity and compliance with data privacy regulations, even when individual data points are removed due to withdrawn consent.

¹ Gartner Press Release, “Gartner Says More Than 80% of Enterprises Will Have Used Generative AI APIs or Deployed Generative AI-Enabled Applications by 2026,” October 11, 2023. https://www.gartner.com/en/newsroom/press-releases/2023-10-11-gartner-says-more-than-80-percent-of-enterprises-will-have-used-generative-ai-apis-or-deployed-generative-ai-enabled-applications-by-2026

² Gartner Research Methodologies, Gartner Hype Cycle, June 4th, 2024, https://www.gartner.com/en/research/methodologies/gartner-hype-cycle

Your partner in trust transformation

Resources

Platform

Company

Latest News

Contact Us

Privacy Matters

Our privacy center makes it easy to see how
we collect and use your information.

Your privacy

When we collect your personal information, we always inform you of your rights and make it easy for you to exercise them. Where possible, we also let you manage your preferences about how much information you choose to share with us, or our partners.

Blog

Navigating three privacy pitfalls of AI adoption

Table of contents

Navigating responsible AI: A privacy professional's guide

3 privacy pitfalls in AI adoption

Purpose limitation in the context of AI

Challenges with generative AI and consent

Proportionality and balancing data robustness and minimization in the context of AI

Privacy-enhancing technologies (PETs)

Business continuity and future-proofing data in the context of AI

Data governance and model retraining

Retrieval-augmented generation (RAG)

You may also like

AI Governance

Ensuring compliance and operational readiness under the EU AI Act

Join our webinar and learn about the EU AI Act's enforcement requirements and practical strategies for achieving compliance and operational readiness.

August 22, 2024

AI Governance

AI Governance in action: A live demo

Whether your AI is sourced from vendors and third parties or developed in-house, AI Governance supports informed decision-making and helps build trust in the responsible use of AI. Join the live demo webinar to watch OneTrust AI Governance in action.

August 06, 2024

Third-Party Risk

Third-Party AI: Procurement and risk management best practices

As innovation teams race to integrate AI into their products and services, new challenges arise for development teams leveraging third-party models. Join the webinar to gain insights on how to navigate AI vendors while mitigating third-party risks.

July 25, 2024

Responsible AI

EU AI Act compliance resource kit

Download this resource kit to help you understand, navigate, and ensure compliance with the EU AI Act.

July 22, 2024

AI Governance

From build to buy: Exploring common approaches to governing AI

In this webinar, we'll navigate the intricate landscape of AI Governance, offering guidance for organizations whether they're developing proprietary AI systems or procuring third-party solutions.

July 10, 2024

AI Governance

Navigating the ISO 42001 framework

Discover the ISO 42001 framework for ethical AI use, risk management, transparency, and continuous improvement. Download our guide for practical implementation steps.

July 03, 2024

Privacy Management

Scaling to new heights with AI Governance

Join OneTrust experts to learn about how to enforce responsible use policies and practice “shift-left” AI governance to reduce time-to-market.

June 25, 2024

AI Governance

AI Governance Leadership Webinar: Best Practices from IAPP AIGG with KPMG

Join out webinar to hear about the challenges and solutions in AI governance as discussed at the IAPP conference, featuring insights and learnings from our industry thought leadership panel.

June 18, 2024

AI Governance

Colorado's Bill on AI: Protecting consumers in interactions with AI systems

Colorado has passed landmark legislation regulating the use of Artificial Intelligence (AI) Systems. In this webinar, our panel of experts will review best practices and practical recommendations for compliance with the new law.

June 11, 2024

AI Governance

Governing data for AI

In this webinar, we’ll break down the AI development lifecycle and the key considerations for teams innovating with AI and ML technologies.

June 04, 2024

AI Governance

GRC strategies for effective AI Governance: OCEG research report

Download the full OCEG research report for a snapshot of what organizations are doing to govern their AI efforts, assess and manage risks, and ensure compliance with external and internal requirements.

May 22, 2024

AI Governance

Global AI Governance law and policy: Jurisdiction overviews

In this 5-part regulatory article series, OneTrust sponsored the IAPP to uncover the legal frameworks, policies, and historical context pertinent to AI governance across five jurisdictions: Singapore, Canada, the U.K., the U.S., and the EU.

May 08, 2024

AI Governance

Embedding trust by design across the AI lifecycle

In this webinar, we’ll look at the AI development lifecycle and key considerations for governing each phase.

May 07, 2024

AI Governance

Navigating AI policy in the US: Insights on the OMB Announcement

April 18, 2024

AI Governance

Data privacy in the age of AI

In this webinar, we’ll discuss the evolution of privacy and data protection for AI technologies.

April 17, 2024

AI Governance

OneTrust's journey to AI governance resource toolkit

What actually goes into setting up an AI governance program? Download this resource kit to learn how OneTrust is approaching our own AI governance, and our experience may help shape yours.

April 11, 2024

Privacy Management

OneTrust Data Privacy Maturity Model self-assessment

This self-assessment will help you to gauge the maturity of your privacy program and understand the areas the areas of improvement that can further mature your privacy operations.

April 01, 2024

Scaling to new heights with AI Governance