Generative AI in Law: Is it Safe to Trust It? (Part I of III)

Young lawyers, myself included , must be aware of the importance of protecting sensitive data in the face of the overwhelming advance of Big Tech and the flaws in the Brazilian legal system itself.

To provide context, the misuse and leaks of data are at the root of several scams plaguing the country: such as the fake lawyer scam , the illegal collection and sale of information , and the use of leaked data by digital gangs . This scenario makes the digital legal security of Brazilians more vulnerable, and makes protecting privacy an unavoidable duty for lawyers.

Within this much-neglected scenario, discussing data protection and privacy not only as a concept but as a right is increasingly important. With this in mind, I researched and compiled some key information on how to deal with generative AI (LLMs, GAIs, and others) within the legal framework, as these are true providers of information.

LLMs are double-edged swords.

The goal of this research is not to cause alarm or spread FUD (Fear, Uncertainty, Doubt). "AI" tools, such as ChatGPT, Claude, and Gemini, based on Large Language Models (LLMs), are not inherently bad enemies; when used correctly, they increase productivity, summarize documents, draft petitions, and even assist in brainstorming legal strategies. But an essential question arises: to what extent can lawyers have freedom and "trust" these tools with confidential information?

To answer this question, I looked for sources both inside and outside of Brazil. And as with much in law, the answer is complex.

What are LLMs and why are they so popular in the legal field?

Simply put, LLMs are language models trained on enormous volumes of text. With this data, they can generate "semi-natural" content, answering questions, reviewing texts, and assisting with legal tasks.

A study by ScienceDirect (“Large Language Models in Law: A Survey”) shows that these systems, when used carefully , can reduce document review time and improve the consistency of legal drafts. However, the same report warns: the models lack real legal understanding , only linguistic patterns, and this directly affects reliability.

In other words, they are great for reducing simple documents, avoiding repetitive legal tasks such as searching for specific legislation, drafting declarations, forms, etc. But they are not true advocates and lack the capacity to genuinely take an interest in defending rights, so there is a clear limit to their use.

Extrapolating this limit carelessly has generated truly macabre spectacles in the judiciary around the world, such as the case in Santa Catarina where a lawyer was ordered by the TJSC (Court of Justice of Santa Catarina ) to pay a fine for bad-faith litigation for applying "non-existent" jurisprudence and doctrines in an appeal, created by hallucinations from the LLM (Law on Mathematics and Statistics) he was using.

Do LLMs collect information?

Right to the point, yes. Companies that offer LLMs collect the information entered by users . Let's look at an excerpt from the ChatGPT Privacy Policy linked above:

Personal Data provided by you: We collect Personal Data if you create an account to use our Services or communicate with us in the following ways:

• Account Information: When you create an account with us, we collect information associated with your account , including your name, contact information, account credentials, date of birth, payment details, and transaction history (collectively, “Account Information”).

• User Content: We collect Personal Data that you provide when you enter information into our Services (“Content”), including your prompts and other content you upload, such as files (opens in a new window), images (opens in a new window), and audio (opens in a new window), depending on the features you use.

• Communications Information: If you communicate with us, for example, by email or through our social media, we may collect Personal Data such as your name, contact details, and the content of the messages you send (“Communications Information”).

• Other information you provide: We collect other information that you may provide to us , such as when you participate in our events or surveys or provide us with information to determine your identity or age (collectively, “Other Information You Provide”).

This highlighted excerpt is only part of the beginning of the privacy policy; the complete document provides more details on how personal data is used, including improvements to Open AI's own services , data disclosure, security, and more. All in a "commercially reasonable" manner, according to the company itself.

OpenAI also states the following:

However, no transmission over the Internet or via email is ever completely secure or error-free . Therefore, you should exercise caution when deciding what information to provide to the Services. Furthermore, we are not responsible for any breach of privacy settings or security measures contained within the Service or on third-party websites.

Translating the terms into plain English, they mean, "We are more than happy to receive and use your information, but any problems, errors, or damage are your responsibility."

For a lawyer, the message is this: "Any damages caused by the use of our services are doubly your responsibility !"

For a trained lawyer, it is not difficult to identify incompatibilities with Brazilian law . Especially regarding abuse according to the Consumer Protection Code and the handling of personal data according to the LGPD (Brazilian General Data Protection Law). These are important topics for future analysis.

The duty of confidentiality and the risk of public clouds.

In Uncle Trump's land, the ABA (American Bar Association), or "US Bar Association," through its ethics and professional responsibility committee, The organization was clear in its first formal guidance on the subject: lawyers must assess confidentiality risks before entering information into AI systems. The ethical duty to protect client data, regardless of the technology used, was acknowledged.

The UK Bar Council has outlined best practices in a very concise and direct way; see our full translation with highlights below:

• Due to potential hallucinations and biases , it is important for lawyers to verify responses generated by LLM -based software and maintain proper procedures for checking outputs produced by generative AI.

• The so-called "black box syndrome" —when it is not understood how the model arrives at a conclusion—shows that LLMs should not replace professional judgment , the quality of legal analysis, or the expertise that clients, courts, and society expect from lawyers.

• Lawyers must be extremely vigilant to avoid sharing any confidential or legally protected information with LLM systems .

• They should also critically assess whether the content generated by LLMs may infringe intellectual property rights and be careful not to use words that could infringe trademarks .

• It is important to stay up-to-date on the relevant Civil Procedure Rules , which may implement specific rules on the use of LLMs in the future — for example, requiring parties to disclose when they have used generative AI in drafting documents , as has already been adopted by the King's Bench Court in Manitoba (Canada) .

Conclusion

The world has changed, and the law has evolved accordingly. Using generative AI makes a lawyer's work simpler, faster, and more efficient, but it comes with a heightened sense of responsibility, as there is a real possibility of harm to the interests and privacy of the data subjects whose information is used. Young lawyers should keep this responsibility in mind from the outset to strengthen their legal practice , and ideally, clearly explain how the collected data will be used.

There is also the risk of more obvious harm, which occurs due to the professional's omission when they fail to carefully analyze the content used by the tool, for which they will always be responsible, such as harm caused by hallucinations and possible unverified information.

In summary: if the lawyer does not have complete control over where the data is processed, they should not enter sensitive information .

Frequently Asked Questions

What is generative AI in law?

Generative AI systems, such as ChatGPT, create legal texts, analyses, and drafts without a genuine legal understanding of the facts.

Is it safe for lawyers to use generative AI?

With caution, yes. Its use must respect professional confidentiality and the LGPD (Brazilian General Data Protection Law), avoiding the inclusion of confidential client data.

What does "LLM" mean and why does it matter?

LLM stands for "Large Language Model," a technology that generates texts. In law, it requires human review to avoid errors and protect confidentiality.

What is the difference between an LLM and a GAI in law?

LLMs generate legal text; GAIs encompass text, images, and sounds. Both require human oversight and adherence to the duty of confidentiality.

Does generative AI collect user data?

Yes. LLMs store prompts, files, and metadata, which can conflict with the LGPD (Brazilian General Data Protection Law) and compromise privacy.

Does the Brazilian Bar Association (OAB) allow the use of generative AI?

The Brazilian Bar Association (OAB) recommends caution. The Brazilian Bar Association (ABA) reinforces that lawyers should assess ethical risks and maintain absolute confidentiality.

Can I incorporate customer information into generative AI?

No. Cloud-based tools can store data on external servers, violating the lawyer's duty of confidentiality.

What are the main risks of generative AI in law?

Hallucinations, data leaks, violations of the LGPD (Brazilian General Data Protection Law), and the ethical responsibility of lawyers for providing incorrect information.

How can generative AI be used ethically?

Review everything, do not include personal data, and inform clients about the use of AI in legal documents.

Could generative AI replace lawyers?

No. She assists with repetitive tasks, but she lacks legal judgment and professional responsibility.

What changes for young lawyers?

They must master generative AI responsibly, combining technology and ethics in the handling of sensitive information.

Bibliographic References

AMERICAN BAR ASSOCIATION (ABA). Formal Opinion 512 – Lawyers' Use of Generative Artificial Intelligence. Chicago: ABA, July 2024. Available at: https://www.americanbar.org/news/abanews/aba-news-archives/2024/07/aba-issues-first-ethics-guidance-ai-tools/ . Accessed on: October 23, 2025.

BAR COUNCIL OF ENGLAND AND WALES. Generative Artificial Intelligence (AI) Guidance for Barristers. London: Bar Council, 2023. Available at: https://www.barcouncil.org.uk . Accessed on: October 23, 2025.

BRAZIL. Law No. 13.709, of August 14, 2018. General Law on the Protection of Personal Data (LGPD). Official Gazette of the Union, Brasília, August 15, 2018.

BRAZIL. Law No. 8,078, of September 11, 1990. Consumer Protection Code. Official Gazette of the Union, Brasília, September 12, 1990.

Federal Council of the Brazilian Bar Association (CFOAB). Recommendation No. 001/2024 – Use of Artificial Intelligence by Lawyers. Brasília: OAB Nacional, 2024. Available at: https://diario.oab.org.br/pages/materia/842347 . Accessed on: October 23, 2025.

LAW SOCIETY OF ENGLAND AND WALES. Generative AI – The Essentials. London: The Law Society, 2023. Available at: https://www.lawsociety.org.uk . Accessed on: October 23, 2025.

SCIENCEDIRECT. Large Language Models in Law: A Survey. Elsevier, 2024. Available at: https://www.sciencedirect.com . Accessed on: October 23, 2025.

OPENAI. OpenAI Privacy Policy (Portuguese version). San Francisco: OpenAI, 2024. Available at: https://openai.com/privacy . Accessed on: October 23, 2025.

COURT OF JUSTICE OF SANTA CATARINA (TJSC). Civil appeal – Bad faith litigation – Citation of non-existent case law produced by AI. Florianópolis: TJSC, 2024. Available at: https://www.tjsc.jus.br . Accessed on: October 23, 2025.