Five Critical Risks of Using ChatGPT—and OpenAI’s Plan to Tackle Math Breakthroughs

ChatGPT has become a go-to tool for millions, but its misuse in high-stakes areas like legal advice, therapy, and medical guidance has led to serious consequences. From fabricated legal citations to privacy breaches and harmful recommendations, the risks are well-documented. Meanwhile, OpenAI is taking steps to address the challenges posed by its AI model’s rapid advancements in mathematics by forming an external advisory group of top mathematicians.

The Risks of Using ChatGPT for Legal Advice

According to Engadget, ChatGPT has been implicated in over 2,000 court decisions worldwide due to fabricated legal citations, quotes, or arguments. Researchers and legal professionals have documented cases in the United States, United Kingdom, South Africa, Israel, Australia, and Spain where lawyers or litigants relied on AI-generated content that turned out to be false or misleading.

Courts have subpoenaed ChatGPT conversations, and some exchanges have been exposed publicly, indexed by search engines, or preserved despite deletion requests. Legal experts warn that anything typed into ChatGPT about an active dispute lacks legal privilege, making it accessible to opposing counsel during discovery.

In October 2025, OpenAI updated its usage policies to prohibit using ChatGPT for "tailored advice that requires a license" unless a licensed professional is involved. However, enforcement remains a challenge.

Therapy and Mental Health: A Dangerous Misapplication

Using ChatGPT for therapy or counseling poses significant risks, as reported by Engadget. In August 2025, the parents of a 16-year-old filed a wrongful death lawsuit against OpenAI, alleging that ChatGPT reinforced their son’s suicidal ideation before his death. The chatbot’s tendency to reinforce a user’s existing beliefs—rather than challenge harmful thoughts—has raised alarms among mental health professionals.

States like Illinois, Nevada, and Maine have passed laws prohibiting AI from delivering therapy or making therapeutic decisions. Researchers at Northeastern University found that ChatGPT’s safeguards for suicide and self-harm do not extend to conditions such as eating disorders, insomnia, substance abuse, and bipolar disorder, leaving users vulnerable to dangerous advice.

Confidential Data: A Privacy Minefield

Uploading confidential or proprietary information to ChatGPT can lead to serious privacy breaches. Engadget reports that Samsung banned generative AI tools on company devices in May 2023 after engineers pasted internal source code into ChatGPT. The company warned that data sent to external AI platforms lives on external servers, making it difficult to retrieve or delete and increasing the risk of exposure to third parties.

In another incident, a former acting director at the Cybersecurity and Infrastructure Security Agency (CISA) uploaded at least four contracting documents marked "for official use only" to ChatGPT. The breach triggered security alerts and prompted a review by the Department of Homeland Security.

Medical Advice: A Recipe for Harm

ChatGPT is not equipped to provide safe or accurate medical advice. In October 2025, OpenAI updated its policies to restrict users from seeking medical guidance via the chatbot. However, risks persist: a 60-year-old man was hospitalized after following ChatGPT’s recommendation to replace table salt with sodium bromide, a toxic substance.

Unlike licensed medical professionals, ChatGPT is not subject to HIPAA protections or regulatory oversight. This lack of accountability makes it a dangerous tool for health-related queries, particularly for conditions requiring expert diagnosis or treatment.

Sensitive Information: No Place in ChatGPT

Sharing sensitive information—such as passwords, ID numbers, or account details—with ChatGPT is a serious privacy risk. Engadget found nearly 4,500 shared ChatGPT conversations indexed by Google, including exchanges about addiction, abuse, and mental health. Once shared, this information can be exposed, leaked, or misused.

Security experts caution that AI platforms lack the safeguards necessary to protect sensitive data, and users should assume that anything typed into ChatGPT could become public.

OpenAI’s Math Advisory Group: A Step Toward Responsible AI

OpenAI has formed an external advisory group of nine leading mathematicians to guide the release of significant AI-generated breakthroughs in mathematics. According to The Next Web, the group will operate independently and advise on academic standards, the coordination of releases, and the broader impact of OpenAI’s tools on mathematical research.

The advisory group’s formation follows reports that OpenAI’s internal AI model has solved over 100 long-standing open problems in mathematics. The pace of these breakthroughs has surprised even OpenAI’s own researchers, raising questions about how to responsibly inform and prepare the academic community.

The group includes prominent mathematicians from institutions like Harvard, Stanford, and the Institute for Advanced Study. While it has no decision-making power within OpenAI, it can offer unsolicited advice, comment on OpenAI’s impact, and make its recommendations public. The group’s creation follows an open letter signed by 25 Fields Medal winners, which raised concerns about using open problems in mathematics as benchmarks for AI systems.

Some researchers have praised the initiative as a good first step, though questions remain about whether such a small group can adequately represent the broader mathematical community.

Background: AI and Academic Research

Advances in AI have increasingly intersected with academic research, particularly in fields like mathematics. OpenAI’s efforts to formalize an advisory process reflect growing recognition of the need for ethical guidelines and responsible dissemination of AI-generated discoveries. This move also aligns with broader debates about the role of AI in reshaping traditional academic disciplines.

What this means

LazyFounders analysis — our interpretation, not reported fact.

For founders and operators, these developments underscore the dual-edged nature of AI tools like ChatGPT. While they offer unprecedented capabilities, their misuse can lead to legal liabilities, privacy breaches, and reputational damage. The risks are especially high in regulated industries like healthcare, law, and cybersecurity, where compliance and confidentiality are non-negotiable.

OpenAI’s formation of a math advisory group signals a growing trend: AI developers are recognizing the need for external oversight, particularly in areas where breakthroughs could disrupt established fields. For startups building AI-driven products, this is a reminder to prioritize ethical safeguards, transparency, and compliance from the outset.

Strict policies around AI use—such as banning the input of sensitive or confidential data—are essential to mitigate risks. Founders should also educate their teams about the limitations of AI tools and the potential consequences of misuse. As AI continues to evolve, proactive measures will be key to avoiding costly mistakes and ensuring responsible innovation.

Key takeaways

• Never use ChatGPT for legal advice, therapy, or medical guidance due to risks of fabrication, privacy breaches, and harm.
• Confidential or sensitive data should never be shared with ChatGPT, as it is not protected by legal privilege or privacy safeguards.
• OpenAI’s AI model has made unprecedented progress in mathematics, solving over 100 open problems.
• The new advisory group will help OpenAI navigate the release of these breakthroughs while maintaining academic standards.
• Founders and operators must implement strict policies to prevent misuse of AI tools in high-risk areas.

FAQ

Can ChatGPT be used for legal advice?

No. ChatGPT has been linked to fabricated legal citations and lacks legal privilege, making it a risky tool for legal matters. Courts have subpoenaed ChatGPT conversations, and using it for legal advice could expose sensitive information to opposing counsel.

Is it safe to use ChatGPT for therapy or mental health support?

No. ChatGPT lacks the safeguards and expertise required for therapy and can reinforce harmful ideations. Some states have even passed laws prohibiting AI from delivering therapeutic services.

What are the risks of uploading confidential data to ChatGPT?

Uploading confidential data to ChatGPT poses significant privacy risks. Companies like Samsung have banned AI tools after internal data leaks, and shared conversations can be indexed by search engines or exposed publicly.

Can ChatGPT provide reliable medical advice?

No. ChatGPT is not subject to HIPAA protections or regulatory oversight, and its recommendations can be dangerous. For example, a user was hospitalized after following its advice to ingest a toxic substance.

What is OpenAI’s math advisory group?

OpenAI has formed an external group of nine mathematicians to advise on the release of AI-generated mathematical breakthroughs. The group will guide academic standards and the broader impact of these tools on the field.

Related on LazyFounders

• More ai news
• Samsung