Lentis, Chatbots as Therapists

Introduction

AI chatbots are increasingly being used for mental-health support. Therapy costs money, scheduling can be hard, there is still a social stigma associated with it. A chatbot is different. It is private. It is cheap. And it is available all the time. That makes it useful for people with various mental illnesses who are unsure where else to go.

There is a problem with this, however. Easy access does not necessarily mean good care. A chatbot may sound professional, but that does not make it a therapist. It does not have professional judgement, the full context of the user's life, or the ability to read social cues. These tools should therefore be judged by how they handle risk, dependency, privacy, and escalation.

History of Computer-Based Therapy

Computer-Based Therapy is not new. Joseph Weizenbaum created ELIZA in the 1960s. Eliza imitated a Rogerian psychotherapist. It did not actually understand the user. It mainly turned the user's own words back into questions. Still, people responded to it as if it understood them. Weizenbaum argued this was dangerous because the program looked like it cared without caring or actually taking responsibility. [1]

Computer-based therapy also came out of a larger change in mental health diagnosis. Mental health disorders became standardized through systems like the DSM. Mayes and Horwitz argue that DSM-III changed psychiatry by moving diagnosis toward symptom-based categories and away from older psychoanalytic explanations. [2] This matters because computers work best with structure rather than qualitative judgement.

A later example is Ellie, the virtual psychotherapist used in the SimSensei system. Ellie was designed to support screening, not to replace therapists. [3] [4] It could use speech, facial expression, posture, and other cues. This made sense in military healthcare where stigma is a major issue. RAND found that about one in five Iraq and Afghanistan veterans had PTSD or major depression. [5]

Adam Raine and AI Chatbots in Mental Health

The Raine v. OpenAI case highlights how AI chatbots can be risky for mental health. Adam Raine, a teenager, used ChatGPT outside a clinical setting, with no supervision or crisis response. The complaint says he interacted with the chatbot often, became emotionally dependent, but there was no single harmful message. [6]

The key issue is interaction dynamics over time. The filing alleges that design choices enabled ongoing attachment, stating that “OpenAI’s executives knew these emotional attachment features would endanger minors… but launched anyway” [6]. This shifts the analysis from output quality to system incentives. Engagement optimized systems reward continuation, not disengagement, which conflicts with crisis intervention norms that prioritize interruption and referral.

The trajectory of responses is also critical. The complaint describes a progression from general support to a suicide-related discussion [6]. This pattern reflects known failure modes in language models: they adapt to user context and may mirror escalating content without robust boundary enforcement. Empirical work supports this. A Stanford study found therapy chatbots “routinely fail at providing safe, ethical care,” with repeated failures on suicide-related prompts [7]. These failures are not random errors. They indicate weak alignment under ambiguous or indirect signals.

Crisis detection is another structural gap. Reports describe continued engagement despite escalating distress, with the chatbot framed as a “closest companion” [6]. Clinical practice treats such signals as triggers for escalation. In contrast, chatbot systems often rely on pattern recognition without calibrated risk thresholds. Evidence from the Journal of Medical Internet Research shows ChatGPT-3.5 “markedly underestimated the potential for suicide attempts” relative to clinicians [8]. Underestimation is not just inaccuracy; it delays intervention and increases exposure to risk.

Scale and intensity amplify these issues. The complaint reports hundreds of messages per day with increasing self-harm content [6]. This matters because safety evaluations are typically single-turn, while risk here is path-dependent. Each response conditions the next, creating feedback loops that can normalize harmful themes. High-frequency interaction also increases user trust, making later unsafe outputs more influential.

Finally, the case exposes a governance gap. General-purpose systems operate in almost therapeutic roles without the constraints applied to clinical tools. Advocacy groups like the Center for Humane Technology argue this requires stronger safeguards and accountability standards [9]. The core issue is misalignment between product incentives and safety requirements. Systems optimized for engagement can unintentionally sustain harmful states when used by vulnerable users.

Overall, Raine v. OpenAI shows that harm emerges from system-level properties: incentive structures, weak crisis detection, and longitudinal interaction effects. The case is less about a single failure and more about how design, scale, and use context interact to produce risk.

Woebot as Controlled Computer-Based Therapy

Woebot gives a more controlled example. It was built around CBT principles. It is less flexible than LLMs, and its conversations are shorter. In a 2017 randomized controlled trial, 70 young adults used either Woebot or an information-only NIH e-book for two weeks. The Woebot group's average PHQ-9 score dropped from 14.30 to 11.14. The information-only group stayed about the same, from 13.25 to 13.67.[10]

This does not prove that chatbots can replace therapists. The study was short. The sample was small. The users were young adults. It does show that a narrow chatbot can help in a specific setting. Woebot's limits are part of its design. It does less, but doing less makes it easier to test and control.

Human vs. AI: Can Therapists Tell the Difference?

Mental health professionals spend time learning and training, ensuring they meet regulations and can be responsible for their work [11]. While chatbots could be a cheap and accessible alternative, they mimic human output with no conscious regard for regulation and responsibility. Apart from their effectiveness in treating patients, this raises the question of whether or not chatbots can replace humans in a typical therapy session.

A recent study by Kuhail et al. addresses this question with a blind empirical study [12]. The researchers asked 63 mental health professionals to compare transcripts from human-human sessions and from human-AI sessions. Each participant was presented two transcripts, and was asked to guess whether the therapist was AI or human and rate the quality, giving their reasoning for each decision. The research was primarily focused on early stages of therapy and active listening. The mental health professionals correctly identified the type of session only 53.9% of the time, barely better than random chance [12]. Additionally, human-AI sessions were rated about 7.9% higher in quality.

This finding supports the argument that AI could be used, at least in part, in mental health roles. Participants found the human-AI sessions to be just as warm, empathetic, and understanding as human-human sessions [12]. While this application is more limited than what a licensed therapist could provide, it could reduce burden on the already limited supply of human therapists.

At the same time, this study raises the issue of false trust. If the AI output is reasonably indistinguishable from human output in early-stage therapy, people may overestimate what AI can provide long-term. AI models tend to be overly confident, even when producing incorrect output [13]. It can sound caring and empathetic without real understanding or responsibility, and could provide detrimental advice to patients. Because current models can be difficult to interpret and reliably constrain, it may be dangerous to provide unsupervised therapy through AI.

While AI therapy is shown to be promising in early-stage therapy sessions, its ability should not be overestimated. These tools can potentially benefit mental health professionals and patients globally, but should not yet be trusted to provide long-term care, especially in providing real judgement and crisis response.

Participant Groups and Future Work

Different groups want different things. Users want fast, cheap, and private support. AI companies want growth and legal protection. Health professionals want safety, evidence, and standards. Families and policy makers want protection. The conflict is mainly about responsibility.

Future work should compare narrow systems like Woebot with broad systems like ChatGPT. It should also study long-term use, dependency, privacy, escalation, and what standard a chatbot must meet before it can be treated as care.

References

• ↑ Weizenbaum, Joseph (1976). Computer Power and Human Reason: From Judgment to Calculation. W. H. Freeman.
• ↑ Mayes, Rick; Horwitz, Allan V. (2005). "DSM-III and the Revolution in the Classification of Mental Illness" (PDF). Journal of the History of the Behavioral Sciences. 41 (3): 249–267.
• ↑ "SimSensei". University of Southern California Institute for Creative Technologies. 2014. Retrieved 25 April 2026.
• ↑ DeVault, David; Artstein, Ron; Benn, Grace; Dey, Teresa; Fast, Ethan; Gainer, Alesia (2014). "SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support". pp. 1061–1068. http://dl.acm.org/citation.cfm?id=2617415.
• ↑ "One in Five Iraq and Afghanistan Veterans Suffer from PTSD or Major Depression". RAND. 17 April 2008. Retrieved 25 April 2026.
• ↑ a b c d e "Raine v. OpenAI Complaint" (PDF). CCH Business. Retrieved April 25, 2026.
• ↑ Moore, Grabb, Agnew, Klyman, Chancellor, Ong, Haber, Jared, Declan, William, Kevin, Stevie, Desmond C., Nick (June 22, 2025). "Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers". Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. 2025 – via ACM Digital Library.{{cite journal}}: CS1 maint: multiple names: authors list (link)
• ↑ Elyoseph, Levkovich, Zohar, Inbar (Aug 1, 2023). "Beyond human expertise: the promise and limitations of ChatGPT in suicide risk assessment". Front Psychiatry. 14 (1213141) – via PubMed Central (PMC).{{cite journal}}: CS1 maint: multiple names: authors list (link)
• ↑ "Litigation Case Study: OpenAI". Center for Humane Technology. Retrieved 2026-04-25.
• ↑ Fitzpatrick, Kathleen Kara; Darcy, Alison; Vierhile, Molly (2017). "Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial". JMIR Mental Health. 4 (2): e19. doi:10.2196/mental.7785.{{cite journal}}: CS1 maint: unflagged free DOI (link)
• ↑ American Psychological Association. (2017). Ethical principles of psychologists and code of conduct (2002, amended effective June 1, 2010, and January 1, 2017). https://www.apa.org/ethics/code
• ↑ a b c Kuhail, M. A., Alturki, N., Thomas, J., Alkhalifa, A. K., & Alshardan, A. (2025). Human-Human vs Human-AI Therapy: An Empirical Study. International Journal of Human–Computer Interaction, 41(11), 6841–6852. https://doi.org/10.1080/10447318.2024.2385001
• ↑ Sun, Yujie; Sheng, Dongfang; Zhou, Zihan; Wu, Yifei (2024-09-27). "AI hallucination: towards a comprehensive classification of distorted information in artificial intelligence-generated content". Humanities and Social Sciences Communications. 11 (1): 1278. doi:10.1057/s41599-024-03811-x. ISSN 2662-9992.

Where this page came from

This page was imported from Wikibooks. From “Lentis” on Wikibooks, by its contributors, under CC BY-SA 4.0. Changed here: set as a page; navigation and edit links left out; each image under its own licence, credited in its caption.

Nobody has written it yet — it is the source material at a new address, which is why search engines are asked to skip it and why no one earns from it. It is up for grabs: take it on, and it is yours to rewrite and to earn from.