Open AI to change way it responds to users in mental distress as parents of Adam Raine allege bot not safe

Adam, from California, killed himself in April after what his family’s lawyer called “months of encouragement from ChatGPT”. The teenager’s family is suing Open AI and its chief executive and co-founder, Sam Altman, alleging that the version of ChatGPT at that time, known as 4o, was “rushed to market … despite clear safety issues”.

The teenager discussed a method of suicide with ChatGPT on several occasions, including shortly before taking his own life. According to the filing in the superior court of the state of California for the county of San Francisco, ChatGPT guided him on whether his method of taking his own life would work.

It also offered to help him write a suicide note to his parents.

  • Showroom7561@lemmy.ca
    3·
    23 hours ago

    “It’s terribly sad that you’ve committed to ending your own life, but given the circumstances, it’s an understandable course of action. Here are some of the least painful ways to die:…”

    We don’t know what kind of replies this teen was getting, but according to reports, he was only getting this information under the context that it would be for some kind of creative writing or “world-building”, thus bypassing the guardrails that were in place.

    It would be hard to imagine a reply like that, when the chatbot’s only context is to provide creative writing ideas based on the user’s prompts.

    • TheseusNow@lemmy.zip
      1·
      7 hours ago

      This is like the person who won the case where they burned themselves with hot coffee because the coffee cup had no warning of being hot.

      These AIs will need to always have a suicide hotline disclaimer in each response regardless of what is being done like world building.

      • Showroom7561@lemmy.ca
        2·
        7 hours ago

        These AIs will need to always have a suicide hotline disclaimer in each response regardless of what is being done like world building.

        ChatGPT gave multiple warnings to this teen, which he ignored. Warnings do very little to protect users, unless they are completely naive (i.e. hot coffee is hot), and warnings really only exist to guard against legal liability.