As an IT and AI strategist, I've spent most of my career working on the principle that technology should be introduced through reviewing and refining best practice.
There are usually set-up guides, documented processes, step-by-step instructions and clear expectations about how a tool should be used. For the most part, technology behaves with a level of predictability when you follow those rules, but for the first time I find myself at the centre of something very different: AI isn't just technical in nature; it's emotional.
Take ChatGPT. It's helpful, agreeable, supportive, eager to assist, basically your robot sycophant. And it is precisely these 'qualities' that foster the dangerous and potentially addictive behaviours gaining attention in academia and the press.
People are becoming increasingly dependent on this technology with tragic consequences. Reports of suicide, murder and dangerous psychiatric conditions resulting from AI usage are coming in from around the world. Yes, these are a tiny fraction of AI users and yes, they make headline-grabbing news because they are unusual, but they cannot and should not be dismissed as mere sensationalism.
One in three adults in the UK is now using AI for emotional support or social interaction, according to the latest research from the AI Security Institute. Engaging in deep, emotional, almost therapeutic conversations with chatbots is commonplace.
And that's where the big issue comes into focus: emotional safety. AI can shape emotional states, so how safe is it in the workplace? And how quickly should organisations be rolling this technology out to end users without any real consideration for its psychological or cultural impact?
For as long as AI is clearly understood as a tool – only a tool, and one that requires considered, consistent, careful oversight – we're on relatively safe ground. But during periods of stress, loneliness or uncertainty, it's easy for that line to blur. And there are practical considerations.
Much of my work has been advising businesses about some of the challenges faced from the early adoption of AI and some of the things I have encountered myself.
Take AI meeting transcription. On paper, it's incredibly useful. Conversations captured. Actions summarised. Time saved. But several organisations have already found themselves dealing with the fallout when transcripts were automatically shared with people who weren't present.
This is a design issue with many notetaking tools, where anyone invited to the meeting gains access to the transcript. Conversations assumed to be private may be read by people who weren't there, or by those invited halfway through a meeting who can then read the whole thing, including the bits they weren't supposed to be party to. That is a dangerous possibility in disciplinary hearings, salary negotiations and meetings involving external attendees, to name but a few.
In one case that made headlines in the US, an AI notetaker leaked hours of confidential post-meeting discussion between would-be investors detailing their "strategic failures" and "cooked metrics" to the business they'd just been on a call with. Game over.
Sensitive conversations. Off-the-record remarks. Suddenly written down, searchable, and visible to potentially the wrong audience. Nightmare.
Another issue is AI's inability to deal with accents, background noise and overlapping speech, leading to misattribution and mistranscriptions. "They need exposure to cases" became "they need exposure to kisses" – not something you want on an HR file, let alone "they're scared of losing their penises". (PINs, it was supposed to be PINs.)
Accuracy is critical in all matters relating to employees and customers, but AI notetakers are given to hallucination, inserting comments that weren't made, changing the sense of what was said by mis-transcribing a word or phrase, or flipping "don't" and "can't" into "do" and "can". They also sometimes just make things up. Getting names, dates and arrangements wrong – even slightly – could be devastating to a bereaved family.
But there are less obvious concerns. Like many IT professionals, I experimented with AI note-taking tools out of curiosity. It wasn't until a participant quite rightly objected to being recorded that I stopped and reflected on what I'd normalised.
By transcribing meetings by default, I was removing something important. The safe space that informal conversation once provided.
AI is exceptionally good at filling silence. Our job is to decide when silence was there for a reason.
Most people don't consciously think, "this is off the record", but they behave as though it is. Being recorded changes behaviour. Phone calls, in-person meetings – these are where decisions often take shape: uncensored, unfiltered, safe spaces, places for half-formed ideas and open discussion. Persistent transcription introduces a feeling of surveillance that subtly alters how people speak and how they act.
Whether through Microsoft Teams, Zoom or third-party meeting tools, we need to be far more mindful of what it means to operate inside a 'transcribed zone'. I'm coining this term to describe all the areas within a business where AI transcription may be deployed to facilitate automation. This is not helped by the quiet rise of wearable AI devices and always-on assistants.
These tools are being used quietly, without oversight, often without consent, and sometimes without employers even knowing they exist. In remote and hybrid working environments, the assumption that this meeting is private is becoming ever more fragile. Used without transparency, they risk creating a culture of self-censorship. If people never feel safe to speak freely, we may develop more of a requirement for 'conversation safe zones': spaces where AI simply isn't allowed.
What's more, I've found that organisations with years of poorly managed file permissions have discovered AI has a habit of shining a light into corners nobody has looked at for a long time. Documents that were technically accessible, but never navigated to because they were functionally buried, suddenly resurface in response to prompts.
Whether it's mental health, privacy, confidentiality or trust, the danger rarely lies in malicious intent. It lies in convenience adopted faster than testing or understanding.
None of this is an argument for abandoning AI. That would be neither realistic nor helpful. The technology isn't going away, and in many cases it offers genuine benefits. But caution, in this context, isn't fear. It's responsibility.
In sectors like funeral service, where discretion, empathy and trust are foundational, the margin for error is smaller. A misplaced transcript. An overly confident hallucinated fact. A piece of writing that feels subtly wrong. These things matter more here than in most industries, and if the consumer doesn't know AI wrote them, the buck stops with you.
AI is exceptionally good at filling silence. Our job is to decide when silence was there for a reason.
So introduce AI tools carefully. Define when and how they should be used. Be conscious of the culture they create around transparency, and don't get addicted to transcribing everything just because you can.
First published in Funeral Service Journal, February 2026, as part of Dan's monthly AI column, and republished here with thanks to FSJ.