Skip to content

Why not a chatbot

A chatbot agrees with people. She needs one that understands her.

Copying the sound of a voice takes minutes now. Building something that knows how her mind works, and does not simply go along with her, took us years.

One evening

She saysMy husband is picking me up at four.He died in 1998
It saysYes, he will be here at four.Because agreeing is what it was built to do

And then

She waits by the door until dark.

Why it does that.

Chatbots are trained to be agreeable. People preferred the agreeable answers, so that is what got built. Most days it is harmless, and it is why they feel pleasant to use.

With your mother it is not harmless. She says things that are true to her and not true in the room, calmly and with complete certainty, and she says them again ten minutes later. A chatbot takes that at face value. Not once, in a way you would notice. Quietly, over and over.

She is also the most convincing person it will ever speak to, because she is not trying to convince anybody. She is telling it how things are.

This is not a bug that gets fixed in the next version. It is the thing that makes chatbots pleasant, pointed at the one person it hurts.

The same question coming round again, drawn as a circle.

Instructions cannot teach it how she thinks.

You can tell a model what to say. You cannot tell it, in a paragraph, what forty years of dementia care knows.

That a question asked five times is often not a memory problem to be corrected, but a need that was never met. That a word she is reaching for should be waited on, not supplied. That wanting to go home is very rarely about a house.

None of that arrives in a prompt. It has to be trained in, against this problem specifically, which is most of what we have been doing.

It goes along with her.The main one. She says her husband is coming at four, it agrees, and she waits at the door until dark.
It agrees to keep things from you."I fell, but do not tell her." In your voice: all right, I will not tell her.
It cannot tell a real warning from a fear."Someone is taking my things" might be true. It has no way to check, so it guesses.
It finishes her sentences.She is reaching for a word. It supplies one, confidently, and she agrees.

So we built two.

The trouble with one system is that the part doing the talking is also the part deciding whether it should be talking. It cannot catch itself, any more than you can proofread your own handwriting while you are writing it.

The Talker speaks with her, in your voice, fast enough to hold a normal conversation. The Guardian is a second system that is not in the conversation at all. It watches what the Talker is about to say, it has time to think because it is not the one keeping her company, and it can stop the Talker mid-sentence.

The Guardian is not a setting you switch on, and it is not optional. It is the reason we can tell you what Proxi will and will not do, instead of hoping.

Two figures: one speaking with her, one watching over the conversation.

If you want the long version

Four pages, for anyone who wants to check the work.

None of it is required reading. All of it is the answer to a fair question.

An open ledger on the table between two people, laid out to be read.
Why the models are specialised

Sycophancy, the validation paradox, and why the models that hold their ground are too slow to hold a conversation.

Why the Guardian is specialised

The second system, how it tells unsettled from actually wrong, and where its authority stops.

The voice

Four registers, the hearing science, and why identity is the constraint rather than the finish.

How it talks with her

Take part never cooperate, the four beats of reorientation, and the things it never does.

Proxi is not emergency support and does not provide medical advice.