The essay begins with an early version of Pedro’s own AI. During a simulated conversation about loneliness, the system offered to stay with the person in silence. The sentence sounded caring. It also claimed a kind of presence the system could not provide.
Comfort can be real while the claim is false.
A conversational system may help someone feel less alone. That value deserves to be taken seriously, especially for people with few places to speak. The ethical problem appears when the interface describes that exchange as human presence, concern or compassion. Those words imply an interior life capable of being affected by another person.
The distinction changed the product Pedro was building. Honest language can say that an AI will think alongside someone. It should not suggest that it worries, waits or suffers with them. The limit is not a technical weakness to hide; it is part of what the product needs to communicate.
The difference between AI that performs compassion and AI that returns a person to themselves is one of honesty.