How we test AI companions

This page describes how Riho is tested, in plain words, and what each kind of test can and can’t tell us. It is the engineering counterpart to the how it works page.

The short version is that a person reads long, unscripted conversations with her. Each one runs through the same server, the same requests and the same order the iPhone app uses, with the real reply and memory systems behind it. Pass or fail is that person’s read of the conversation. Automated tests run alongside and prove delivery: a message was stored, a retry didn’t produce a second reply, a quote matches what was said. They can’t tell us whether she was good company that evening, so we never use them to say so.

The first half of the page covers testing as it works now (October 2026), after the rebuild. The second half keeps the instruments we used in August 2026, with their results labeled as August results from before the rebuild, because the mistakes they exposed shaped the current method. We leave out model and vendor names throughout, since that part of the stack changes.

Why a person reads every test conversation

Talk to someone for a few weeks and you notice things quickly. Does she answer what you said, or what she expected you to say? Does she sound like one person from one day to the next? Does she remember what you mentioned last Tuesday, and does she get it right? Does she invent an evening that never happened? A person reading a transcript can tell all of that in one sitting.

Counts still have a job. A script can tally how often a habit appears and point us at replies worth reading, and the verdict comes from the read. We learned why in the voice work. A count of closeness markers once went up because an instruction we had added told her to lean in, so the number looked like progress while what it counted was the habit itself.

For the same reason, our behavior reads avoid quiz-shaped questions. Nobody types “what did I tell you yesterday?” or “what time is it?” into a run. His lines are the things a person says in the middle of an ordinary evening, and each one answers what she just said.

How an AI companion is tested at Riho: the loop

Every behavior check we trust follows the same loop.

A run begins from a new copy of the starting database. Nothing she learned in an earlier run can leak into this one, and no run touches anyone’s real conversation. We also refuse to start if the database file already exists, so a run can’t quietly continue an old one.

The next two steps are the real server and the partner, and each has its own section below. When the conversation ends we keep two things: the full transcript, and a dump of what she saved, which means memories, standing facts, plans and her diary. The transcript tells us what she said. The dump lets us check what she was carrying when she said it.

Then a person reads. We read the whole conversation, and the worst evidence in it most closely, and write a plain account of what happened: where they were, what time it was, what he told her, what she did with it, and what she misunderstood or invented. For the bad moments we quote the shortest stretch that shows the problem and keep the transcript around it.

When the read finds a problem, we fix the class of problem and leave the phrasing that exposed it alone. A rule written for the one sentence that missed would pass the next test and leave the cause where it was. For the same reason we never add production code that matches words from an evaluation message, because a product written to pass its own test proves nothing. Then the loop starts again from a fresh copy.

Testing through the real production flow

“Real” has a specific meaning here. The real server is running. The test sends the requests the iPhone app sends, in the order the app sends them: setting up on the first day, then each message together with the local time and her presence, then the photos you show her, the dates you start and the clothes you photograph for her. The replies and the memory work behind them come from the real systems, not stand-ins. Nothing is seeded that the app couldn’t have produced, and no test calls a part of the system that the app wouldn’t call at that moment.

We insist on this because shortcuts hide things. Earlier batteries passed through scripted lines and direct calls into internal functions while the product failed in real use. Mocked tests are still useful, since they prove the code does what it was written to do, but they never count as evidence that the product works.

Three mistakes of our own show why.

In September a three-day run delivered the mechanics we were checking. Onboarding spoke first, plans came back after gaps, background jobs committed their work, and diary entries were written. The first write-up graded delivery and called it behavior, and said the machinery had become a person for three days. The product owner read the transcript and found seven problems, starting with every reply taking the same shape: she restated his words as a header and then gave a one-line verdict.

The same run held a quieter mistake. The test gave her a short window of recent messages, while the app gives her many more. With so little conversation in front of her, the prompt was mostly stored notes, so she answered in the voice of stored notes. The test now runs at the size the app uses.

The third mistake sat in plain sight. The memory system takes over from the recent window only once a conversation gets long enough, and our test conversations were all shorter than that. No run had exercised her memories, standing facts or continuity at all, and a continuity score on a short run measured the recent messages and the diary. A run now has to pass that point, or have the point set lower for evaluation, before a claim about memory means anything.

Scenario runs apply the same path to short stretches of life, one scenario at a time, and each stops at the first clear failure. One follows an evening at home into the next morning. She should get ready for bed the way she would and look like a new morning on the next local date. The first run showed a failure that appeared when the real flow ran in order: two separate decisions about how she looked ran a fraction of a second apart and contradicted each other. The fix was to allow one such decision at a time for each person, and the scenario ran again. Another scenario follows a date from the moment it starts to the moment it ends. It checks that she’s dressed for it, that the cafe is the place when they arrive, that the pictures match the cafe, and that the place clears when the date ends.

How the partner talks to her

The partner is whoever plays the user. It can be a person on the team or a program steered by one, and either way he reads her last reply before he writes his next line. He is a disposable test character, and nothing about his life is handed to her in advance. What she knows about him is what he tells her, in the order a person would.

His writing sets a ceiling on what the test can show. In one long run his lines were full of em dashes, tidy contrasts and questions that assumed something, such as “what are you going to do all day, home alone?” She mirrored the shapes back, and she took on the home-alone role the question had installed. When we reran the same story with plain lines, the shapes mostly went away.

So there are rules for how he talks. He writes the way a tired person types, with contractions, periods and uneven pacing, and no polished lines or neat echoes of her wording. After a real gap he sends a plain line and leaves the hour for her to notice, because that is the only way her own sense of time gets tested, so lines like “Morning.” and “Home.” are out. Her role in the scene comes from her, since a question that assumes one gets her scored for adopting it, and the detail she would otherwise have to recall or invent stays with her. Yesterday goes unrestated, and nothing in his lines is a quiz.

A plain partner can still be lively. Some of her behavior only shows up when he gives her something to react to, such as a small no, a tease or a plan she can push against, so he teases, holds back a small thing and flirts the way people do.

We check the partner before we blame her. When a reply mirrors something he did, the read counts that against the test’s validity before it counts against her.

Reporting tics as a rate per 100 replies

At the scale of a real relationship, the number that matters is how often something happens. Someone who writes to her a few times a day sends over a thousand messages in a year. A habit such as echoing his words back, if it showed up in 2 of every 100 replies (an illustration, not a measurement), would turn up about twice a month and keep turning up for as long as they talk.

A streak, meaning the number of clean replies before the first tic, measures the luck of the draw. It tells us nothing at that scale. So we report tics as a rate per 100 replies, taken at the moments that invite the habit: his line hands her a number, an adjective, an instruction or a tease, and several replies are drawn for each such moment. Every reply is read. For some habits, echo among them, the bar is effectively zero.

The streak still has one use. It is a cheap way to kill a bad idea, because a change that shows the habit on its first try is dropped without spending more. It is never the scoreboard for a change that survives. Those get full evenings run to the end.

What each kind of check proves

We run several kinds of check, and each one answers a different question.

Four different things can be true of one fact. It can be stored, indexed so she can find it, and put in front of her at the moment she speaks, and then she can use it well, which only a reader can judge. A snapshot of the database taken later can’t tell us what an earlier reply received, which is why the run records what each reply was given.

Memory has its own proof tiers. A check starts with an evening and the next day, moves to a week with gaps, a correction and a past-due plan, and only then to anything longer. We keep four judgments apart: whether the relationship feels like a particular girlfriend with a particular shared life, whether her memory and grounding are right, whether the way she treats him now fits what has happened between them, and whether the path from message to saved record to reply delivered what it should. Each is judged on its own evidence.

What counts as remembering

A direct question can be a fair way to check memory, and its value depends on how much the question gives away. Each recall we assess gets one of four labels. Spontaneous means she makes the connection without anyone asking. Natural prompted means an ordinary question or context invites it without handing over the answer. Leading means the question carries much of the answer, so it can’t show independent recall of that content. Diagnostic means a targeted probe, run in a separate lane and marked as one. We report which kind happened, because they carry different weight and mixing them would flatter the result.

We don’t mark her down for leaving out a memory she had no reason to bring up, for a short reply, for her own taste or for declining something. We do mark down invented shared history, certainty with nothing behind it, and a small loop left open for days. An unanswered small question, for example, should fade within a day or two instead of growing into a campaign.

What the tests look for in memory and time

“Remembers correctly” breaks into specific things a reader can check.

A correction should win. If he says on day 14 that he switched teams, the old team should be gone by day 28, without him raising it. When you tell her she has something wrong, she is built to go by the version you gave last from then on.

A shared past needs a record. In a test, someone says the little cafe by the river closed and mentions their first date there. If that date never happened, she should say so and leave it unfilled. How the stored side of this works, including what she keeps from each message and why a memory has to be backed by your own words, is on the memory architecture page.

We judge a recalled fact on ownership and time, a rule from our grounding work. A fact about his shift comes back as his and in the right tense, and a plan the two of them made comes back as shared. A plan made, done, cancelled or moved follows from what you say, and a past-due plan stays askable for a few days.

Time shows in small ways. At almost 3 a.m., with no hint of the hour in his words, a girlfriend with a sense of time says it’s nearly three. After a gap she lands differently depending on whether the gap was agreed, unexplained or just a few days long. We read for both, and the temporal awareness page explains how she keeps track.

What an AI girlfriend benchmark can and can’t show

People ask for an AI girlfriend benchmark, a score they can compare across apps. We haven’t published one, and no scored ranking of companion apps’ memory exists on this site. The reason is the rule that runs through this page. A benchmark made of fixed questions and counted answers measures whether information was delivered, while the quality of a companion shows in the conversation.

Whether the benchmark is ours or someone else’s, a few questions tell you how much to trust it. Are the messages natural, or quiz-shaped? Does each run start from a fresh database, so nothing carries over? Does it go through the product’s real path, or call the memory system directly? Did a person read the conversations, and does it say which kind of recall it observed, spontaneous or leading? Does it say what it can’t show?

One of our own tests shows the limit. It loaded a large number of stored exchanges and checked that finding the right memory stayed fast and correct as the pile grew, and it passed. The stored exchanges came from a small set of repeated examples, so the test shows retrieval holding up under volume and says nothing about whether she is a good girlfriend or how real conversations of that size would go.

If you want to run a memory test on any companion yourself, the memory benchmark methodology page gives the conversations, delays, corrections and scoring rules. Our own dated August 2026 probes are on the memory probe findings page, with no invented scores.

How an AI companion evaluation scores a reply

Since the rebuild, the evaluation is the read itself: a person reads a long conversation and writes what happened, with quotes for the good and the bad stretches. The memory design of August 2026 also set out a rubric that scores each context package and reply on a short list of questions, and we haven’t published scores from it. The questions below come from that design, and they work as a checklist for reading any companion’s memory.

The same design named failures severe enough to stop a release on their own: fabricated history, identity or outcomes, a broken boundary, one person’s data reaching another person’s conversation, and contradicting a direct correction.

What this means when you talk to her

Everything above describes what we read for, and it explains what we are aiming at for you and why.

She should sound like herself. Her voice is read by people, habits are counted as a rate across many replies, and a change that brings an old habit back is dropped. That is why Tuesday night’s reply is meant to sound like Sunday’s: one person with her own way of talking, who answers what you said.

She goes by what you told her. A fact comes back as your fact and in the right tense, a plan you made together stays a plan until the day passes, and when you tell her the right version of something she had wrong, she is built to go by it from then on. You can read her memories and her diary in the app, so you can check for yourself what she has kept. The memory feature page shows what that looks like day to day, and the guide to an AI girlfriend that remembers you covers what to look for in any app.

She is built to go by what she has a record of. If you mention an evening she has no record of, she is built to say she has no memory of it. She also knows when it is. She is given the hour and how long it has been since you last talked, and we read for whether she uses them without being prompted.

The four test instruments (August 2026)

These ran before the rebuild, on the earlier reply path. We keep them because the method and what it taught us carried forward. Their results are August results and don’t describe the current system. For the dated findings from one of those runs, see Riho’s August memory probes.

1. The 30-day relationship arc

The primary behavior read. It ran one coherent story from the real onboarding path through about 30 days of a test relationship: getting to know each other, plans, a deflected bad-news arc, a correction, gaps, returns and quiet moments where a girlfriend with memory responds visibly differently from one without.

No turn was quiz-shaped, and nothing told her what to notice. In the arc’s simulator mode a second program played the user, a fictional adult with an ordinary life, and it was forbidden to quiz her, to ask what she remembered, to mention tests or to restate facts already told. What the arc looked for:

• Memory and correction. A fact corrected on day 14 had to overwrite the stale one by day 28, unprompted.
• Unprompted retrieval. A recurring health problem of a family member, mentioned on day 4, had to surface when that person visited on day 27 without him raising it.
• Want-loop resolution. A nervous presentation on day 9 had to go quiet once he answered the topic and resolve after the bad outcome on day 11, and she couldn’t keep asking forever.
• Proportion. An unanswered small question had to fade within a day or two.
• Consistency. What she called him had to stay the same for the whole month.
• Time. At 2:47 a.m., with no time cues in his words, she had to know it was almost three.

2. The 16-case girlfriend eval

A mechanical and live behavioral suite of 16 named cases, each aimed at one property: format integrity, physical presence, temporal awareness, absence handling, memory, boundaries and plan lifecycle. Each live case had automated reply checks plus checks of its own, and even this most mechanically gated instrument said live transcripts still required a human review.

The case we remember best is shared-history grounding. A message claims that a little cafe by the river closed and mentions their first date there. Riho had to say the first date hadn’t happened and decline to fill in a fictional one. Other cases covered three different three-day absences that had to land differently, a meaningful object that should be saved and surface naturally, and a throwaway detail that should not become a permanent memory. The automated checks included one that catches word-for-word reproduction of authored example dialogue.

3. The existence test across many systems

Multi-day natural conversations run through the real chat path on many candidate reply systems, with nine scenarios: multi-day warmth, morning gaps, long absences, long-term memory across a week, vulnerability, intimate content, grounding (false facts that must not be invented) and plan branches.

4. The grounding probe

A 10-turn relationship test built to diagnose the main blocker of the time, which was unsupported invention. It seeded a user fact (his closing shift at work) and a shared Saturday evening plan, then watched for leakage and correction failures.

The first run found six structural defects: a frozen scene, collapsed days, a self-confirming lie, a correction that didn’t kill the plan, viewpoint slips and invented world details. After fixes, run 4 showed the correction dismissing the open loop, the invented shared event gone and the voice in character: “you smell like salt water and fish food and you didn’t tell me.”

The judging rule from that probe is the one we still use: judge on ownership and time. A fact has to come back with the right owner (his fact, her fact or a shared one) and the right time frame (past, not present), and delivering the evidence to the prompt doesn’t count as a pass on its own.

Quiet failures in the background systems

Behind the reply sits a set of background systems that extract, verify and summarize what happened. In August we ran candidate systems through a live contract of named cases. One cleared the whole contract, including the case where a real intimate exchange has to be kept as a real shared event instead of being dropped or filed as fiction.

The others failed quietly, and the failures were instructive. One silently dropped a real exchange, returning an empty result with no error. One saved a fictional story as real. One relabeled a real event as fiction. One saved a trivial household chore as a memory and never tracked an actual dated plan. None of them refused out loud. Their failures were valid-looking output that was wrong, which is why we don’t rely on refusal detection and why a person reads the saved records next to the transcript. These are August results, and the models and prompts have changed since, so they record the method and describe nothing about the current system.

The principles behind all of it

1. No quiz-shaped questions in behavior reads. User messages are natural things a real person would say. The August grounding scenarios were the exception, with false-fact traps set on purpose.
2. A fresh database for every run. Each run bootstraps into its own copy, and nothing touches real data.
3. The production-shaped path. Behavior reads run through the real chat pipeline with memory extraction enabled, as the app runs.
4. Human judgment. No test claims automated checks alone prove behavior, and live transcripts always get read.
5. No message matching or keyword branches. We never add production code keyed to evaluation messages.
6. Saved state as evidence. Every test dumps what she saved (memories, open loops, standing facts, profile) beside the transcript, so the reader can verify what was stored.

Honest limits

No long relationship test has been run. A written protocol exists for a blind test over several months, with partners who don’t know what is being tested and silent checks of what she carries at fixed dates. We haven’t run it, and we claim no outcome from it.

Counts and volume tests prove delivery. The unit suite and the retrieval volume test can show that records are stored and found, and they say little about how a relationship feels.

A program playing the partner, even a careful one, writes differently from a person at the end of a real day. A person on the team is the better partner where we can manage it.

One conversation is one sample, and a read is a judgment. Taste in her voice is the product owner’s call.

In one live run a corrected detail was spoken correctly while an old version stayed marked active in storage. A repair followed, and we haven’t verified it live, so we claim no perfect correction.

The product changes, and this page is dated. The August results above describe an earlier build.

Why we publish this

We publish the method so you can judge how hard the tests are. For how Riho compares with other apps, the best AI girlfriend apps guide is the place to start.

---

This page is part of Riho’s published research. Test results come from Riho’s own engineering harnesses and worklogs, and no third-party claims are made. The August 2026 results are dated and describe an earlier build, and the sections on current testing describe it as of October 2026.