What Does Testing an Idea With 1,000 AI-Simulated Participants Actually Mean?

On this page

1. Step 1: The audience is split into groups
2. Step 2: Code creates each participant
3. Step 3: How each participant “answers”
4. Step 4: How the totals are calculated
5. Why the count isn’t a survey sample size
6. What a report can reveal
7. What results cannot establish
8. Precision is not accuracy
9. Turning findings into real experiments
10. Your next step

“Tested with 1,000 participants” sounds like a survey. It isn’t, and knowing exactly what happens inside the test is the best protection against reading too much into the result.

This guide walks through one SellTo.us test from start to finish: how the simulated personas are made, how each answer is produced, how the totals are calculated, what the report can tell you and what it can’t. Everything here matches our published methodology page, which describes the engine as it currently runs.

If you’re still deciding whether simulated testing belongs in your process at all, start with our comparison of AI-simulated audiences and real customer research. This article assumes you’re about to read a report and want to understand it.

Step 1: The audience is split into groups

You describe your idea and who it’s for. An AI model (Claude Sonnet 5.5, by Anthropic) reads both and divides the audience into groups. A 1,000-participant test has 8 groups; larger tests have 25 or 100.

Each group gets:

• a name and its share of the audience,
• an age range,
• 4 typical jobs or life roles,
• a budget level: tight, average, comfortable or high,
• a short note on how they shop.

If you add a photo, the same model looks at it once and writes a short text description. Participants react to that description, not to the image itself.

This is the most consequential step. The groups are the AI’s interpretation of your audience settings, so a vague audience (“everyone who cooks”) produces broad, generic groups, while a specific one (“parents of toddlers who batch-cook on Sundays”) produces sharper ones.

Step 2: Code creates each participant

Individual participants are created by our code, not by the AI. For 1,000 participants, the code makes exactly 1,000 profiles:

• each is placed in a group in proportion to that group’s share,
• age is picked at random within the group’s range,
• job is picked at random from the group’s 4 jobs.

The random choices are seeded from the test’s ID, so the same crowd can be rebuilt exactly. In the 100,000-participant test, each profile also gets a city or area.

Notice what does not vary: everyone in a group shares the same budget level and shopping note. So 1,000 participants are really 8 group descriptions, each expanded into many profiles that differ by age and job.

Step 3: How each participant “answers”

Profiles go to a second model, Claude Haiku 4.5, in requests of 40 at a time. A 1,000-participant test is 25 requests. Each request contains your audience, every version you’re testing (text, price, photo descriptions) and 40 numbered profiles.

The model returns one answer per profile. For a physical product that’s would buy, maybe or wouldn’t buy; for an app it’s would try, maybe or wouldn’t try; for a service, would consider, maybe or wouldn’t consider. Each answer comes with a reason under 12 words.

So every participant has their own profile and answer, but they are not separate conversations. The model judges 40 profiles in one go.

Step 4: How the totals are calculated

The arithmetic is deliberately simple.

Who answered. Answers that come back malformed or not at all are skipped, never guessed. The report shows how many actually answered, for example “988 of 1,000”. If fewer than half answer, the test fails and the credit comes back automatically.

Each percentage. For each version: the number of participants who gave that answer, divided by the number who answered. Say 391 of 988 answered “would try”. The report shows that as 40%, rounded to the nearest whole percent.

Group results. The same count, done inside each group.

What isn’t calculated. No margins of error and no statistical confidence. We haven’t measured results against real sales, so attaching a confidence figure would suggest a kind of accuracy nobody has established.

Why the count isn’t a survey sample size

In a real survey, a larger sample shrinks random error because each new respondent is an independent draw from the population. That logic doesn’t carry over here.

• The answers aren’t independent draws. They come from one model, working from a handful of group descriptions.
• There’s no population being sampled. The groups are an AI’s interpretation of your audience text, not a selection of real people.
• Reruns differ. Running the same test again gives somewhat different numbers, and the size of that difference hasn’t been measured.

What a larger test does give you is more groups, more variety in profiles, more sample interviews (12 at 1,000; 50 at 10,000; 200 at 100,000) and room for more versions. That’s more detail to explore, not more certainty.

What a report can reveal

Read a report as a structured brainstorm with numbers attached. It is good at showing:

• Reasons. The report is written from the totals, group results and a random sample of 160 participant reasons. Recurring reasons for “no” are often the most useful part.
• Group differences. If one group reacts much more coolly than another, that points to a segment worth checking with real people.
• Like-for-like comparisons. All versions go to the same participants in the same requests, so a version comparison inside one test is fair, though the AI can be influenced by the order versions are listed in.
• Hypotheses to test. Suggested fixes, risks, alternatives and the SWOT are labeled as AI hypothesis, not findings.

Code checks the written report before you see it: numbers that don’t match the results are removed, invented product claims, competitor prices, market sizes and unsourced citations are stripped, and quotes have to match what the participant actually answered. Every page is labeled Simulated result, AI hypothesis, Your input or Verified source.

What results cannot establish

• Demand. No one spent money or signed up.
• A real conversion rate. “40% would try” says nothing about how many visitors to a real page would.
• The right price. Any “best price” in a report is the AI’s judgment from the answers, not calculated from a price curve.
• That a small gap matters. Treat close results between versions as a tie.
• Anything outside your description. Participants don’t see your website, your competitors or reviews.

Precision is not accuracy

A report might say 40% for one version and 41% for another. Those numbers are exact tallies of simulated answers. They are not estimates of real behavior to the nearest percent.

Precision describes how finely something is measured. Accuracy describes how close it is to reality. A simulation can be perfectly precise about its own answers and still be far from what real customers would do, and no accuracy number is claimed for SellTo.us results.

A useful habit: round every simulated figure in your head to “low”, “middling” or “high”, and focus on the reasons.

Turning findings into real experiments

Imagine you test a subscription app that plans weekly workouts around a parent’s schedule. The report shows a middling share would try, with “Doesn’t work around my kids’ sleep” and “Another subscription I’ll forget” as frequent reasons, and one group, shift-working parents, far cooler than the rest.

Here’s how to turn that into real evidence:

1. Write each recurring reason as a testable statement. “Parents worry about forgetting a subscription” becomes “a free first month with a reminder before billing increases real sign-ups.”
2. Pick the riskiest one first. Our guide to finding your riskiest assumptions shows how to rank them.
3. Check it with real people. Talk to five to ten shift-working parents about how they exercise now.
4. Run a small behavior test. A landing page or waitlist with two offers measures what people do, not what personas say.
5. Use the re-test for wording changes. After a rewrite, a re-test compares old and new versions with the same simulated crowd and calls a difference only above a fixed threshold of at least 2 percentage points. That’s a decision rule, not proof.

Your next step

Before you run a test, write your audience as specifically as you can, because the groups are built from it. When the report arrives, read the reasons first, the group differences second and the headline percentage last.

You can run your first 1,000-participant test (free if eligible) or open the sample report to practice reading one. For the full technical detail, see how it works.

Written by the SellTo.us Team. Examples in this article are hypothetical. SellTo.us results come from AI-simulated participants, not real people, and don’t predict sales or guarantee success.