IdeaPulley
ArloProductBenchmarkFor foundersPricing
Log inBook a demo

How often we are right

Each number on this page comes from a test with a fixed method and a control. The misses come first. Updated Nov 10, 2026.

What does not work yet

Listed first, so you read our limits before our results.

  1. 1

    Weft finds almost no companies before their first raise

    We swept Show HN, Launch HN, the YC directory and X for 400 early Form D filers. 14 names matched before the filing. A placebo set of 2016 filers matched just as often, and 0 of the matches we checked by hand were the same company.

  2. 2

    No signal on research abstracts

    On 1,295 SBIR Phase I award abstracts the scorer ranks at chance: AUC 0.53, lift 1.02 in the top 20%.

  3. 3

    The score is a rank, not a probability

    Scores sit between 0.38 and 0.58 while 8.6% of companies win. A constant guess of the base rate has a better Brier score (0.079) than ours (0.224). Read the score as an order, never as a chance.

  4. 4

    The model remembers famous companies

    From masked launch posts, the model named 147 of 524 YC companies (28%). Every number below removes the companies it can name.

  5. 5

    We withdrew an earlier result

    We once reported AUC 0.86 on 161 YC launches. The model knew 74 of those companies; on the ones it did not know, AUC was 0.76 to 0.79. The 0.86 was memory, and we no longer use it.

  6. 6

    A bug flattened scores in production

    Before a fix on Nov 10, 2026, the scorer returned 50 for 519 of 524 launch posts, because the prompt was cut before its output format. Every real application longer than about 700 characters was affected.

  7. 7

    Some sets are too small to decide

    The YC held-out split has 19 winners after leak removal. The protocol needs 30, so that split alone proves nothing.

Picking winners from launch text

How many of the scorer's top-20% picks became winners, and the lift: that share against the share of winners overall, where picking at random gives 1.0. 95% intervals in brackets. The YC scores are from the read after a label fix, which recorded no top-20% count; the plain-Claude control on that row is from the first read and was not re-run.

TestCompaniesWinnersWinners among top-20% picksLiftAUCPlain Claude, same rows
Founder posts on Hacker News, 2020 to 2023 (held out)5,9836534 of 1,1972.62 [2.09, 3.28]0.71 [0.63, 0.78]Lift 1.15 [0.00, 2.50], AUC 0.54
YC launch posts35332Not recorded2.49 [1.33, 2.94]0.68AUC 0.50 [0.39, 0.60], 4 of 72 in the top 20%
SBIR Phase I awards, 2019 to 20231,295449 of 2591.02 [0.40, 1.54]0.53AUC 0.46

Founder posts on Hacker News, 2020 to 2023 (held out)

Companies5,983
Winners65
Winners among top-20% picks34 of 1,197
Lift2.62 [2.09, 3.28]
AUC0.71 [0.63, 0.78]
Plain Claude, same rowsLift 1.15 [0.00, 2.50], AUC 0.54

YC launch posts

Companies353
Winners32
Winners among top-20% picksNot recorded
Lift2.49 [1.33, 2.94]
AUC0.68
Plain Claude, same rowsAUC 0.50 [0.39, 0.60], 4 of 72 in the top 20%

SBIR Phase I awards, 2019 to 2023

Companies1,295
Winners44
Winners among top-20% picks9 of 259
Lift1.02 [0.40, 1.54]
AUC0.53
Plain Claude, same rowsAUC 0.46

Read: on founder posts and YC launches the scorer finds about 2.5 times more winners than chance, and plain Claude does not. On research abstracts it finds none. Part of the Hacker News lift comes from the Launch HN tag, which marks YC companies; without it, on Show HN posts alone, the lift is 1.93 [1.28, 2.75]. That misses our bar, a lower bound of 2.0, so it is not met. One limit we cannot rule out: the scorer is a language model that knows companies up to 2025. The Hacker News test removed every company it could name (554 rows, which did not score better than the rest), but a company it knows without naming it could remain.

Finding companies before they raise

400 early companies that filed a first Form D from Aug to Nov 2026. Found means seen at least 30 days before the filing. The placebo is 400 filers from 2016 run through the same matcher.

SourceFound earlyPlaceboSame company or person, checked by hand
Show HN and Launch HN posts10 of 400 (2.5%)8 of 400 (2.0%)0 of 10
YC directory4 of 400 (1.0%)12 of 400 (3.0%)0 of 4
Founder posts on X (6 of 15 months)0 of 4000 of 400None to check
People on earlier Form D filings78 of 400 (19.5%)12 of 400 (3.0%)First 40 read: roughly 22 the same person, 6 a coincidence, 12 unclear

Show HN and Launch HN posts

Found early10 of 400 (2.5%)
Placebo8 of 400 (2.0%)
Same company or person, checked by hand0 of 10

YC directory

Found early4 of 400 (1.0%)
Placebo12 of 400 (3.0%)
Same company or person, checked by hand0 of 4

Founder posts on X (6 of 15 months)

Found early0 of 400
Placebo0 of 400
Same company or person, checked by handNone to check

People on earlier Form D filings

Found early78 of 400 (19.5%)
Placebo12 of 400 (3.0%)
Same company or person, checked by handFirst 40 read: roughly 22 the same person, 6 a coincidence, 12 unclear

Read: public company names find no more than the placebo. People do better. For 1 in 5 early filers, an officer was already on an earlier Form D, years before. That finds the founder, not the new company: seeing the company needs a feed of job changes, which we do not have yet.

How we test

The outcome

A company wins when it files a Form D of $5M or more within 36 months. The label is fixed before any scoring. It is a proxy: it misses rounds that never reach the SEC.

Only what was known then

The model reads the text the founders wrote at launch, with company names, founder names and links masked. No later facts.

A leak check first

The model tries to name each company from the masked text. Every company it names is removed from every number.

Controls on the same rows

Each test reports the random base rate and plain Claude asked for a chance from 0 to 100. Weft sources also run on a placebo set of 2016 filers with the same matcher.

Ranking metrics, not accuracy

The main number is how many winners land in the top 20% of picks, against the base rate (lift), with a 95% interval. AUC is second. Plain accuracy is never a headline.

Reports behind this page: docs/EVAL-REALWORK-YC-V2-2026-11-10.md, docs/EVAL-REALWORK-HN-TOUCH1-2026-11-10.md, docs/EVAL-REALWORK-SBIR-2026-11-10.md, .plans/weft-discovery.md, docs/JUDGE-PROTOCOL.md. Each test scored the scorer as it ships, with nothing tuned on the rows it is scored on.

See it on your own deal flow
Book a 30-minute demo

IdeaPulley

Know which startups will last, and prove it.

Product

  • Arlo
  • Deal intake
  • Decision log
  • LP reports

Resources

  • VCBench results
  • API documentation
  • MCP server
  • Changelog
  • Learn
  • Free VC templates

Company

  • About
  • Founder portal
  • Pricing
  • Contact
© 2026 IdeaPulley. All rights reserved.Privacy policyTerms of serviceSecurityStatus