How often we are right
Each number on this page comes from a test with a fixed method and a control. The misses come first. Updated Nov 10, 2026.
What does not work yet
Listed first, so you read our limits before our results.
- 1
Weft finds almost no companies before their first raise
We swept Show HN, Launch HN, the YC directory and X for 400 early Form D filers. 14 names matched before the filing. A placebo set of 2016 filers matched just as often, and 0 of the matches we checked by hand were the same company.
- 2
No signal on research abstracts
On 1,295 SBIR Phase I award abstracts the scorer ranks at chance: AUC 0.53, lift 1.02 in the top 20%.
- 3
The score is a rank, not a probability
Scores sit between 0.38 and 0.58 while 8.6% of companies win. A constant guess of the base rate has a better Brier score (0.079) than ours (0.224). Read the score as an order, never as a chance.
- 4
The model remembers famous companies
From masked launch posts, the model named 147 of 524 YC companies (28%). Every number below removes the companies it can name.
- 5
We withdrew an earlier result
We once reported AUC 0.86 on 161 YC launches. The model knew 74 of those companies; on the ones it did not know, AUC was 0.76 to 0.79. The 0.86 was memory, and we no longer use it.
- 6
A bug flattened scores in production
Before a fix on Nov 10, 2026, the scorer returned 50 for 519 of 524 launch posts, because the prompt was cut before its output format. Every real application longer than about 700 characters was affected.
- 7
Some sets are too small to decide
The YC held-out split has 19 winners after leak removal. The protocol needs 30, so that split alone proves nothing.
Picking winners from launch text
How many of the scorer's top-20% picks became winners, and the lift: that share against the share of winners overall, where picking at random gives 1.0. 95% intervals in brackets. The YC scores are from the read after a label fix, which recorded no top-20% count; the plain-Claude control on that row is from the first read and was not re-run.
| Test | Companies | Winners | Winners among top-20% picks | Lift | AUC | Plain Claude, same rows |
|---|---|---|---|---|---|---|
| Founder posts on Hacker News, 2020 to 2023 (held out) | 5,983 | 65 | 34 of 1,197 | 2.62 [2.09, 3.28] | 0.71 [0.63, 0.78] | Lift 1.15 [0.00, 2.50], AUC 0.54 |
| YC launch posts | 353 | 32 | Not recorded | 2.49 [1.33, 2.94] | 0.68 | AUC 0.50 [0.39, 0.60], 4 of 72 in the top 20% |
| SBIR Phase I awards, 2019 to 2023 | 1,295 | 44 | 9 of 259 | 1.02 [0.40, 1.54] | 0.53 | AUC 0.46 |
Founder posts on Hacker News, 2020 to 2023 (held out)
YC launch posts
SBIR Phase I awards, 2019 to 2023
Read: on founder posts and YC launches the scorer finds about 2.5 times more winners than chance, and plain Claude does not. On research abstracts it finds none. Part of the Hacker News lift comes from the Launch HN tag, which marks YC companies; without it, on Show HN posts alone, the lift is 1.93 [1.28, 2.75]. That misses our bar, a lower bound of 2.0, so it is not met. One limit we cannot rule out: the scorer is a language model that knows companies up to 2025. The Hacker News test removed every company it could name (554 rows, which did not score better than the rest), but a company it knows without naming it could remain.
Finding companies before they raise
400 early companies that filed a first Form D from Aug to Nov 2026. Found means seen at least 30 days before the filing. The placebo is 400 filers from 2016 run through the same matcher.
| Source | Found early | Placebo | Same company or person, checked by hand |
|---|---|---|---|
| Show HN and Launch HN posts | 10 of 400 (2.5%) | 8 of 400 (2.0%) | 0 of 10 |
| YC directory | 4 of 400 (1.0%) | 12 of 400 (3.0%) | 0 of 4 |
| Founder posts on X (6 of 15 months) | 0 of 400 | 0 of 400 | None to check |
| People on earlier Form D filings | 78 of 400 (19.5%) | 12 of 400 (3.0%) | First 40 read: roughly 22 the same person, 6 a coincidence, 12 unclear |
Show HN and Launch HN posts
YC directory
Founder posts on X (6 of 15 months)
People on earlier Form D filings
Read: public company names find no more than the placebo. People do better. For 1 in 5 early filers, an officer was already on an earlier Form D, years before. That finds the founder, not the new company: seeing the company needs a feed of job changes, which we do not have yet.
How we test
The outcome
A company wins when it files a Form D of $5M or more within 36 months. The label is fixed before any scoring. It is a proxy: it misses rounds that never reach the SEC.
Only what was known then
The model reads the text the founders wrote at launch, with company names, founder names and links masked. No later facts.
A leak check first
The model tries to name each company from the masked text. Every company it names is removed from every number.
Controls on the same rows
Each test reports the random base rate and plain Claude asked for a chance from 0 to 100. Weft sources also run on a placebo set of 2016 filers with the same matcher.
Ranking metrics, not accuracy
The main number is how many winners land in the top 20% of picks, against the base rate (lift), with a 95% interval. AUC is second. Plain accuracy is never a headline.
Reports behind this page: docs/EVAL-REALWORK-YC-V2-2026-11-10.md, docs/EVAL-REALWORK-HN-TOUCH1-2026-11-10.md, docs/EVAL-REALWORK-SBIR-2026-11-10.md, .plans/weft-discovery.md, docs/JUDGE-PROTOCOL.md. Each test scored the scorer as it ships, with nothing tuned on the rows it is scored on.