Backtest, regenerated 2026-11-02

We priced 1,320 public bid openings the model had never seen.

Trained only on contracts opened 2013 to 2024, then scored on every bid filed in 2025 and 2026: 7,728 bids, each bidder playing the customer in turn. Here is what it gets right, what a simple rule matches, and where it misses.

Cut your price 10%, plumbing and HVAC

44% said

46% would really have been lowest against the bids filed (43% to 49%). A count of bidders says the same chance at every price.

Whole price curve, plumbing and HVAC (AUC)

0.78

Each bid moved from 30% under to 20% over. Ranking by the move alone, no model, scores 0.77: the lift is small, the calibration above is the point.

Where the low bid lands

90%

of lowest competing bids fell inside the 80% band. Median miss 16%.

At the price bid: Brier, all trades

0.133

vs 0.143 for a flat base rate and 0.133 for 1 / predicted bidders, about a tie (AUC 0.70 against 0.70). Lower is better.

5,752 contracts 29,404 bids $14.8B in winning bids 2013 to 2026 MA, NJ, NY, OH, PA

The short version

What the price model adds: how your chance to be low moves when your price moves. At the price you bid, a count of bidders does as well.

What a count cannot give you: the curve. Move your price and the model says how your chance moves. On plumbing and HVAC test bids, cutting 10% took the predicted chance to 44% where 46% would really have been lowest; at the price bid it said 28%, 27% won; raising 10% it said 18%, 16% would have won.
It beats the base rate, out of time. Brier 0.133 against 0.143 for giving every bid the training years' win rate, on 1,320 contracts it never saw (difference -0.012 to -0.009, 95% interval, bootstrap by contract).
Most of that comes from predicting how many will bid. The field-size model alone, which ignores every price, scores 0.1334 against the model's 0.1328, with the same AUC (0.70). At the price a firm actually bids, its own number tells the model where the market sits, not whether that number is low.
Told the true number of bidders, it ties 1-in-N. 0.115 against 0.115 for one over the true field size (AUC 0.77 vs 0.77). Knowing who shows up is worth more than anything else here, which is why plan holders and the firm's own record matter.
It does not yet know better than the firm what to bid for profit. Following its most-profit price on 1,015 plumbing and HVAC test bids earned −18% against bidding the number the firm actually bid (95% interval −35% to −1%, cost taken as bid / 1.12). So the app starts from the firm's own number and shows what moving it would do.
Plumbing and HVAC are harder than the average job. AUC 0.61 (0.57 to 0.64) on 291 contracts, against 0.70 across all trades. Brier 0.194 against 0.203 for the base rate.

How we tested

The same code that prices a live bid, fitted only on the past.

Strict time split

The model is fitted on 4,394 contracts opened 2013 to 2024 and never sees a 2025 or 2026 result. The year-by-year check below refits before every test year.

Every bidder plays you

For each test bid, the bidder's own number is its reference price, the number of bidders is predicted, and the model gives its chance of being lowest. The published tab says whether it was.

Four bars to clear

A flat base rate; one over the predicted number of bidders (no prices at all); one over the true number (which no one knows before opening); and the model itself told the true number.

Ranges counted by contract

Bids on one contract rise and fall together, so every range is worked out by redrawing whole contracts (1,000 times), not single bids.

Scores against the bars

Brier score at the price actually bid

Lower is better. Whiskers are 95% intervals. The two bars at the bottom know the true number of bidders, which nobody knows before bids open: they are the ceiling, not a rival.

7,728 bids on 1,320 contracts, 2025 and 2026. Plumbing and HVAC: 1,063 bids on 291 contracts.

Method	Brier	95% interval	Log loss	AUC	Mean chance
Base rate Base rate: every bid gets the training years' win rate	0.143	0.138 to 0.148	0.462	0.50	21.1%
1 / predicted bidders 1 / predicted field size: the field-size model alone, no prices	0.133	0.128 to 0.139	0.427	0.70	19.8%
Win-chance model Price to win, field size predicted (what the app shows before bids open)	0.133	0.127 to 0.139	0.425	0.70	18.5%
1 / true bidders 1 / true field size: knows how many bid, ignores every price	0.115	0.110 to 0.120	0.375	0.77	17.1%
Win-chance model, bidders known Price to win, told the true field size	0.115	0.110 to 0.120	0.375	0.77	17.2%

Test bids won: 17.1%. Base rate learned in training: 21.1%.

Each dot is a tenth of the 7,728 test bids, sorted by the chance the model gave. On the diagonal = said 20%, won 20%.

Calibration

When it says 25%, does it happen a quarter of the time?

In 5 of 10 deciles what it said falls inside the 95% interval of what happened; in 5 more it said too much. It runs a little high, worst at the ends: the lowest tenth was given 7.7% and won 3.6%; the highest tenth was given 35.9% and won 31.6%. Vertical bars are 95% intervals (bootstrap by contract).

Decile	Bids	Said	Won	95% interval
1	773	7.7%	3.6%	2.8% to 4.3%
2	773	8.4%	7.0%	6.1% to 8.2%
3	773	10.2%	7.6%	6.3% to 9.5%
4	772	11.8%	10.6%	9.1% to 12.4%
5	773	15.3%	12.7%	10.8% to 14.7%
6	773	18.8%	18.4%	16.5% to 20.5%
7	772	21.9%	22.8%	20.8% to 25.0%
8	773	25.1%	26.9%	24.6% to 29.2%
9	773	29.6%	29.8%	27.4% to 32.4%
10	773	35.9%	31.6%	29.3% to 34.5%

The price curve

Move your price, and your chance moves the way the model said.

Every test bidder's price moved from −30% to +20% and checked against the bids actually filed (assuming the others bid the same). The line is the model's average chance; the dots are how often the moved price really was lowest, with 95% intervals.

The whole-curve AUC is 0.81; ranking by the price move alone, with no model, already scores 0.78, because lower prices win more often. What the model adds is the level: how much a given cut is worth on this kind of job. On plumbing and HVAC, 10% above the price bid it runs 2 points high.

1,063 plumbing and HVAC bids, 7,728 across all trades, 2025 and 2026.

Each year scored by a model fitted only on the years before it.

Every year, fitted only on the past

Better than a flat guess in 8 of 9 years.

Refit before every test year from 2018, the model beat the base rate in 8 of 9 years, but beat the price-blind field-size model in only 3. Pooled over all 5,226 contracts: model 0.151, field-size model alone 0.148, base rate 0.159. That is the plainest statement of where the edge comes from today.

By trade and by state

The same test, cut the ways a contractor would read it.

Brier at the price bid (lower is better) and AUC. A slice under 15 contracts is marked thin: read it as an anecdote.

By trade

Trade	Contracts	Model	Base	Equal shot, real bidder count	AUC
Plumbing and HVAC	291	0.194	0.203	0.160	0.61
Plumbing	95	0.214	0.223	0.174	0.55
HVAC	196	0.186	0.194	0.154	0.63
All trades	1,320	0.133	0.143	0.115	0.70

By state, plumbing and HVAC

State	Contracts	Model	Base	Equal shot, real bidder count	AUC
Massachusetts	30	0.188	0.161	0.149	0.59
New Jersey	24	0.172	0.173	0.158	0.59
New York	168	0.197	0.215	0.155	0.68
Ohio thin	5	0.234	0.251	0.191	0.53
Pennsylvania	64	0.200	0.212	0.180	0.63

The model trails the base rate here: Massachusetts. Read those prices with more caution.

All trades: MA 0.189 vs 0.182; NJ 0.170 vs 0.172; NY 0.110 vs 0.125; OH 0.193 vs 0.195; PA 0.195 vs 0.203.

Where the low bid lands

Given your own number, within 16% half the time. Without one, contract value is a weak guess.

Median miss between the predicted and actual lowest competing bid on plumbing and HVAC test bids. In short: from your own number, the model's median ties a no-model rule (your number times the training years' typical ratio, 0.875), which misses by 16%. What the model adds is the band: 90% of actual low bids fell inside its 80% range (a little wide). With no number of your own, similar past jobs miss by 68%, so the app shows them for size and never prices from them.

Median absolute percent error, plumbing and HVAC, 2025 and 2026. n is bids (own number) or contracts (estimate, similar jobs).

Would following it have paid?

We pretended to be every plumbing and HVAC bidder and followed each pricing rule.

1,015 test bids on 243 contracts. A public tab has no costs, so each bidder's cost is its bid divided by a markup. A win is a price under the lowest other bid filed. Money left on the table = second-lowest bid minus ours, on a win. No rule beat bidding the firm's own number at every markup with its 95% interval clear of zero, so the app does not move a firm's number for profit.

Profit against bidding the number the firm actually bid. Whiskers: 95% interval, bootstrap by contract.

Rule (cost = bid / 1.08)	Won	Win rate	Profit	vs own number	95% interval	Money left
Bid the number the firm actually bid	243	23.9%	$37.19M			$71.27M
Most profit within 5% of the firm's own number	173	17.0%	$35.74M	−4%	−24% to +18%	$54.5M
Most profit on average (cap 35% over cost)	61	6.0%	$24.34M	−35%	−60% to −6%	$32.59M
Highest price with a 20% chance	147	14.5%	$37M	−1%	−11% to +10%	$61.39M
Highest price with a 30% chance	258	25.4%	$27.29M	−27%	−37% to −17%	$90.85M
Highest price with a 40% chance	266	26.2%	$30.69M	−17%	−31% to −7%	$81.79M

Rule (cost = bid / 1.12)	Won	Win rate	Profit	vs own number	95% interval	Money left
Bid the number the firm actually bid	243	23.9%	$53.79M			$71.27M
Most profit within 5% of the firm's own number	179	17.6%	$49.41M	−8%	−20% to +3%	$61.63M
Most profit on average (cap 35% over cost)	93	9.2%	$44.11M	−18%	−35% to −1%	$46.18M
Highest price with a 20% chance	148	14.6%	$50.57M	−6%	−15% to +3%	$61.49M
Highest price with a 30% chance	260	25.6%	$48.83M	−9%	−20% to +1%	$93.37M
Highest price with a 40% chance	364	35.9%	$28.52M	−47%	−56% to −38%	$122.73M

Rule (cost = bid / 1.20)	Won	Win rate	Profit	vs own number	95% interval	Money left
Bid the number the firm actually bid	243	23.9%	$83.68M			$71.27M
Most profit within 5% of the firm's own number	216	21.3%	$93.51M	+12%	−1% to +24%	$81.44M
Most profit on average (cap 35% over cost)	189	18.6%	$91.44M	+9%	−3% to +21%	$80.44M
Highest price with a 20% chance	161	15.9%	$75.67M	−10%	−18% to −2%	$62.03M
Highest price with a 30% chance	261	25.7%	$91.77M	+10%	−3% to +21%	$93.39M
Highest price with a 40% chance	394	38.8%	$79.13M	−5%	−17% to +5%	$132.51M

Agent replay

The pricer, the day before 12 real openings. Misses included.

The 12 most recent plumbing, HVAC or mechanical contracts opened in 2025 to 2026 with 3 or more responsive bids, newest first. Chosen by date alone; misses stay in. For each, the model was refitted on every contract opened before that day, and priced the bidder in the middle of the field ("you", a fixed rule) from its own number, with cost taken as its bid / 1.12.

11 of 12lowest competing bids landed inside the predicted 80% band

14%median miss of the predicted low bid

1 of 9won at the 30%-chance price (the chances added up to 2.8 wins)

2.95bidders off, on average, in the predicted field size

2026-09-11City of WorcesterMAHVAC

Boiler Replacement, Vernon Hill School / WPS

Low bid in band Target price would have won

Bidders: 4 (predicted 4.3)
"You" bid: $139K chance 17%
Needed to win: under $133K (−4%, chance 22%)
30% target: $125K (−10%)
Low bid band: $74K to $171K median $115K
Trained on: 5,696 contracts to 2026-09-10

2026-09-03NJ DPMCNJPlumbing

SSP Restroom Renovations, NJDOT Cherry Hill Maintenance Yard, Cherry Hill,...

Low bid in band No price above cost had a 30% chance

Bidders: 5 (predicted 6.2)
"You" bid: $1.91M chance 7%
Needed to win: under $1.77M (−7%, chance 14%)
30% target: none above cost
Low bid band: $822K to $1.92M median $1.41M
Trained on: 5,689 contracts to 2026-09-02

2026-09-02State University Construction FundNYHVAC

Replace Steam Boilers 3 & 4, Physical Plant

Low bid in band Target price would have lost

Bidders: 17 (predicted 5.3)
"You" bid: $9.96M chance 21%
Needed to win: under $6.99M (−30%, chance 84%)
30% target: $9.46M (−5%)
Low bid band: $6.55M to $11.52M median $8.95M
Trained on: 5,682 contracts to 2026-09-01

2026-09-02State University Construction FundNYHVAC

Renovate Mechanical Systems Neuberger Museum

Low bid in band Target price would have lost

Bidders: 5 (predicted 5.2)
"You" bid: $7.96M chance 21%
Needed to win: under $7.15M (−10%, chance 44%)
30% target: $7.56M (−5%)
Low bid band: $5.18M to $9.31M median $7.14M
Trained on: 5,682 contracts to 2026-09-01

2026-08-27DASNYNYHVAC

CUNY New York City College of Technology Namm Building Hot Water Perimeter...

Low bid in band No price above cost had a 30% chance

Bidders: 11 (predicted 10.0)
"You" bid: $25.8M chance 7%
Needed to win: under $21.44M (−17%, chance 47%)
30% target: none above cost
Low bid band: $16.18M to $25.72M median $21.55M
Trained on: 5,679 contracts to 2026-08-26

2026-08-27NJ DPMCNJHVAC

Chiller Replacement and Associated Repairs, Joseph Kohn Training Center,...

Low bid outside band Target price would have lost

Bidders: 10 (predicted 6.0)
"You" bid: $896K chance 18%
Needed to win: under $572K (−36%, chance 92%)
30% target: $833K (−7%)
Low bid band: $596K to $991K median $790K
Trained on: 5,679 contracts to 2026-08-26

2026-08-26NYS Office of General ServicesNYHVAC

Replace Fan Coil Units, Building 80, Phase II

Low bid in band Target price would have lost

Bidders: 14 (predicted 6.7)
"You" bid: $5.8M chance 15%
Needed to win: under $4.45M (−23%, chance 70%)
30% target: $5.34M (−8%)
Low bid band: $3.65M to $6.34M median $5.08M
Trained on: 5,667 contracts to 2026-08-25

2026-08-26NYS Office of General ServicesNYHVAC

Abate/Replace Flooring & Ceiling

Low bid in band Target price would have lost

Bidders: 3 (predicted 4.3)
"You" bid: $561K chance 26%
Needed to win: under $490K (−13%, chance 47%)
30% target: $539K (−4%)
Low bid band: $318K to $806K median $499K
Trained on: 5,667 contracts to 2026-08-25

2026-08-20Boston Public FacilitiesMAHVAC

City Hall Automatic Switch

Low bid in band Target price would have lost

Bidders: 6 (predicted 3.5)
"You" bid: $141K chance 32%
Needed to win: under $116K (−18%, chance 65%)
30% target: $143K (+1%)
Low bid band: $86K and up median $133K
Trained on: 5,661 contracts to 2026-08-19

2026-08-19NYS Office of General ServicesNYHVAC

Provide Central Air Conditioning & Upgrade Bathrooms, Building 110

Low bid in band Target price would have lost

Bidders: 8 (predicted 8.2)
"You" bid: $9.39M chance 13%
Needed to win: under $8.25M (−12%, chance 40%)
30% target: $8.55M (−9%)
Low bid band: $6.03M to $9.9M median $8.11M
Trained on: 5,643 contracts to 2026-08-18

2026-08-19NYS Office of General ServicesNYHVAC

Provide Transitional Residence, South Campus

Low bid in band Target price would have lost

Bidders: 12 (predicted 8.3)
"You" bid: $3.58M chance 12%
Needed to win: under $2.5M (−30%, chance 77%)
30% target: $3.22M (−10%)
Low bid band: $2.14M to $3.8M median $3.02M
Trained on: 5,643 contracts to 2026-08-18

2026-08-19NYS Office of General ServicesNYPlumbing

Provide Transitional Residence, South Campus

Low bid in band No price above cost had a 30% chance

Bidders: 5 (predicted 7.0)
"You" bid: $2.83M chance 15%
Needed to win: under $2.48M (−12%, chance 33%)
30% target: none above cost
Low bid band: $1.48M to $3.18M median $2.31M
Trained on: 5,643 contracts to 2026-08-18

Over only these contracts; a dozen openings is an illustration, not a score. The scores are the holdout sections above.

Naming the GC before the permit does

The top pick was the GC later named 21% of the time in New York City, against 4% for picking the busiest GC.

From a new building filing that names an owner and an architect but no GC yet, how often do the public histories put the GC that was later named first (top-1) or in the first three (top-3)? It is a ranked guess. Backtested in New York City, Chicago, Philadelphia; in the app it ranks open New York City filings today.

GC prediction backtest by city
City	Filings tested	Top pick right	In the top three	Busiest GC, top pick	Busiest GCs, top three
New York City	4,648	21% (20% to 22%)	35% (33% to 36%)	4%	10%
Chicago	2,032	25% (23% to 27%)	34% (32% to 36%)	5%	13%
Philadelphia	2,272	12% (11% to 14%)	18% (17% to 20%)	3%	6%

Method. Score each GC by the share of earlier jobs with a known GC it built for this owner, this owner's signatory, this architect or engineer (license number) and this filing representative, plus its share of jobs in the borough and size band, nudged by how close the job is to its typical size. Weights tuned on the last fifth of the training period only. The test filings come after the training cutoff (2025-10-30 in New York City); the signals are the owner, the owner's signatory, the architect or engineer, the filing representative. Baseline: the GCs with the most jobs in the filing's borough and size band before the cutoff. Ranges are 95% Wilson score intervals.

What the New York City test does not show.

DOB NOW publishes each job filing as it stands today, not as first filed: a field amended after the GC was hired (a new filing representative, a corrected owner) is seen as if it were known at filing. The check below counts the obvious case (the filing representative is the GC itself).
Test jobs are those filed in the last 12 months whose GC permit has already been issued; jobs still waiting for a GC permit are not in the test, so the test leans to jobs that moved quickly.
The GC is the permittee holding a GC license on the General Construction permit (else Foundation, Structural or Earth Work); a construction manager that is not the permittee is not seen.
Owners that file as single-purpose LLCs rarely repeat; no parent company is guessed. The signatory signal (a person's name on the filing, stored hashed) links many of them.

Generated 2026-11-02 by python scripts/gc_predict_report.py.

Limitations

What this backtest does not show.

Contract value is a weak prediction. With no price of your own and no engineer's estimate, similar past jobs placed the low bid with a median miss of 68% (174 of 243 plumbing and HVAC contracts had enough matches). The engineer's estimate alone missed by 26% as printed and 22% through the model, on only 40 contracts.
The edge at the price bid is mostly field size. One over the predicted number of bidders scores 0.1334 against the model's 0.1328; with the true field size the model ties one over N. A firm's own record and plan-holder lists are where we expect the next gain; they are not in these scores.
Counterfactual prices assume the others bid the same. The curve check and the policy replay move one bidder's price against the bids actually filed. A real market would react a little.
Costs are assumed. Public tabs show prices, not costs, so every profit figure uses cost = bid / markup at three markups. Firm history uploaded to the app replaces that assumption for that firm only.
Too confident at the ends. The lowest and highest deciles of predicted chance were off by several points (see calibration), and above the price bid the curve runs a few points high.
Coverage. 5,752 contracts from 14 public owners in MA, NJ, NY, OH, PA; plumbing and HVAC are 1,155 of them. Owners outside these sources are priced from trade and state alone.

Reproduce it

One command, every number on this page.

python scripts/backtest_report.py refits the model on the public warehouse (75 seconds) and writes backtest.json, which this page reads. Generated 2026-11-02T03:44:34+00:00; the app's live model is ptw-20261001T220731.

Try the live demo Raw report