# SimBasketball Sim Engine Testing Report
This post explains, in plain basketball language, how the SimBasketball engine is tested to make sure it behaves credibly and consistently.
The short version is this: we do **not** rely on one vague “does this feel right?” check. We run a layered validation stack that asks different questions:
1. Is the game internally legal and coherent?
2. Does the overall league environment look like real basketball?
3. Do player traits survive the engine?
4. Do matchups, scouting, and coaching choices move outcomes in the right direction?
5. Can gimmick rosters or one-stat exploits break the sim?
6. Do special situations like transition, late game, and injuries behave plausibly?
7. Are the downstream outputs—box scores, play-by-play, lineup stats, advanced stats—trustworthy?
This post focuses on the tests that touch the **on-court simulation itself** or the integrity of the game data it produces, not generic API, auth, billing, or admin tests.
## The testing stack at a glance
Fast sim tier = “Did we obviously break basketball?”
Slow sim tier = “Did a basketball change really move the underlying math?”
Full validation runner = “Does the full league environment still look like believable basketball?”
Rate-capture tests = “Do players still behave like the players they are supposed to be?”
Matchup and scouting tests = “Do tactics and assignments work like basketball tactics?”
Exploit tests = “Can a user cheese the engine with a fake roster-building trick?”
Output integrity tests = “Can we trust the box score, play-by-play, lineup data, and advanced stats?”
A big methodological point: many of the multi-game tests use **paired series** that alternate home and away and reuse controlled seed patterns. In basketball terms, that means we are trying to isolate the variable under study, not accidentally measure random noise or home-court bias.
## 1. Foundation: can it produce a valid basketball game?
Before we ask whether the engine is realistic, we first ask whether a single game is structurally sound.
This layer checks things like:
1. The engine returns a complete, valid game result.
2. Scores are in believable ranges.
3. Quarter scores add up to the final score.
4. Possession counts are reasonable for the pace setting.
5. Team minutes stay near the real basketball target of roughly 240 regulation team-minutes.
6. Overtime only happens when regulation ends tied.
7. The same seed produces the same result, and a different seed produces a different result.
8. Possession logs appear when they should and contain valid outcomes.
9. Even degenerate roster constructions—like all bigs or all guards—still complete without the engine breaking.
This is the bedrock. If this layer fails, there is no point talking about realism yet.
## 2. Macro realism: does the overall league environment look like basketball?
This is the broadest realism layer. The engine is run across several matchup families, including:
1. Modern vs modern
2. Classic vs classic
3. Cross-era matchups
4. Stacked team vs average team
5. Vintage vs vintage
6. Modern elite vs modern elite
Across those runs, we check whether the environment lands in believable ranges for:
- average team score
- score variance
- field-goal percentage
- three-point percentage
- free-throw percentage
- foul volume
- free-throw attempt volume
- home-court advantage
- blowout frequency
- stronger-roster win rate
- same-era and cross-era scoring levels
This is the closest thing we have to a league-wide smell test, except formalized into numeric gates.
The core question here is simple:
**If you watched this simulated league on TV, would it still look like basketball?**
## 3. Player identity: do players still look like themselves?
This is one of the most important layers philosophically.
The engine is designed to be **stats-driven**, not ratings-driven. So if we feed in a player who is supposed to be:
- a high-usage creator
- a pass-first playmaker
- a paint-dominant big
- a two-way wing
- a stretch forward
- a low-usage energy player
then over a long enough sample, the output should still resemble that player.
These tests look at whether observed results stay close to the declared player profile for things like:
- usage
- assist rate
- turnover rate
- offensive rebound rate
- defensive rebound rate
- foul-drawing rate
- 2P%
- 3P%
- FT%
- shot mix
In other words, if you tell the engine “this player is a high-usage playmaker,” the box score over time should still look like a high-usage playmaker—not randomly drift into a totally different archetype.
This is crucial because a sim can produce believable average scores while still being wrong about individual player identity. This layer is what protects against that.
## 4. Cause-and-effect testing: when one basketball skill changes, does the right thing happen?
This layer is less about broad realism and more about basketball logic.
The engine runs one-stat-at-a-time sweeps and asks whether the expected downstream effect moves in the proper direction. Examples:
- more passing should improve team shot quality
- more steals should create more opponent turnovers
- more blocks should reduce opponent efficiency
- more offensive rebounding should create more offensive boards
- more defensive rebounding should suppress second chances
- more turnover tendency should produce more turnovers
- more foul-drawing should create more free throws
There are also roster-swap tests that ask questions a coach would instantly understand:
- If we replace a center with an elite rim protector, does opponent scoring drop?
- If we insert an elite scorer, does team offense improve?
- If we insert an elite playmaker, does the team function better?
- If we use a poor shooter or a bad positional fit, does performance suffer?
This is important because it checks **causality**, not just averages. The sim is being asked whether basketball inputs still create basketball consequences.
## 5. Matchups, scouting, and coaching choices
This layer checks whether tactical choices matter the way a basketball person would expect.
### Matchup assignment
The engine verifies that:
1. Every offensive player is covered by exactly one defender.
2. The biggest offensive threat gets the best eligible defender.
3. Fallback logic works when a roster is badly built.
4. Out-of-position matchups are genuinely penalized.
### Matchup quality sweeps
For stars at PG, SG, SF, PF, and C, defender quality is stepped from weak to elite. The tests then check whether:
- star scoring drops
- star FG% drops
- team offense drops
as defender quality improves.
That is a very basketball-native question: if a scorer sees better defense, he should become less productive.
### Scouting behavior
The scouting tests verify that:
- double-teaming a star lowers the star’s points and efficiency
- double-teaming raises the star’s turnover pressure
- double-teaming helps non-target teammates by opening the floor
- hard close-outs reduce a perimeter star’s 3-point volume and shift shot mix inward
### Custom defensive assignments
The engine also tests forced mismatches directly. A good example: if you force a guard to defend a center, the center should see a real scoring and efficiency boost. If that does **not** happen, the matchup logic is not truly routing basketball size/position penalties into the shot outcome math.
This whole layer matters because it answers a key credibility question:
**Do tactical decisions matter in the sim for the same reasons they matter in basketball?**
## 6. Anti-exploit testing: can nonsense roster builds break the sim?
This is a product-integrity layer as much as a realism layer.
The engine creates deliberately extreme or gimmicky rosters such as:
- all 3-point shooting
- all rebounding
- all steal-heavy guards
- all ball-dominant scorers
- minimum-salary junk rosters
- all bigs
- all guards
Those rosters are then tested against balanced teams. The balanced team is not required to win every time, but the exploit roster is **not** supposed to become a magic unbeatable build.
There are also strategy-balance checks:
- fast pace vs slow pace should be tradeoffs, not automatic wins
- perimeter-heavy focus vs balanced should not create a dominant universal strategy
- inside-heavy focus should also be a tradeoff
- no-playmaker lineups should be punished
- one-dimensional shot profiles should be penalized
In simple terms:
**The game should reward coherent roster-building, not cheese.**
## 7. Special situations: transition, late game, injuries, and floor balance
These are the more specific but very important realism layers.
### Transition / fastbreak
The engine checks whether:
- transition happens at a believable frequency
- steals create more transition than ordinary defensive rebounds
- transition offense is more efficient than half-court offense
- transition produces more paint-oriented attack patterns
- “get back” and “crash the boards” change opponent transition frequency the right way
That matters because transition should feel like a different basketball environment, not just normal half-court possessions with a cosmetic label attached.
### Late game
The late-game suite looks at naturally occurring late-game possessions and asks:
- do trailing teams take more threes?
- do trailing teams hunt fouls more aggressively?
- do leading teams shift away from threes?
- does garbage time show up distinctly?
This layer is deliberately a bit looser because late-game possessions are rarer and noisier, but it is still extremely useful. It tests whether the engine’s clutch logic actually shows up in aggregate behavior.
### Injuries and fatigue
This suite works at the season level rather than the single-game level, but it directly affects sim credibility.
It checks whether:
- ironmen stay mostly available
- injury-prone players miss the amount of time they are supposed to miss
- durability ordering is preserved across player types
- the model avoids unrealistic early-season injury avalanches
- injury lengths follow a believable distribution
- fatigue meaningfully affects risk
This is important because a season sim cannot feel credible if availability behaves like pure chaos.
### Floor balance
The floor-balance mechanic is designed to penalize one-dimensional offenses.
The tests use extreme all-paint, all-perimeter, and balanced teams and check whether:
- one-dimensional teams are dragged down
- balanced teams stay competitive or outperform them
- heavily skewed teams do not score as efficiently as a properly varied offense
That matters because a real offense becomes easier to guard when everybody is threatening the same zone in the same way.
## 8. Output integrity: can we trust the data after the game is played?
This is the final layer. Even if the engine simulates good basketball, the product still fails if the outputs are unreliable.
These tests verify:
1. Play-by-play event counts are reasonable.
2. Running scores never break.
3. Final play-by-play score matches the final box score.
4. Offensive-rebound chains appear in the correct chronological order.
5. Lineup stints and 5-man lineup stats build correctly from possession data.
6. Advanced stats like PER, BPM, VORP, and Win Shares are internally consistent.
7. Season summaries carry those advanced stats forward correctly.
Why this matters:
- if the play-by-play is wrong, the game story is wrong
- if lineup data is wrong, on/off analysis is wrong
- if advanced stats are inconsistent, downstream evaluation is wrong
So this layer is really asking:
**After the engine finishes a game, can an analyst trust the evidence it produced?**
## 9. How the tiers are actually used
The testing stack is deliberately split into tiers because not every question needs the same sample size.
### Fast sim tier
This is the quick regression layer. It answers:
**Did we obviously break basketball?**
It is the fastest guardrail after a sim change and includes deterministic checks, golden snapshots, and short-form matchup/scouting tests.
### Slow sim tier
This is the real calibration layer. It answers:
**Did the basketball actually move?**
This tier is used when a change is large enough that subtle statistical drift matters.
### Heavy calibration runs
These are the lower-noise versions of the slow tier and are used when investigating subtle engine behavior. The point is to reduce the chance of drawing conclusions from randomness instead of real signal.
### Standalone validation script
This is the broad environment audit. It is the “step back and look at the whole league” test.
## 10. What all of this means in plain English
When we say the sim is tested for efficacy, we do **not** just mean:
“We ran a few games and the scores looked okay.”
We mean:
- the engine can produce a structurally valid basketball game
- the league environment lands in believable statistical ranges
- players preserve their basketball identities over time
- changing one basketball input moves the right downstream result
- matchups, scouting, and coaching choices behave credibly
- gimmick rosters do not become dominant exploits
- transition, late-game, fatigue, and injuries behave like recognizable basketball contexts
- the play-by-play, lineup data, and advanced stats remain trustworthy
So the standard is much higher than visual plausibility.
The sim is being asked to be:
**legal, believable, causally sane, exploit-resistant, situationally plausible, and analytically trustworthy all at once.**
## Addendum: one current gap
The one meaningful gap identified in the follow-up audit is **path-parity testing**.
The repo strongly validates the engine’s basketball behavior itself, but it does **not** appear to have one explicit automated end-to-end suite proving that the different ways of triggering games—such as scheduler-driven sim, admin/manual sim, and commissioner/manual sim—produce identical outputs from identical inputs.
That is a **consistency gap**, not a realism gap.
In other words:
- the basketball logic itself is being tested heavily
- the product could still benefit from one explicit test proving that every “button that runs a game” is wired to the same behavior
## Bottom line
For a basketball-savvy reader, the validation stack is strong because it attacks the problem from multiple angles:
1. **Environment realism** — do games and leagues look like real basketball?
2. **Player fidelity** — do players still behave like themselves?
3. **Basketball causality** — do traits and tactics move outcomes the right way?
4. **Exploit resistance** — can users break the sim with nonsense?
5. **Situational realism** — do transition, late game, and season attrition feel right?
6. **Data trustworthiness** — can we believe the outputs after the game is played?
That is the real standard the engine is being held to.