Evidence
What was measured, on what data, and what it does not prove.
The forecast beats both baselines worth beating by a wide margin. It does so on synthetic data the system generated itself, and no real clinic data has ever entered it. Both halves of that sentence matter, so both are on this page.
Accuracy
Against the two baselines worth beating
Mean absolute error in technician minutes. Time-based split: 12,154 training rows, then 2,318 holdout rows covering 2026-06-10 to 2026-09-07, all later in time than every training row. Model v1-20260907003430. Measured on synthetic data.
| Predictor | Error, minutes |
|---|---|
| This model | 24.44 |
| Fixed technician-to-provider ratioConverted to minutes so both are scored on the same target. | 208.61 |
| Historical mean by subspecialty | 63.12 |
- Root mean squared error
- 31.2 minAbove the mean error, which says the misses are uneven: most sessions are close and a few are much further off.
- Mean absolute percentage error
- 9.22 percentOn the same synthetic holdout.
- Interval coverage
- 78.39 percentAgainst a nominal 80 percent, so the range is slightly narrower than advertised. That gap is reported, not smoothed.
Source docs/model-card.md, Evaluation
What this result does not prove
The generator wrote the ground truth the model then learned, so a model fitted to it will look better than the same approach on real scheduling data, where mix, staffing and behaviour all shift. This is evidence the pipeline works end to end. It is not evidence that workload in your clinic is predictable to this degree, and it should not be quoted as if it were. Only a validation on real data could say otherwise, and none has been run.
Why the two error measures are both here
The two failures are not symmetrical. A session over-forecast by 20 minutes wastes a fraction of a shift. A session under-forecast by 90 minutes runs short, and patients wait. Watching the squared measure alongside the average is how a recalibration would notice the tail growing while the average holds. Neither number is a service level guarantee, and the range is a prediction interval for one session, not a confidence interval about an average.
Where it is weakest
Error by subspecialty
| Subspecialty | Error, minutes | Sessions |
|---|---|---|
| Comprehensive | 26.9 | 495 |
| Retina | 26.0 | 570 |
| Glaucoma | 24.6 | 498 |
| Pediatrics | 23.4 | 175 |
| Neuro-ophthalmology | 22.6 | 167 |
| Oculoplastics | 22.5 | 175 |
| Cornea | 18.6 | 238 |
Source docs/model-card.md, Known limitations
What drives the forecast
Permutation importance
- Scheduled patients65.07
- Provider history22.57
- Provider pattern on visual field3.60
- Imaging share of the mix3.47
- Follow-up share of the mix2.70
- Provider pattern on new patient2.11
- Complex patients1.97
- Subspecialty1.38
- Half-day0.89
- Site0.85
Source docs/model-card.md, Permutation importance on the holdout
The refusals
Claims this product will not make
- That any accuracy, coverage or operational figure describes a real clinic. It describes a generated world.
- That a pilot has run, or that cycle time, idle time, overtime or delayed testing improved anywhere. No outcome has been observed.
- Any return on investment, payback or savings figure of our own. The economics module returns arithmetic on rates you supply, applied to a simulated improvement over synthetic history. The output is your figure, not our claim.
- That the fairness report detects real bias. It detects a pattern the generator planted.
- That the weekly hours rule is proven under pressure. This data never stresses it, so it binds only in hand-built tests.
- That a plan is optimal. Within its time limit the solver returns a feasible plan, and two runs can differ.
- That the system is HIPAA compliant or production ready.
- That schedulers find the explanations usable. No human factors evidence exists.
Source docs/paper-traceability.md, Claims the product must never make
Limits
What this prototype cannot show
- No outcome has been observed. Cycle time, idle time, overtime and delayed testing all come from the generator, not from a clinic.
- The baseline comparison is a simulation that reuses the same hidden workload function that generated the data. It is a check on internal consistency, not evidence of any real-world effect.
- The fairness report detects a pattern deliberately planted in the synthetic history so the report has something to find. It says nothing about any real technician or any real site.
- The weekly hours cap is never stressed by this data. Planning a full week leaves the busiest technician at about 20 hours against caps of 24 to 40, so that rule is exercised only in hand-built tests.
- No human factors evidence exists. Whether schedulers trust the recommendations, and whether the explanations are understandable at six in the morning before clinic, has not been tested.
- No cost figures. No claim about return on investment, payback or savings is supported by anything in the repository.
Source docs/pilot-evaluation-design.md, What this prototype cannot show; docs/paper-traceability.md, Claims the product must never make
Engineering
What is actually tested
The backend test suite runs 512 passing tests with 2 skipped, covering the solver, the eight hard constraints, override validation, plan lineage, the audit chain and the API contract. This is evidence about the code, not about a clinic.
Source backend/tests
Verified Measured 2026-09-07 by running the backend suite with a five second solver limit.