Why a model can be 99% accurate and still struggle when it matters most

01 / A striking headline result

An estimator can classify a broad set of synthetic forests extremely well.

The first experiment creates 324 mathematically generated forest worlds. The experimenter knows the underlying state and demographic parameters, so reference Health classifications can be calculated. The estimator instead receives noisy or incomplete observations and must reconstruct the state, estimate the dynamics, and classify prospective Health.

The result is a property of this controlled synthetic population and estimator. It is not a measured diagnostic accuracy for the real Harvard Forest.

02 / Understand the metric

Balanced accuracy gives positive and negative cases equal weight.

Sensitivity asks how often reference-positive cases are correctly classified. Specificity asks how often reference-negative cases are correctly classified. Balanced accuracy averages these two rates.

The metric helps us compare two kinds of correct answer. It does not make the test population representative of the hard decisions we eventually care about. In particular, a test with few examples near its decision boundary cannot establish precision at that boundary.

03 / Deliberately difficult conditions

What happens when we deliberately place worlds near 75%?

In Lesson 15, Q₁ asks whether viable mass reaches θ = 0.75. A viable mass of 0.90 passes easily and 0.40 fails easily. But 0.749 and 0.751 lie on opposite sides despite differing by just 0.002—two-tenths of one percentage point.

Paper V constructs 81 synthetic worlds: 72 designed around specified capacity targets and nine present-realization controls. Among the 72 capacity-targeted worlds, 67 end up within 0.10 of the threshold and 23 within 0.02. This is a more demanding designed population.

Source: Paper V, Section 6.6. The Q₁ and Q₂ near-boundary subsets differ from one another and from the 324-world broad-range experiment. The figures compare performance across different designed test populations, not repeated measurements on one identical sample.

These results can all be correct simultaneously. Broad-range accuracy can be excellent while near-threshold classification is unreliable in the chosen experimental conditions. The reported approximately 56% values do not establish a universal accuracy for real-world health measurement.

04 / Where a tiny error becomes a different answer

A small probability error can flip a Boolean Health judgment.

Suppose the underlying probability in a specified model is 0.751 while the estimator reports 0.748. The numerical error is only −0.003, or −0.3 percentage points. Nonetheless, the estimated decision is false while the reference decision is true.

05 / Uncertainty from sampling futures

With 64 simulated histories, one different outcome can reverse the decision.

In a conditional bank with 48 viable histories, the finite-bank estimate is exactly 48/64 = 0.75. But 47 successful futures out of 64 is below 75%, while 49 is above it.

There are two questions here. We can evaluate the finite-bank rule exactly for the histories actually sampled. Estimating the probability of the underlying stochastic generator needs a separate uncertainty analysis. Paper V uses Wilson intervals for binomial proportions to flag cases where the generator-level threshold remains unresolved at the available numerical resolution.

More simulated futures can reduce Monte Carlo uncertainty. They do not demonstrate that the mortality, growth, or entry laws actually describe biological dynamics.

06 / Where is classification information being lost?

A four-way oracle audit separates two information sources.

Because the synthetic worlds have known generating states and parameters, I can supply the estimator with one or both instead of requiring it to infer them. “True” here means known from a synthetic generating world, not independently measured in the real forest.

Now compare the balanced accuracies measured in those conditions on the near-boundary certified cases.

The true starting state helps in this particular experiment when model parameters remain estimated. Providing true parameters without the true state performs worse here than the ordinary estimated/estimated procedure. This is an interaction inside a particular approximate inference pipeline, not a general principle that more reliable information has negative value.

With both generating inputs supplied, the correctly specified synthetic classifier reaches 100% balanced accuracy on its certified test cases. That verifies agreement within the experiment's model family. It provides no independent validation of real-world forest forecasts.

07 / A deliberately selected reconstruction extreme

Similar starting summaries can generate dramatically different continuation estimates.

Paper V compares two approaches to the unequal dates of individual stem measurements. One uses pooled observation times; another respects individual exposure times. In a selected one-year comparison, the starting-state summaries are close—but the separately sampled future banks disagree strongly.

This specific comparison requires juvenile support at one year to reach 80% of the census reference J₀ = 82,077:

The audit localizes every failed trajectory in these banks to the terminal juvenile-support condition. But the two response banks were separately generated from different reconstructions. We cannot attribute the entire difference to 300 juvenile-proxy stems alone. Nor do these numbers measure an observed jump in the real forest's survival probability. They reveal an extreme sensitivity of this stipulated computational specification in a selected case.

08 / When an important process is missing

Simulating a misspecified model more precisely does not make it biologically right.

The next experiment keeps the comparison within synthetic worlds but deliberately changes the truth-generating process. It includes an extra annual hemlock mortality hazard of 0.12 that the estimator omits.

These are broad-range synthetic comparisons across 324 worlds (Paper V, Section 6.6). The estimator may assign excessive probability to continued organization because its future law excludes a mortality mechanism present in the generating law.

More future draws can reduce sampling noise around that estimator's answer. They cannot supply the omitted biological mechanism. This is a problem of model misspecification, which requires model evaluation and independent evidence rather than simply more Monte Carlo draws.

09 / Three problems with different remedies

To improve a measurement, I first need to know what kind of error I'm confronting.

These problems can coexist. The oracle experiment shows state and parameter errors interacting; the near-boundary experiments show classification changing under tiny mass errors; the misspecified hazard experiment shows a source of error that cannot be fixed by counting more histories.

Specification execution and empirical measurement are separate achievements. Paper V demonstrates how to execute declared Health calculations and audit their limitations. It does not assert that those conditional classifications establish the actual ecological Health of Harvard Forest.