Testing whether an ML model can predict the next chemistry round

A property model produces an attractive validation plot, then misses the compounds synthesized the following month. One explanation is that the validation split was easier than the project task. Close analogs, duplicate measurements, or data generated after the prediction date may have crossed between training and testing.

Representation improved faster than some evaluation habits

Quantitative structure-activity models originally relied heavily on selected descriptors and statistical relationships. Molecular fingerprints and nonlinear methods broadened the patterns that could be learned. Graph neural networks introduced learned representations built from molecular connectivity. Yang and colleagues evaluated learned molecular representations across property-prediction tasks, including comparisons of data-splitting strategies ^[1].

The choice of representation matters, but a sophisticated model cannot repair an ambiguous endpoint. Solubility measured at different pH values, clearance measured in different species, or potency measured with different assay formats should not be treated as interchangeable labels. Retain protocol metadata and define the intended prediction precisely.

Reconstruct the decision date

For a model intended to guide the next synthesis round, train on data available before a defined date and test on later measurements. Sheridan's time-split work examined this approach as a way to estimate prospective prediction quality ^[2]. It asks a question closer to the project's actual use than a random partition of the final dataset.

Keep repeat measurements and duplicate chemical identities from creating leakage. Fit preprocessing steps using training data only, and reserve a validation set for selecting hyperparameters. The final test set should remain untouched until those decisions are fixed. If the project expects new scaffolds, add a scaffold-based evaluation, recognizing that scaffold definitions themselves affect the difficulty of the split.

Measure the decision, as well as the error

A global error statistic can hide poor performance in the compounds that matter most. For potency ranking, examine the top selections and the ability to resolve meaningful differences. For a liability filter, inspect false negatives near the decision threshold. Compare against a simple baseline such as nearest-neighbor estimates or an established fingerprint model using the same data and split.

Report performance by chemical series and by distance from training chemistry. Prediction intervals need calibration against held-out outcomes. Agreement among several models can be reassuring, but it can also reflect shared training data and shared blind spots. An unfamiliar scaffold should be identifiable in the output rather than buried inside a precise-looking point estimate.

Spend experiments on uncertainty deliberately

An active-learning round can include promising compounds and selected compounds that test uncertain regions. Keep the purpose of each selection visible. If every compound is chosen for maximum predicted potency, the data may reveal little about the model's errors elsewhere. If every compound is chosen for uncertainty, the round may fail to advance the chemistry.

Freeze predictions before the assay results arrive and retain the full selection record, including compounds rejected after synthetic review. Evaluate the next round without silently retraining on its answers. When the model fails, examine whether the cause is assay noise, a new mechanism, chemical novelty, or a change in experimental conditions. Those distinctions determine whether the remedy is better data, another representation, or a narrower claim about what the model can predict.