I converted 126 tree models to ONNX by following the docs. Here is what changed.

I wanted to know how often the documented way of converting a tree model to ONNX produces a file that is not the same model. So I built the pairs the way the skl2onnx and onnxmltools documentation shows, with default settings and float32 input, and checked every one with an equivalence checker.

The setup: 7 datasets (iris, wine, breast cancer, digits, diabetes, a synthetic regression, and a synthetic table of integer counts), 6 model types, 3 sizes each (10 trees at depth 3, 50 at depth 6, 100 at depth 8). That is 126 pairs. Each pair was checked twice, once on finite inputs only and once on all inputs including NaN.

On real data, none of this shows. I ran every original and every ONNX file on the rows of its own dataset, 19,908 rows in total across the 126 pairs, and not one output differed. A test set check would have passed all 126.

LightGBM: one float32 value goes the wrong way

LightGBM stores split thresholds as float64. The ONNX file stores them as float32. When a threshold is a round decimal, the rounding can move it past the one input value that equals that decimal.

In the breast cancer model with 10 trees, the first split on feature 23 is x <= 868.2000000000002 in LightGBM and x <= 868.2000122070312 in the ONNX file. The input 868.2, as float32, is exactly 868.2000122070312. LightGBM sends it right (it is larger than the threshold), the ONNX file sends it left (it is equal to the threshold).

I took the input the checker reported and ran it through the real LightGBM and the real onnxruntime. LightGBM predicts class 0 (raw score -0.853). The ONNX file predicts class 1 (probability 0.786). The raw scores differ by 2.156, which is what the report said was the largest possible difference. For 9 of the 15 pairs the report says the predicted class can change.

The affected band is tiny: one float32 value per threshold. That is why 19,908 real rows never hit it. It is also why it is a real problem when your features are decimals like prices or measurements, because round decimals are exactly the values people type in.

The 6 LightGBM pairs that were fine are the digits and the integer count datasets, where every threshold is exactly representable.

scikit-learn trees and forests: NaN goes a different way

scikit-learn accepts NaN at prediction time for trees and forests, even if the model never saw a NaN in training. According to its documentation, such samples are sent to the child with the most training samples. The converted ONNX file sends NaN a fixed way instead.

Breast cancer, one decision tree, one NaN in the input: scikit-learn returns probabilities [0.0, 1.0], the ONNX file returns [1.0, 0.0]. The labels are opposite. On the wine random forest, an input with four NaN values gets class 2 from scikit-learn and class 0 from the ONNX file. For 45 of the 63 scikit-learn pairs the report says the predicted class can change.

This only matters if NaN can reach your model, for example from a failed join or a missing sensor value. If you impute upstream, it does not. But then nothing is checking that for you either.

What was fine

All 42 XGBoost and scikit-learn GradientBoosting pairs were equivalent, with and without NaN. I also tried an old XGBoost report (an issue about base_score in onnxruntime) and it no longer reproduces with current versions. Those converters did the right thing in these settings.

Limits

These are models I trained on built-in datasets following the documentation. They are not third-party production models, so this says how the documented recipes behave, not how common the problem is in the wild. The checker supports numeric XGBoost, LightGBM and scikit-learn tree models and ONNX TreeEnsemble operators up to opset 3. Categorical splits and HistGradientBoosting are not supported, so they were not part of this.

Run it yourself

The script is in the repository. A quick run takes under a minute, the full run about 12 minutes.

pip install leafparity[all] skl2onnx onnxmltools
python audit.py --quick

Script and README: https://github.com/Leafparity/leafparity/tree/main/examples/audit

I am the author of the checker, Leafparity (open source, Apache 2.0, https://leafparity.com). If you have a model and its ONNX file where a check like this finds something, or finds nothing, I would like to hear about it.