Hfrxdjt

I am currently using XGBoost for risk prediction, it seems to be doing a good job in the binary classification department but the probability outputs are way off.

i.e., Changing the value of a feature in an observation by a very small amount can make the probability output jump from %50 to %99.

I barely see outputs in the 60%-80% range it's all less than %1 or 99%.

I am aware of post training calibration methods such as Platt Scaling and Logistic Correction, but I was wondering if there is anything I can tweak in the XGBoost training process.

I call XGBoost from different languages using FFI, so it would be nice if I can fix this issue without introducing other calibration libraries, e.g: changing eval metric from AUC to logloss ?

Yes, check the log-loss distribution as the number of iterations increases. If it start's shooting up before your final boosting iteration then it's over-fitting.

One of the differences of using Tree based methods vs. Regression based methods is that the output is discretised so is not continuous due to the inherent nature of the process.

Also, a benefit of boosted tree method like XGBoost is that the ensemble gives a much more continuous output compares to a single decision tree algorithm.

However, in this case it seems you've identified the boundary value whose slight change decides one of the early branches of the tree ensemble. Hence, the sudden change.

Although ensemble methods (and specifically XGBoost) often give higher accuracy, but a downside is they are not as easily interpretable as regression based methods.

You should try applying more regularization and use cross-fold validation to get a better estimate of the performance.

First, you should be sure on that your data is large enough when working with tree-based algorithms like XGBoost and LightGBM, such sudden changes may indicate overfitting. (10,000 samples at least, rule of thumb)

Second, how is your cardinality; if you have 3-4 features, it would be expected that a change of feature causing such an affect.

Third, what are your selection of hyperparameters? Tree-based models are much sensitive to changes of the parameters. Be sure that you carefully implement your hyperparameter tuning.

Lastly, when dealing with binary classification; error metrics gets really important. You can do a combination of binary log loss and binary error (XGBoost allows you to choose multiple); also be sure to implement early stopping by choosing early_stopping_rounds = N in the train method of XGBoost, where N is the selection of iterations. By that, your algorithm will stop early at a reasonable point where your loss stops to decrease, avoiding overfitting.