Outcome Definition Before Model Choice in Credit Risk

Machine-Learning Models versus Ratio-Based Scorecards

Abstract

This paper examines how outcome definition affects the evaluation of traditional and machine-learning credit-risk models. Using U.S. Small Business Administration lending data, consumer-credit benchmarks, and controlled simulations, the study compares ratio-based credit scorecards with machine-learning models across discrimination, calibration, credit-allocation efficiency, and robustness to changes in the lending environment. The results show that machine-learning models can improve predictive performance over traditional scorecards, particularly in precision-recall performance and calibration. However, the analysis also demonstrates that model choice can matter less than how the credit outcome itself is defined. Restricting observations based on loan status or maturity can substantially alter measured default rates and produce apparently strong predictive performance while misrepresenting underlying credit risk. Additional analyses examine model stability across prediction horizons, calibration under distribution shift, and the relationship between predictive complexity and lending decisions. The findings suggest that financial institutions should define credit outcomes and observation windows before selecting predictive models, evaluate calibration alongside ranking metrics, and consider simpler models when their transparency and stability outweigh modest gains in predictive accuracy. Overall, the study shows that responsible adoption of AI in lending depends not only on choosing more powerful algorithms, but also on constructing the credit-risk problem correctly.