AI-based feature selection for craniometric sex classification: A nested cross-validation study on the Howells dataset

Abstract

Sex estimation is an important part of forensic biological-profile reconstruction. However, machine-learning results can be overly optimistic when feature selection and hyperparameter tuning are performed without separating them from model evaluation. This study compared five feature-selection methods, namely Information Gain, ReliefF, LASSO, SHAP-based ranking, and support vector machine recursive feature elimination (SVM-RFE), using five classifiers: support vector machine (SVM), random forest, logistic regression, decision tree, and multilayer perceptron. The study used 2,524 individuals from a Howells-derived craniometric dataset, consisting of 1,156 females and 1,368 males. A stratified hold-out set containing 20% of the data was reserved before model development. After missingness filtering based on the development data, 71 predictors remained. Model development used 5-fold outer and 5-fold inner cross-validation, with 3, 5, 7, 10, and 15 candidate features. SVM-RFE combined with SVM obtained a nested-cross-validation accuracy of 0.8767 ± 0.0098, with an F1-score of 0.8852 and ROC-AUC of 0.9481. After retuning using the full development set, the final 15-feature model achieved 0.8970 accuracy, 0.9055 F1-score, 0.9511 ROC-AUC, 0.9088 sensitivity, and 0.8831 specificity on 505 internal hold-out observations. The AI-oriented selectors had a mean accuracy only 0.0074 higher than the filter selectors, which was below the predefined 0.05 improvement threshold. Classifier choice had a significant effect on accuracy, while the feature-selection method and its interaction with the classifier were not significant. SHAP analysis identified ZYB, AUB, and MDH as the three highest-ranked features. The results show that a relatively small number of craniometric features can provide good classification performance under the evaluation procedure used in this study. However, the differences between feature-selection approaches were relatively small, and further validation on independent populations and data collected at different time points is needed before the model can be considered for forensic use.