Comprehensive validation of AI models and diagnostic tests using cross-validation, model selection, and advanced performance metrics. Designed for AI diagnostic research including comparison of AI vs human performance with statistical significance testing.
This function addresses the complete workflow needed for AI diagnostic research:
Cross-validated performance assessment
Automated model selection with AIC/BIC
Statistical comparison between AI and human performance
Advanced metrics (NRI, IDI, calibration)
Publication-ready visualizations
Details
Cross-Validation Methods:
k-fold cross-validation (5-fold, 10-fold)
Leave-one-out cross-validation
Repeated cross-validation with multiple iterations
Stratified sampling to maintain outcome proportions
Model Selection:
AIC/BIC-based stepwise selection
Forward, backward, or bidirectional selection
LASSO and Ridge regularization
Variable importance ranking
Statistical Tests:
DeLong test for AUC comparison
McNemar's test for paired predictions
Hosmer-Lemeshow calibration test
Bootstrap confidence intervals
Super classes
jmvcore::Analysis -> ClinicoPath::aivalidationBase -> aivalidationClass
Methods
Inherited methods
jmvcore::Analysis$.createImage()jmvcore::Analysis$.createImages()jmvcore::Analysis$.createPlotObject()jmvcore::Analysis$.load()jmvcore::Analysis$.render()jmvcore::Analysis$.save()jmvcore::Analysis$.savePart()jmvcore::Analysis$.setCheckpoint()jmvcore::Analysis$.setParent()jmvcore::Analysis$.setReadDatasetHeaderSource()jmvcore::Analysis$.setReadDatasetSource()jmvcore::Analysis$.setResourcesPathSource()jmvcore::Analysis$.setStatePathSource()jmvcore::Analysis$addAddon()jmvcore::Analysis$asProtoBuf()jmvcore::Analysis$asSource()jmvcore::Analysis$check()jmvcore::Analysis$init()jmvcore::Analysis$optionsChangedHandler()jmvcore::Analysis$postInit()jmvcore::Analysis$print()jmvcore::Analysis$readDataset()jmvcore::Analysis$run()jmvcore::Analysis$serialize()jmvcore::Analysis$setError()jmvcore::Analysis$setStatus()jmvcore::Analysis$translate()ClinicoPath::aivalidationBase$initialize()