Cross-Validation Methods

doi:10.1006/jmps.1999.1279

Journal of Mathematical Psychology

Volume 44, Issue 1, March 2000, Pages 108-132

https://doi.org/10.1006/jmps.1999.1279 Get rights and content

Abstract

This paper gives a review of cross-validation methods. The original applications in multiple linear regression are considered first. It is shown how predictive accuracy depends on sample size and the number of predictor variables. Both two-sample and single-sample cross-validation indices are investigated. The application of cross-validation methods to the analysis of moment structures is then justified. An equivalence of a single-sample cross-validation index and the Akaike information criterion is pointed out. It is seen that the optimal number of parameters suggested by both single-sample and two-sample cross-validation indices will depend on sample size.

References (25)

H Bozdogan
Akaike's information criterion and recent developments in informational complexity
Journal of Mathematical Psychology
(2000)
W Zucchini
An introduction to model selection
Journal of Mathematical Psychology
(2000)
H Akaike
Information theory and an extension of the maximum likelihood principle
M.W Browne
The decomposition of multitrait-multimethod matrices
British Journal of Mathematical and Statistical Psychology
(1984)
M.W Browne
Predictive validity of a linear regression equation
British Journal of Mathematical and Statistical Psychology
(1975)
M.W Browne
A comparison of single sample and cross-validation methods for estimating the mean squared error of prediction in multiple linear regression
British Journal of Mathematical and Statistical Psychology
(1975)
M.W Browne et al.
Specification and estimation of mean- and covariance structure models
M.W Browne et al.
Single sample cross-validation indices for covariance structures
Multivariate Behavioral Research
(1989)
M.W Browne et al.
Alternative ways of assessing model fit
G.R Burket
A study of reduced rank models for multiple prediction
(1963)

R Cudeck et al.

Cross-validation of covariance structures

Multivariate Behavioral Research

(1983)

R Cudeck et al.

Model selection in covariance structures analysis and the “problem” of sample size: A clarification

Psychological Bulletin

(1991)

Cited by (872)

Performance improvement of atherosclerosis risk assessment based on feature interaction
2024, Computer Methods and Programs in Biomedicine
Cardiovascular disease is a leading cause of mortality and premature death. Early intervention in asymptomatic individuals through risk assessment can reduce the incidence of disease. Atherosclerosis is a major cause of cardiovascular disease and early detection can effectively prevent and treat it. In this study, we used real patient data to evaluate the risk of atherosclerosis, assisting doctors in diagnosis and reducing the incidence of cardiovascular disease.
We proposed a multi-stage atherosclerosis risk assessment model that includes three main stages: (i) SMOTE and decorrelation weighting algorithm technology were added to the causal stability middle layer to address class imbalance in the dataset and reduce the impact of feature-induced dataset distribution shifts on model differences. (ii) The feature interaction layer considered possible feature interactions and classified features by different categories. By adding more effective feature information, the accuracy and generalizability of the model were improved. (iii) In the integrated model layer, we chose LightGBM as the decision tree integration model for risk assessment because it has higher accuracy and robustness compared to other machine learning algorithms.
The final model used a dataset containing 21 original features and 17 interaction features, achieving excellent performance under a 10-fold cross-validation strategy. The macro accuracy reached 93.86%, macro precision was 94.82%, macro recall was 93.52%, and macro F1 score was as high as 93.37%. These indicators demonstrate the accuracy and robustness of the model in atherosclerosis risk assessment.
The model provides strong support for the prevention and diagnosis of cardiovascular disease. Through atherosclerosis risk assessment, the model can help doctors develop personalized prevention and treatment plans, which is of great significance for the prevention and treatment of cardiovascular disease.
Bayesian NMR petrophysical characterization
2024, Journal of Magnetic Resonance
Identification of reservoir rock types is necessary for the exploration and recovery of oil and gas. It involves determining the petrophysical properties of rocks such as porosity and permeability which play a significant role in developing reservoir models, estimating the volumes of oil and gas reserves, and planning production methods. Nuclear magnetic resonance (NMR) technology is a fast and accurate tool for petrophysical rock characterization. The distributions of relaxation times ( $T_{2}$ distributions) offer valuable insights into the distribution of pore sizes in rocks, and these distributions are closely linked to important petrophysical parameters like porosity, permeability, and bound fluid volume (BFV).
This work introduces a Bayesian estimation method for analyzing NMR data. The Bayesian approach uses prior knowledge of $T_{2}$ distributions in the form of the prior mean and covariance. The Bayesian approach combines prior knowledge with observed data to obtain improved estimation. We use the Bayesian estimation method where prior information regarding the rock sample type, for example shale, is available.
The estimators were evaluated on decay data simulated from synthesized distributions that replicate the features of experimental $T_{2}$ distributions of three types of reservoir rocks. We compared the performance of the Bayesian method with two existing methods using porosity, bound fluid volume (BFV) geometric mean (T2LM) and root mean square error (RMSE) of the estimated $T_{2}$ distribution as evaluation criteria. Additional experiments were carried out using experimental $T_{2}$ distributions to validate the results.
The performance of the Bayesian methods was also tested using mismatched priors.
The experimental results illustrate that the Bayesian estimator outperforms other estimators in estimating the $T_{2}$ distribution. The Bayesian method also outperforms the ILT method in estimating derived petrophysical properties except in cases where the noise level is below 0.1 and the $T_{2}$ distributions are associated with short relaxation times.
Computer-assisted design of CGC catalysts for ethylene/1-octene copolymerization: A combined DFT and artificial neural network approach
2024, Polymer
High temperature solution polymerization of ethylene/1-octene is the main means to produce polyolefin elastomer (POE), one of the most important polyolefin materials, and the metallocene catalysts with high-efficiency are the key to achieve POE production. However, there remains a deficiency in a universal approach for creating catalysts for ethylene/1-octene copolymerization. In this document, we outlined a method for designing catalysts for ethylene/1-octene copolymerization by combining density functional theory (DFT) with an artificial neural network (ANN) was proposed. By optimizing the structure and computing electronically a sequence of constrained geometry catalysts (CGC) by DFT, descriptors related to steric and electronic descriptors were gathered to create the dataset. After ANN training, an ANN-based model using for predicting catalyst activity with high R² value was optimized, and a class of novel highly efficient CGC have been successfully designed. Furthermore, an exploration was conducted into how the structure of catalysts correlates with their performance. The electronic properties of the catalysts were found to have more significantly effect on the catalytic performance than the steric properties. In addition, the effect of the β-site substitution of the cyclopentadienyl group on the catalytic activity of predicted CGC was studied and discussed. The groups with ring-structure, especially with conjugate system, were proved to possess a good promoting effect on the catalytic activity.
Bootstrap sampling style ensemble neural network for inverse design of optical nanoantennas
2024, Optics Communications
The process of machine learning-assisted nanophotonicsinverse design has been plagued by the problem of non-uniqueness for a long time, which is a problem worth studying. In this paper, we present a novel methodology for the design of bowtie optical nanoantennas (BONAs) by employing a Bootstrap Sampling Style Ensemble Neural Network (BSENN) model. Our approach combines a bagging algorithm with a tandem neural network to address the non-uniqueness challenge inherent in the inverse design process of BONAs. By splitting the data, training in batches, and integrating the results, our BSENN model is able to provide reliable predictions and offer a solution to the non-uniqueness problem. The main objective of our work is to explore diverse BONAs design structures that yield identical spectral responses, thereby providing a broader range of alternatives for the design of optical nanoantennas. Through the utilization of the BSENN model, we aim to enhance the design process and offer increased flexibility and versatility in the field of optical nanoantenna design.
Spectroscopic food adulteration detection using machine learning: Current challenges and future prospects
2024, Trends in Food Science and Technology
Food adulteration has emerged as a significant challenge in the food industry, impacting consumer health and trust in the market. Utilizing machine learning especially deep learning with spectroscopic methods has revolutionized food adulteration detection enabling the development of more sophisticated and automated solutions.
This review aims to provide a comprehensive overview of the challenges and opportunities in machine learning-based spectroscopic techniques for detecting food adulteration by exploring various spectroscopic techniques commonly employed in the food industry, such as infrared spectroscopy, Raman spectroscopy, NMR spectroscopy, fluorescence spectroscopy, multi-spectral imaging, and hyperspectral imaging. The article addresses data pre-processing, feature engineering, model complexity, interpretability and their performance, and the need for large-scale diverse datasets.
To develop a commercial spectroscopic adulteration detection system that uses machine learning, one needs to optimize not only the model, but also the dataset size, the combination of pre-processing methods, the feature selection and extraction methods, the model selection, the hyperparameters by validation and the performance criteria. In addition, new machine learning algorithms are growing rapidly but creating a specialized model for adulteration detection using spectroscopy is still an area of research.
Machine learning aided uncertainty quantification for engineering structures involving material-geometric randomness and data imperfection
2024, Computer Methods in Applied Mechanics and Engineering
In real-world engineering, uncertainty is ubiquitous within material properties, structural geometry, load conditions, and the like. These uncertainties have substantial impacts on the estimation of structural performance. Furthermore, information or datasets in real life commonly contain imperfections, e.g., noise, outliers, or missing data. To quantify these impacts induced by uncertainties on structural behaviours and reduce the effects of data imperfections simultaneously, a machine learning-aided stochastic analysis framework is proposed. A novel supervised machine learning technique, namely the Capped Extended Support Vector Regression (CX-SVR) technique, is developed to effectively suppress the effects of outliers and noise in datasets. Its inherent convexity in optimization and capped strategy theoretically supports the accuracy of CX-SVR, especially in handling datasets with imperfections. Once the effective surrogate model is established, subsequent analyses, like sampling-based methods, can circumvent the cumbersome physical model, which is potentially the nest of computational burden and errors in engineering applications. The high robustness of the proposed approach can be summarized in four main aspects: unrestrictive selection of the system inputs and their statistical information, ‘perfect’ or ‘imperfect’ data, enough statistical information (including statistical moments, probability density functions, and cumulative distribution functions) of the system outputs, and physical problems from various engineering fields.

View all citing articles on Scopus

View full text

Regular ArticleCross-Validation Methods

Abstract

Journal of Mathematical Psychology

Journal of Mathematical Psychology

Information theory and an extension of the maximum likelihood principle

The decomposition of multitrait-multimethod matrices

British Journal of Mathematical and Statistical Psychology

Predictive validity of a linear regression equation

British Journal of Mathematical and Statistical Psychology

A comparison of single sample and cross-validation methods for estimating the mean squared error of prediction in multiple linear regression

British Journal of Mathematical and Statistical Psychology

Specification and estimation of mean- and covariance structure models

Single sample cross-validation indices for covariance structures

Multivariate Behavioral Research

Alternative ways of assessing model fit

A study of reduced rank models for multiple prediction

Cross-validation of covariance structures

Multivariate Behavioral Research

Model selection in covariance structures analysis and the “problem” of sample size: A clarification

Psychological Bulletin

Regular Article
Cross-Validation Methods