Leakage-Safe Covariate Handling in CPM
Source:vignettes/anti-leakage-covariates.Rmd
anti-leakage-covariates.RmdWhy This Matters
When covariates are regressed out on the full dataset before cross-validation, held-out test information leaks into the training pipeline. This can inflate or distort out-of-sample performance.
In cpmr, leakage-safe behavior means:
- fit covariate regression on each fold’s training split only;
- apply the learned regression parameters to both train and test data inside the same fold;
- keep other parameter-learning steps (for example scaling or model selection) inside CV as well.
Minimal Example with Synthetic Data
set.seed(123)
n <- 80
p <- 120
# Synthetic connectivity matrix (rows = subjects, cols = edges)
conmat <- matrix(rnorm(n * p), nrow = n, ncol = p)
# A synthetic covariate (for example age-like nuisance variable)
covariates <- matrix(rnorm(n), ncol = 1)
# Build behavior with both signal and covariate effects
edge_signal <- rowMeans(conmat[, 1:10, drop = FALSE])
behav <- 0.7 * edge_signal + 0.6 * covariates[, 1] + rnorm(n, sd = 0.5)
# Leakage-safe CPM call: covariates are handled fold-wise inside assessment
res <- cpm(
conmat = conmat,
behav = behav,
spec = spec(),
covariates = covariates,
resamples = 5
)
summary(res)
#> CPM summary:
#> Number of folds: 5
#> Pooled correlations (Pearson):
#> Joint: 0.241
#> Positive: 0.261
#> Negative: -0.104
#> Fold-wise correlations (Pearson):
#> Joint: 0.265 (SE 0.136) (5/5 folds)
#> Positive: 0.291 (SE 0.134) (5/5 folds)
#> Negative: -0.270 (1/5 folds)
#> Pooled RMSE:
#> Joint: 0.518
#> Positive: 0.513
#> Negative: 0.529
#> Pooled MAE:
#> Joint: 0.407
#> Positive: 0.411
#> Negative: 0.387
#> Selected edges:
#> Positive: 1.67%
#> Negative: 0.17%Native API
model_spec <- spec()
res_with_edges <- cpm(
conmat = conmat,
behav = behav,
spec = model_spec,
covariates = covariates,
resamples = 5,
return_edges = "sum"
)
summary(res_with_edges)
#> CPM summary:
#> Number of folds: 5
#> Pooled correlations (Pearson):
#> Joint: 0.255
#> Positive: 0.273
#> Negative: -0.080
#> Fold-wise correlations (Pearson):
#> Joint: 0.215 (SE 0.122) (5/5 folds)
#> Positive: 0.237 (SE 0.121) (5/5 folds)
#> Negative: -0.115 (SE 0.080) (2/5 folds)
#> Pooled RMSE:
#> Joint: 0.513
#> Positive: 0.510
#> Negative: 0.539
#> Pooled MAE:
#> Joint: 0.403
#> Positive: 0.404
#> Negative: 0.393
#> Selected edges:
#> Positive: 1.50%
#> Negative: 0.33%
head(res_with_edges$predictions)
#> row fold observed joint positive negative
#> 1 1 4 -0.5862850 -0.13713172 -0.13713172 8.673617e-18
#> 2 2 4 0.7262082 -0.06258281 -0.06258281 8.673617e-18
#> 3 3 2 0.4776754 0.10974448 0.07822397 6.622293e-02
#> 4 4 3 0.4263198 0.13974339 0.13974339 -1.561251e-17
#> 5 5 5 -1.5329525 -0.07914174 -0.07914174 1.908196e-17
#> 6 6 3 0.4996146 0.57309166 0.57309166 -1.561251e-17
dim(res_with_edges$edges)
#> [1] 120 2Using a reusable spec() object is helpful when you want
the same CPM settings to drive multiple analyses under different
assessment plans.
Edge Storage Tips
For large feature spaces, fold-wise edge storage can grow quickly.
- Use
return_edges = "sum"to keep only fold-aggregated edge counts. - Use
return_edges = "none"if you only need predictive performance.
Anti-Leakage Checklist
Use this checklist when building CPM workflows:
- Split train/test folds before any parameter-learning step.
- Fit covariate regression on the training split only.
- Apply training-fitted covariate regression to the fold’s test split.
- Keep scaling and feature selection inside each fold.
- If comparing multiple settings, use nested CV for model selection.
- Report the CV design and covariate strategy explicitly.
References
- Snoek L, Miletic S, Scholte HS (2019). How to control for confounds in decoding analyses of neuroimaging data. NeuroImage. https://doi.org/10.1016/j.neuroimage.2018.09.074
- Scheinost D, Noble S, Horien C, et al. (2019). Ten simple rules for predictive modeling of individual differences in neuroimaging. NeuroImage. https://doi.org/10.1016/j.neuroimage.2019.02.057
- Rosenblatt M, Tejavibulya L, Jiang R, Noble S, Scheinost D (2024). Data leakage inflates prediction performance in connectome-based machine learning models. Nature Communications. https://doi.org/10.1038/s41467-024-46150-w