Skip to contents

Why This Matters

When covariates are regressed out on the full dataset before cross-validation, held-out test information leaks into the training pipeline. This can inflate or distort out-of-sample performance.

In cpmr, leakage-safe behavior means:

  • fit covariate regression on each fold’s training split only;
  • apply the learned regression parameters to both train and test data inside the same fold;
  • keep other parameter-learning steps (for example scaling or model selection) inside CV as well.

Minimal Example with Synthetic Data

set.seed(123)
n <- 80
p <- 120

# Synthetic connectivity matrix (rows = subjects, cols = edges)
conmat <- matrix(rnorm(n * p), nrow = n, ncol = p)

# A synthetic covariate (for example age-like nuisance variable)
covariates <- matrix(rnorm(n), ncol = 1)

# Build behavior with both signal and covariate effects
edge_signal <- rowMeans(conmat[, 1:10, drop = FALSE])
behav <- 0.7 * edge_signal + 0.6 * covariates[, 1] + rnorm(n, sd = 0.5)

# Leakage-safe CPM call: covariates are handled fold-wise inside assessment
res <- cpm(
  conmat = conmat,
  behav = behav,
  spec = spec(),
  covariates = covariates,
  resamples = 5
)

summary(res)
#> CPM summary:
#>   Number of folds: 5
#>   Pooled correlations (Pearson):
#>     Joint: 0.241
#>     Positive: 0.261
#>     Negative: -0.104
#>   Fold-wise correlations (Pearson):
#>     Joint: 0.265 (SE 0.136) (5/5 folds)
#>     Positive: 0.291 (SE 0.134) (5/5 folds)
#>     Negative: -0.270 (1/5 folds)
#>   Pooled RMSE:
#>     Joint: 0.518
#>     Positive: 0.513
#>     Negative: 0.529
#>   Pooled MAE:
#>     Joint: 0.407
#>     Positive: 0.411
#>     Negative: 0.387
#>   Selected edges:
#>     Positive: 1.67%
#>     Negative: 0.17%

Native API

model_spec <- spec()

res_with_edges <- cpm(
  conmat = conmat,
  behav = behav,
  spec = model_spec,
  covariates = covariates,
  resamples = 5,
  return_edges = "sum"
)

summary(res_with_edges)
#> CPM summary:
#>   Number of folds: 5
#>   Pooled correlations (Pearson):
#>     Joint: 0.255
#>     Positive: 0.273
#>     Negative: -0.080
#>   Fold-wise correlations (Pearson):
#>     Joint: 0.215 (SE 0.122) (5/5 folds)
#>     Positive: 0.237 (SE 0.121) (5/5 folds)
#>     Negative: -0.115 (SE 0.080) (2/5 folds)
#>   Pooled RMSE:
#>     Joint: 0.513
#>     Positive: 0.510
#>     Negative: 0.539
#>   Pooled MAE:
#>     Joint: 0.403
#>     Positive: 0.404
#>     Negative: 0.393
#>   Selected edges:
#>     Positive: 1.50%
#>     Negative: 0.33%
head(res_with_edges$predictions)
#>   row fold   observed       joint    positive      negative
#> 1   1    4 -0.5862850 -0.13713172 -0.13713172  8.673617e-18
#> 2   2    4  0.7262082 -0.06258281 -0.06258281  8.673617e-18
#> 3   3    2  0.4776754  0.10974448  0.07822397  6.622293e-02
#> 4   4    3  0.4263198  0.13974339  0.13974339 -1.561251e-17
#> 5   5    5 -1.5329525 -0.07914174 -0.07914174  1.908196e-17
#> 6   6    3  0.4996146  0.57309166  0.57309166 -1.561251e-17
dim(res_with_edges$edges)
#> [1] 120   2

Using a reusable spec() object is helpful when you want the same CPM settings to drive multiple analyses under different assessment plans.

Edge Storage Tips

For large feature spaces, fold-wise edge storage can grow quickly.

  • Use return_edges = "sum" to keep only fold-aggregated edge counts.
  • Use return_edges = "none" if you only need predictive performance.

Anti-Leakage Checklist

Use this checklist when building CPM workflows:

  • Split train/test folds before any parameter-learning step.
  • Fit covariate regression on the training split only.
  • Apply training-fitted covariate regression to the fold’s test split.
  • Keep scaling and feature selection inside each fold.
  • If comparing multiple settings, use nested CV for model selection.
  • Report the CV design and covariate strategy explicitly.

References