Appendix C — Comparing CV AUC with pROC::roc

A common question we get with regard to area under the curve (AUC) is why the AUC values from cv.glmnet do not match those from the roc function from the pROC package, or AUC functions from other packages. We illustrate this supposed mismatch with the following dataset:

data(BinomialExample)
x <- BinomialExample$x
y <- BinomialExample$y

We perform 4-fold cross-validation (CV) with the observations in each fold being specified, and pull out the estimated AUC value for the first \(\lambda\) value in the lambda path:

foldid <- rep(1:4, each = 25)
cvfit <- cv.glmnet(x, y, family = "binomial", foldid = foldid,
                   type.measure = "auc", keep = TRUE)
cvfit$cvm[1]
#> [1] 0.7423718

However, if we pull out the pre-validated fits (that is, the out-of-fold predictions) associated with the first \(\lambda\) value and use the roc function from the pROC package, we get a different answer:

library(pROC)
y.preval <- cvfit$fit.preval[, 1]
roc(y, y.preval)$auc
#> Area under the curve: 0.429

Why is there a mismatch (0.7423718 vs. 0.4289773), and which value should we use?

C.1 Explaining the mismatch

The two methods are computing two different things.

  • In cv.glmnet, for each fold \(k\), the function is trying to compute an estimate of the AUC, \(\widehat{AUC}_k\), based on the ROC curve for just the out-of-fold samples. The cvm element of the returned object reports the mean of the \(K\) AUC estimates (\(K\) being the number of folds).
  • In contrast, the second approach takes the out-of-fold predictions across all the folds, estimates a single ROC curve, then computes a single AUC estimate based on that.

If we think of AUC as a function of a vector of true responses and predictions, auc(y, yhat), in general mean(auc(y1, yhat1), ..., auc(yK, yhatK)) is not equal to auc(c(y1, ..., yK), c(yhat1, ... yhatK)) .

The code below confirms cv.glmnet’s behavior:

# computing AUC estimates explicitly for each fold, then taking mean
mean(c(
  roc(y[foldid == 1], y.preval[foldid == 1])$auc,
  roc(y[foldid == 2], y.preval[foldid == 2])$auc,
  roc(y[foldid == 3], y.preval[foldid == 3])$auc,
  roc(y[foldid == 4], y.preval[foldid == 4])$auc
))
#> Setting levels: control = 0, case = 1
#> Setting direction: controls < cases
#> Setting levels: control = 0, case = 1
#> Setting direction: controls < cases
#> Setting levels: control = 0, case = 1
#> Setting direction: controls < cases
#> Setting levels: control = 0, case = 1
#> Setting direction: controls < cases
#> [1] 0.7423718

# cv.glmnet's estimate
cvfit$cvm[1]
#> [1] 0.7423718

C.2 Which value should we use?

We recommend using the values from the cvm element in cv.glmnet’s returned object as an estimate of the CV AUC. This is because it is averaging \(K\) unbiased estimates of CV AUC, and hence will give an unbiased estimate of CV AUC. It is unclear what the AUC from the ROC curve with all pre-validated fits is estimating.