Please leave me comment below if you have any questions, if you notice any errors or if you would like to discuss a topic further.

Questions and Comments

Review by John Mulcahy-Stanislawczyk:

This slecture looks at ROC curves and the Neyman Pearson Criterion. I really liked the introduction that uses confusion matrices to explain ROC curves. This was different than I how I was initially introduced to ROC curves, which was more along the statistical method in the later section. I thought that the text was very clear and informative. The plots in particular were very useful for understanding what was going on. In particular I liked how two plots were shown for different values of gamma in the statistics section. Doing this makes it kind animate in the mind of the reader, so that its clear that the curve is drawn by sliding the threshold around.

A few nitpicks:

1. I guess it is true to say that an "ideal" ROC curve could be given by one where the total area under the curve is 1, but I don't think this is what people usually refer to when they are talking about ideal ROC curves. In general that curve is never attainable except for in trivial cases. Instead, I think plots like the second one showing the ROC curve for the two Gaussian case is referred to as the ideal for those class distributions. Because the Bayesian decision rule produces optimal error rate, any real classifier's ROC is bounded above by it. If I recall, this is because the Bayesian decision rule achieves the Cramer-Rao bound.

2. I think it would be good to mention that shifting gamma around is equivalent to trying different priors for the classes and/or different cost functions for misclassification. This would make the purpose of these curves more clear, too.

3. One good bit of terminology to introduce might be type I and type II error. It is basically covered here, but it might be good to explicitly say, since that language comes up in the literature often.