So way back in November, I promised the next post would elaborate a bit on model comparison. This part of the chapter, and the accompanying exercises, blew my mind a little.

Bayesian priors are part of the model

I think one of the hardest things to get a handle on, coming from “traditional” NHST stats is the concept of priors. Not just the concept, but how the heck you’re supposed to use them. Up to this point, Kruschke’s presented them in very consistent, intuitive way. Priors are a mathematical expression of your beliefs “prior” to the consideration of a particular set of data. In the ongoing example in the book, this may be your belief about how fair a coin is before you start flipping. In a more real research context, priors are often discussed (and Kruschke presents this idea, too) as characterizing “what we know already” before an experiment.

Extrapolating a bit, maybe you’re doing a lexical decision priming experiment, where reaction times are usually in the 300-500 millisecond range, then you can be pretty sure you won’t see priming effects of more than 100-200 milliseconds or so at the outer limit. Maybe this is a lame example, but pick any real experiment or data set, and unless the experimental paradigm is completely new, you probably have some information from previous experiments about what generally you might find.

But I think where non-Baysians start to get nervous is when they realize that the posteriors (which seem like “results”) depend in part on the priors. So if you can make up the priors, can’t you use that to manipulate your results? And if priors represent prior “beliefs,” isn’t it hard to argue about what someone’s beliefs are? What if the experimenter’s beliefs are different from my (the reader’s) beliefs? I think most of these worries are missing the point, but I’m still too novice to understand why, really. But Kruschke’s discussion of model comparison, and in particular the exercise 5.6, has given me a different way to think about it. Not sure if I’m right yet, but it feels right.

The way I understand it is that priors aren’t some squishy thing that are “off-limits” for discussion. They are part of the model of data being constructed, and therefore can be compared using model comparison. When I think of “model comparison” in a regression sense, what I think of is comparing different regression models (i.e., different sets of predictor variables), and checking fit statistics to see which model provides a better fit to the data. Maybe a model with an interaction effect is better than one without it, so you take that as evidence that there is a “significant” interaction. In Bayesian model comparison, this might play out as comparing different sets of priors, because it’s the priors where things are set like the range of possible (or likely) values that a parameter might take. Back to the coin-flipping example, you could compare a model in which the prior is a strong belief in a fair coin to a model in which the prior is a weak belief in fairness (i.e., a wide spread in possible values for the fairness of the coin).

To put it another way, pitting different priors against each other looks like a way to explicitly test two competing hypotheses. Maybe I’m missing something, but this is a totally different spin on priors than I had understood up to this point, and I think it makes sense. If you want to stick to Kruschke’s phrasing of “beliefs,” then what you can do is compare different sets of beliefs. There doesn’t have to be one “most appropriate” set of beliefs before you analyze your data.

In the next post (on Kruschke’s book), I’ll try to finally look at some data and go through a very simple Bayesian analysis, parallel to what Kruschke does in the book. Hopefully then some of these abstract discussion will start to make a little more sense.