Model Weights and Model Selection Uncertainty

Remember these Results from the Harlequin Duck analysis.....

As with the Dipper data, you see these AIC weights listed alongside those AIC values in the MARK output. An AIC weight, w, for model i, is  calculated as ....

across the set of R models.  Remember that exp(stuff) means that number e (which = 2.71828...) taken to the power (exponent) of stuff.  This is very analogous to weighted means or weighted variances that you probably have seen before.  We put the most weight on the model with which we have the most confidence.  The sum of all weights = 1.  

Up til now in this class and in your prior use of models, you have probably spoken of specific parameter estimates from a particular model.  For example, survival rates of dippers in 1984, a flood year, was x based on the best model.  In other words, the estimate of 1984 survival rate of dippers is conditional on that best model - the estimate comes from that model and presumes it is the correct model.  If you looked at the parameter estimate for 1984 from the phi(.),p(.) model will note that it is different; that estimate is the survival rate of dippers conditional on the phi(.),p(.) model.  So, clearly, our perspective on what survival is for a group at a particular point in time is dependent on how certain we are that we can identify the truly best model for these data.  When AIC values are similar for a set of models, then we are uncertain which model really is the best.  Given only 2 models in an analysis, one with a weight of 0.55 and the other with a weight of 0.45, then we would interpret that to mean that if this same study was replicated 100 times, 55 of those replicate studies and analyses would choose the 1st model as the best model and 45 would choose the 2nd model as the best model.  It is therefore desirable to have estimates that reflect this uncertainty about which model best fits these data; in other words, we want unconditional estimates - estimates not dependent on a single model.  

Unconditional estimates and variances

  This provides the unconditional estimates, simply a weighted estimate.

Note that the variance of this estimate contains 2 components to it.  The first part within the square root notation is the variance associated with each model (i.e., the variance of estimate i, given Model i).  This reflects the sampling variance of estimating that parameter for that model.  Sampling variance will always contribute to your overall estimate of the variation associated with a parameter, whether or not you speak of conditional or unconditional estimates.  The second term reflects how much the parameter estimate from each model differs from the mean weighted estimate.  

Below is the output from selecting the Model Averaging option for the Harlequin Duck data. Remember that the full design matrix for these data estimated 52 parameters and thus Parameter 1 is survival during week 1 of the Oiled ducks in Early Winter.  Parameter 10 is survival during week 1 of the Oiled ducks in Mid Winter.  For each of the 8 candidate models, MARK produced an estimate for Parameter 1 and each of those estimates is shown below along with each estimate's standard error; these standard errors reflect the sampling variance associated with each model.  The first summary row shows the weighted parameter estimate and a standard error that is the weighted average of all the model specific standard errors.  This weighted standard error is thus still a reflection just of sampling variance.  If all the 8 estimates for the 8 models happened to be identical, then there would be no additional variance in our estimate due to model selection uncertainty, in which case the unconditional SE would be identical to the weighted average SE.  However, this virtually never happens, so there is some more variance to be accounted for (the second term in the equation above). Then, 1 minus the ratio of the weighted and unconditional variances provides the percent of overall estimate variation that is attributable to the the fact that different models gave different estimates for the same parameter. 

Hypothesis Testing/Weight of Evidence:

A second use of the AIC weights is as a "weight of evidence" approach, which is in the spirit of hypothesis testing.  Note from the Results at the top of this page that the best model had an AIC weight of just 0.44.  If Dan wants to address the hypothesis of whether oil (i.e., area) affected survival, touting just this best model and proclaiming an oil affect (as this model includes area as an effect) isn't very satisfying since other models also fit the data fairly well.  What then matters is whether these other models also include area as an effect.  The sum of AIC weights for models with area as an effect is 0.93, thus given these 8 models, there is very strong evidence for an area (oil) effect.