Seeming inconsistencies can occur when comparing the overall test of a CLASS effect in a model with a set of multiple comparison tests among the levels of the effect.
Overall tests use different tests and answer different questions than the pairwise comparisons tests. A nonsignificant overall test of a CLASS effect does not necessarily imply that none of the multiple pairwise comparison tests among its levels will be significant. Similarly, a significant overall test does not imply that one or more of the multiple pairwise comparison tests will be significant.
Possible reasons for these inconsistencies are:
- Insufficient sample size to detect a difference
- Unbalanced data
- The presence of covariates in the model. Note that by default, the LSMEANS are measured at the mean value of the covariate. Differences in LSMEANS may be detected at values of the covariate other than their mean value.
- When multiple comparison tests are done at a given error level (such as alpha=0.05), the actual level over the set of tests (called the family-wise error rate, FWE) is much larger. This is the multiple testing problem. The procedure of first using an overall test before doing multiple comparison tests, each at the same error level as the overall test, as a way to "protect" against making errors is known as Fisher's protected least significant difference (LSD). But as discussed in Westfall et al. (2011), which is referenced in SAS Note 22950, it only provides partial protection of the FWE and therefore is not a recommended procedure. In order to maintain the desired error level, the p-values of the multiple comparison tests must be adjusted using an appropriate multiple testing procedure (MCT). Many MCT methods are available with the ADJUST= option in the LSMEANS statement and in the MULTTEST procedure. The Holm method in MULTTEST is a general method that can be applied to multiple pairwise comparisons. No MCT can guarantee consistency with a particular overall test, but an overall test is not needed since the MCT alone controls the FWE. See the example titled "Inputting raw p-values" in the MULTTEST documentation showing how a set of p-values can be provided to the procedure for adjustment using one or more MCT methods.