1.17 Methods of permutations

As for the one-way case, the distribution of each of the pseudo-F ratios in a multi-way design is generally unknown. Thus, a permutation test (or some other approach using re-sampling methods) is desirable. When there is more than one factor, situations commonly arise which prevent the possibility of obtaining an exact test of individual terms in the model using permutations. For example, there is no exact permutation test for an interaction (but see Pesarin (2001) , who describes a synchronised permutation method for testing interactions). In addition, restricted permutation methods for testing main effects in ANOVA models generally have low power ( Anderson & ter Braak (2003) ). However, several good approximate permutation methods can be used instead to get accurate P-values (e.g., Anderson & Legendre (1999) ). PERMANOVA provides three general options regarding the method of permutation to be used: (i) unrestricted permutation of raw data, (ii) permutation of residuals under a reduced model, or (iii) permutation of residuals under the full model. These methods and their properties are described in detail elsewhere (e.g., Anderson & Legendre (1999) , Anderson (2001b) , Anderson & Robinson (2001) , Anderson & ter Braak (2003) , Manly (2006) ). Although these methods do not give exact P-values for complex designs in all cases, they are asymptotically exact26 and give very reliable results. In practice, these three approaches will give very similar results, so there is (thankfully) no need to agonise much about making a choice here. All three of the methods are implemented in PERMANOVA so as to ensure that the correct exchangeable units (identified by the denominator of the pseudo-F ratio) are used for each individual test (see Anderson & ter Braak (2003) for details). Some of the known properties of the three methods are outlined below.

In general, we recommend using method (ii), which is the default. Method (i), however, does provide an exact test for the one-way case, so should be used for one-way ANOVA models. Otherwise, note that methods (ii) and (iii) both require estimation of parameters (means) in order to calculate residuals as deviations from fitted values. When sample sizes are small, these estimates are not very precise (i.e., they may not be very close to their “true” values), so the residuals being permuted, in turn, may not be good representatives of the “true” errors ( Anderson & Robinson (2001) ). Thus, in the case of relatively small sample sizes (say, n < 4 replicates per cell), method (i) is also recommended (provided there are no outliers in covariables, as mentioned above). Method (iii) is probably only advisable if you wish to use method (ii), but the time required is getting overly burdensome.


26 An asymptotically exact test is a test for which the type I error (probability of rejecting the null hypothesis when it is true) asymptotically approaches (converges on) the a priori chosen significance level ($\alpha$) with increases in the sample size (N).

27 Note that “less powerful” does not necessarily mean that using (i) will give you a smaller P-value than (ii) or (iii) for any particular data set. It means that, in repeated simulations, the empirical power (estimated probability of rejecting the null hypothesis when it is false) was, on average, smaller for method (i) than for either of the other two methods in most situations.

28 For unbalanced designs, it also depends on which Type of SS is chosen for the test. For Type I SS, the order in which the terms are fitted will also matter here.


Revision #11
Created 7 August 2022 10:43:52 by Arden
Updated 26 November 2024 23:30:56 by Abby Miller