3.6 Summarising the comparison

In summary:

• I recommend using PERMANOVA in PRIMER.
• I do not recommend using adonis2 in R.
• Except in very limited circumstances, adonis2 does not construct:
  • (i) correct F-ratios; or
  • (ii) correct permutation algorithms.
• The implementation in adonis2 is far too limiting (it can only be correct for one-way cases, and possibly correct for fixed factors only in fully crossed, fully balanced designs).
• Furthermore, $R^2$ values for individual terms in ANOVA models are not a sensible way of comparing their relative importance.

It would be exhausting, fruitless and probably upsetting to list all of the papers that have used adonis2 (or adonis, its predecessor) in R to perform a PERMANOVA for a complex design that have failed to notice these important problems and limitations. No-one can really be blamed for trying to use adonis2. It seems (on the face of it) like it should work. It has been used to run all sorts of designs and gets cited rather a lot. Unfortunately, the results of analyses done using adonis2 might be wrong, and the inferences drawn misleading, depending on the model/study design.

In deference to the excellent people who wrote the adonis2 routine (it's clearly a good thing that they created it), I feel certain that they (probably) never intended for this function to be used to analyse complex experimental designs with random factors, nested factors, etc. It would be helpful for the truly limited scope of adonis2 to be more plainly acknowledged somewhere in the documentation and/or description of the routine, so that end-users are not mis-led. Perhaps a future R package will address some of these issues.

Importantly, the PERMANOVA routine in PRIMER allows the user:

• to specify whether factors are fixed or random,
• to specify whether a factor is nested in one or more other factors,
• to test interaction terms,
• to include one or more quantitative covariates in the analysis,
• to remove individual terms from a model or to perform pooling,
• to correctly analyse:
  • fixed models, random models & mixed models
  • user-specified contrasts
  • BACI designs (before-after/control-impact),
  • asymmetrical designs (e.g., in environmental impact studies),
  • randomised blocks,
  • split plots,
  • hierarchical designs,
  • repeated measures,
  • unbalanced designs (Type I, II or III sums of squares),
  • ... and more.