Rotation Tests

Introduction

Rotation testing is a framework for doing significance testing by computer simulations. An important application is the adjustment of univariate p-values in multiresponse experiments. When analysing several responses by individual significance tests, ordinary significance testing (F-test) is questionable since we will expect a lot of type I errors ("incorrect significance").

An alternative is to adjust the p-values according to the familywise error rate (FWE) criterion. The classical Bonferroni’s correction is, however, extremely conservative and becomes useless in cases with a large number of responses. By using rotation testing it is possible to adjust the p-values in an exact and non-conservative way (Langsrud, 2005).

FWE adjustment can be viewed as being too strict. Instead of considering the probability of at least one type I error, an alternative is to estimate the false discovery rate (FDR) which is the (expected) proportion of type I errors among all responses reported as significant. The appendix (written by Ø. Langsrud) of Moen et al. (2005) describes a new way of adjusting p-values according to FDR by using rotation testing (or permutation testing). Unlike other FDR procedures, this method allows any kind of dependence among the responses.

Papers

This paper describes a generalised framework for doing Monte Carlo tests in multivariate linear regression. The rotation methodology assumes multivariate normality and is a true generalisation of the classical multivariate tests - any imaginable test statistic is allowed. The generalised test statistics are dependent on the unknown covariance matrix. Rotation testing handles this problem by conditioning on sufficient statistics.

Compared to permutation tests, we replace permutations by proper random rotations. Permutation tests avoid the multinormal assumption, but they are limited to relatively simple models. On the other hand, a rotation test can, in particular, be applied to any multivariate generalisation of the univariate F-test.

As an important application, a detailed description of how each single response p-value can be non-conservatively adjusted for multiplicity is given. This method is exact and non-conservative (unlike Bonferroni), and it is a generalisation of the ordinary F-test (except for the computation by simulations). Hence, this paper offers an exact Monte Carlo solution to a classical problem of multiple testing.

KEY WORDS: Conditional inference, Multiple testing, Random orthogonal matrix, Adjusted p-value, Multiple endpoints, Spherical distribution, Microarray data analysis.

ABSTRACT OF AN OLDER VERSION

Motivated by specialised significance tests, Wedderburn (1975) ¹⁾ described how to simulate a multinormal sample conditioned on the sample mean and covariance matrix. Unfortunately, Wedderburn died suddenly and his research report was never published. Later, Cheng (1985) ²⁾ presented a simulation algorithm for the same problem. However, during the years, the potential of such simulations has not been recognised.

The present paper extends Wedderburn's methodology to a generalised framework for doing Monte Carlo tests in multivariate linear regression. Compared to permutation tests, we replace permutations by proper random rotations. Thereby, more general problems can be solved. The rotation methodology is a true generalisation of the classical multivariate tests. The unknown parameters are handled by conditioning on their sufficient statistics. As a specific application, a detailed description of how each single response p-value can be non-conservatively adjusted for multiplicity is given.