Overview

This documents reanalysis a dataset from an Experiment performed by Singmann and Klauer (2011) using the ANOVA functionality of afex followed by post-hoc tests using package lsmeans (Lenth, 2015). After a brief description of the dataset and research question, the code and results are presented.

Description of Experiment and Data

Singmann and Klauer (2011) were interested in whether or not conditional reasoning can be explained by a single process or whether multiple processes are necessary to explain it. To provide evidence for multiple processes we aimed to establish a double dissociation of two variables: instruction type and problem type. Instruction type was manipulated between-subjects, one group of participants received deductive instructions (i.e., to treat the premises as given and only draw necessary conclusions) and a second group of participants received probabilistic instructions (i.e., to reason as in an everyday situation; we called this “inductive instruction” in the manuscript). Problem type consisted of two different orthogonally crossed variables that were manipulated within-subjects, validity of the problem (formally valid or formally invalid) and plausibility of the problem (inferences which were consisted with the background knowledge versus problems that were inconsistent with the background knowledge). The critical comparison across the two conditions was among problems which were valid and implausible with problems that were invalid and plausible. For example, the next problem was invalid and plausible:

If a person is wet, then the person fell into a swimming pool.
A person fell into a swimming pool.
How valid is the conclusion/How likely is it that the person is wet?

For those problems we predicted that under deductive instructions responses should be lower (as the conclusion does not necessarily follow from the premises) as under probabilistic instructions. For the valid but implausible problem, an example is presented next, we predicted the opposite pattern:

If a person is wet, then the person fell into a swimming pool.
A person is wet.
How valid is the conclusion/How likely is it that the person fell into a swimming pool?

Our study also included valid and plausible and invalid and implausible problems.

In contrast to the analysis reported in the manuscript, we initially do not separate the analysis into affirmation and denial problems, but first report an analysis on the full set of inferences, MP, MT, AC, and DA, where MP and MT are valid and AC and DA invalid. We report a reanalysis of our Experiment 1 only. Note that the factor plausibility is not present in the original manuscript, there it is a results of a combination of other factors.

Data and R Preperation

require(afex) # needed for ANOVA, lsmeans is loaded automatically.
require(multcomp) # for advanced control for multiple testing/Type 1 errors.
require(lattice) # for plots

data(sk2011.1)
str(sk2011.1)

## 'data.frame':    640 obs. of  9 variables:
##  $ id          : Factor w/ 40 levels "8","9","10","12",..: 3 3 3 3 3 3 3 3 3 3 ...
##  $ instruction : Factor w/ 2 levels "deductive","probabilistic": 2 2 2 2 2 2 2 2 2 2 ...
##  $ plausibility: Factor w/ 2 levels "plausible","implausible": 1 2 2 1 2 1 1 2 1 2 ...
##  $ inference   : Factor w/ 4 levels "MP","MT","AC",..: 4 2 1 3 4 2 1 3 4 2 ...
##  $ validity    : Factor w/ 2 levels "valid","invalid": 2 1 1 2 2 1 1 2 2 1 ...
##  $ what        : Factor w/ 2 levels "affirmation",..: 2 2 1 1 2 2 1 1 2 2 ...
##  $ type        : Factor w/ 2 levels "original","reversed": 2 2 2 2 1 1 1 1 2 2 ...
##  $ response    : int  100 60 94 70 100 99 98 49 82 50 ...
##  $ content     : Factor w/ 4 levels "C1","C2","C3",..: 1 1 1 1 2 2 2 2 3 3 ...

An important feature in the data is that each participant provided two responses for each cell of the design (the content is different for each of those, each participant saw all four contents). These two data points will be aggregated automatically by afex.

with(sk2011.1, table(inference, id, plausibility))

## , , plausibility = plausible
## 
##          id
## inference 8 9 10 12 13 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
##        MP 2 2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2
##        MT 2 2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2
##        AC 2 2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2
##        DA 2 2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2
##          id
## inference 37 38 39 40 41 42 43 44 46 47 48 49 50
##        MP  2  2  2  2  2  2  2  2  2  2  2  2  2
##        MT  2  2  2  2  2  2  2  2  2  2  2  2  2
##        AC  2  2  2  2  2  2  2  2  2  2  2  2  2
##        DA  2  2  2  2  2  2  2  2  2  2  2  2  2
## 
## , , plausibility = implausible
## 
##          id
## inference 8 9 10 12 13 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
##        MP 2 2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2
##        MT 2 2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2
##        AC 2 2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2
##        DA 2 2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2  2
##          id
## inference 37 38 39 40 41 42 43 44 46 47 48 49 50
##        MP  2  2  2  2  2  2  2  2  2  2  2  2  2
##        MT  2  2  2  2  2  2  2  2  2  2  2  2  2
##        AC  2  2  2  2  2  2  2  2  2  2  2  2  2
##        DA  2  2  2  2  2  2  2  2  2  2  2  2  2

ANOVA

To get the full ANOVA table for the model, we simply pass it to aov_ez (aov_car or aov4 would be alternatives producing the same results) using the design as described above. We save the returned object for further analysis.

a1 <- aov_ez("id", "response", sk2011.1, between = "instruction", 
       within = c("inference", "plausibility"))

## Warning in aov_car(formula = as.formula(formula), data = data, fun.aggregate =
## fun.aggregate, : More than one observation per cell, aggregating the data using mean (i.e,
## fun.aggregate = mean)!

## Contrasts set to contr.sum for the following variables: instruction

a1 # the default print method prints a data.frame produced by nice

##                               Effect           df     MSE         F  ges p.value
## 1                        instruction        1, 38 2027.42      0.31 .003     .58
## 2                          inference 2.66, 101.12  959.12   5.81 **  .06    .002
## 3              instruction:inference 2.66, 101.12  959.12   6.00 **  .07    .001
## 4                       plausibility        1, 38  468.82 34.23 ***  .07  <.0001
## 5           instruction:plausibility        1, 38  468.82  10.67 **  .02    .002
## 6             inference:plausibility  2.29, 87.11  318.91    2.87 + .009     .06
## 7 instruction:inference:plausibility  2.29, 87.11  318.91    3.98 *  .01     .02

As mentioned before, the two responses per cell of the design and participants are aggregated for the analysis as indicated by the warning message. Furthermore, the degrees of freedom are Greenhouse-Geisser corrected per default for all effects involving inference, as inference is a within-subject factor with more than two levels (i.e., MP, MT, AC, & DA). In line with our expectations, the three-way interaction is significant.

The object printed per default for afex_aov objects (produced by nice) can also be printed nicely using knitr:

knitr::kable(nice(a1))

Effect	df	MSE	F	ges	p.value
instruction	1, 38	2027.42	0.31	.003	.58
inference	2.66, 101.12	959.12	5.81 **	.06	.002
instruction:inference	2.66, 101.12	959.12	6.00 **	.07	.001
plausibility	1, 38	468.82	34.23 ***	.07	<.0001
instruction:plausibility	1, 38	468.82	10.67 **	.02	.002
inference:plausibility	2.29, 87.11	318.91	2.87 +	.009	.06
instruction:inference:plausibility	2.29, 87.11	318.91	3.98 *	.01	.02

Alternatively, the anova method for afex_aov objects returns a data.frame of class anova that can be passed to, for example, xtable for nice formatting:

print(xtable::xtable(anova(a1), digits = c(rep(2, 5), 3, 4)), type = "html")

	num Df	den Df	MSE	F	ges	Pr(>F)
instruction	1.00	38.00	2027.42	0.31	0.003	0.5830
inference	2.66	101.12	959.12	5.81	0.063	0.0016
instruction:inference	2.66	101.12	959.12	6.00	0.065	0.0013
plausibility	1.00	38.00	468.82	34.23	0.068	0.0000
instruction:plausibility	1.00	38.00	468.82	10.67	0.022	0.0023
inference:plausibility	2.29	87.11	318.91	2.87	0.009	0.0551
instruction:inference:plausibility	2.29	87.11	318.91	3.98	0.013	0.0177

Post-Hoc Contrasts and Plotting

To further analyze the data we need to pass it to package lsmeans, a package that offers great functionality for both plotting and contrasts of all kind. A lot of information on lsmeans can be obtained in its vignette. lsmeans can work with afex_aov objects directly as afex comes with the necessary methods for the generic functions defined in lsmeans. lsmeans uses the ANOVA model estimated via base R’s aov function that is part of an afex_aov object.

Some First Contrasts

Main Effects Only

This object can now be passed to lsmeans, for example to obtain the marginal means of the four inferences:

m1 <- lsmeans(a1, ~ inference)

## NOTE: Results may be misleading due to involvement in interactions

m1

##  inference   lsmean       SE     df lower.CL upper.CL
##  MP        87.51250 3.783074 126.87 80.02641 94.99859
##  MT        76.68125 3.783074 126.87 69.19516 84.16734
##  AC        69.41250 3.783074 126.87 61.92641 76.89859
##  DA        82.95625 3.783074 126.87 75.47016 90.44234
## 
## Results are averaged over the levels of: instruction, plausibility 
## Confidence level used: 0.95

This object can now also be used to compare whether or not there are differences between the levels of the factor:

pairs(m1)

##  contrast  estimate       SE  df t.ratio p.value
##  MP - MT   10.83125 4.611854 114   2.349  0.0933
##  MP - AC   18.10000 4.611854 114   3.925  0.0008
##  MP - DA    4.55625 4.611854 114   0.988  0.7566
##  MT - AC    7.26875 4.611854 114   1.576  0.3963
##  MT - DA   -6.27500 4.611854 114  -1.361  0.5266
##  AC - DA  -13.54375 4.611854 114  -2.937  0.0206
## 
## Results are averaged over the levels of: instruction, plausibility 
## P value adjustment: tukey method for comparing a family of 4 estimates

To obtain more powerful p-value adjustments, we can furthermore pass it to multcomp (Bretz, Hothorn, & Westfall, 2011):

summary(as.glht(pairs(m1)), test=adjusted("free"))

## 
##   Simultaneous Tests for General Linear Hypotheses
## 
## Linear Hypotheses:
##              Estimate Std. Error t value Pr(>|t|)    
## MP - MT == 0   10.831      4.612   2.349 0.068998 .  
## MP - AC == 0   18.100      4.612   3.925 0.000797 ***
## MP - DA == 0    4.556      4.612   0.988 0.325273    
## MT - AC == 0    7.269      4.612   1.576 0.281795    
## MT - DA == 0   -6.275      4.612  -1.361 0.296932    
## AC - DA == 0  -13.544      4.612  -2.937 0.017434 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- free method)

A Simple interaction

We could now also be interested in the marginal means of the inferences across the two instruction types. lsmeans offers two ways to do so. The first splits the contrasts across levels of the factor.

m2 <- lsmeans(a1, ~ inference|instruction)

## NOTE: Results may be misleading due to involvement in interactions

m2

## instruction = deductive:
##  inference  lsmean       SE     df lower.CL  upper.CL
##  MP        97.2875 5.350074 126.87 86.70057 107.87443
##  MT        70.4000 5.350074 126.87 59.81307  80.98693
##  AC        61.4875 5.350074 126.87 50.90057  72.07443
##  DA        81.8125 5.350074 126.87 71.22557  92.39943
## 
## instruction = probabilistic:
##  inference  lsmean       SE     df lower.CL  upper.CL
##  MP        77.7375 5.350074 126.87 67.15057  88.32443
##  MT        82.9625 5.350074 126.87 72.37557  93.54943
##  AC        77.3375 5.350074 126.87 66.75057  87.92443
##  DA        84.1000 5.350074 126.87 73.51307  94.68693
## 
## Results are averaged over the levels of: plausibility 
## Confidence level used: 0.95

Consequently test are also only performed within each level:

pairs(m2)

## instruction = deductive:
##  contrast estimate       SE  df t.ratio p.value
##  MP - MT   26.8875 6.522147 114   4.122  0.0004
##  MP - AC   35.8000 6.522147 114   5.489  <.0001
##  MP - DA   15.4750 6.522147 114   2.373  0.0882
##  MT - AC    8.9125 6.522147 114   1.366  0.5229
##  MT - DA  -11.4125 6.522147 114  -1.750  0.3031
##  AC - DA  -20.3250 6.522147 114  -3.116  0.0122
## 
## instruction = probabilistic:
##  contrast estimate       SE  df t.ratio p.value
##  MP - MT   -5.2250 6.522147 114  -0.801  0.8538
##  MP - AC    0.4000 6.522147 114   0.061  0.9999
##  MP - DA   -6.3625 6.522147 114  -0.976  0.7636
##  MT - AC    5.6250 6.522147 114   0.862  0.8241
##  MT - DA   -1.1375 6.522147 114  -0.174  0.9981
##  AC - DA   -6.7625 6.522147 114  -1.037  0.7282
## 
## Results are averaged over the levels of: plausibility 
## P value adjustment: tukey method for comparing a family of 4 estimates

The second version treats all factor combinations together, producing a considerably larger number of pairwise comparisons:

m3 <- lsmeans(a1, ~ inference:instruction)

## NOTE: Results may be misleading due to involvement in interactions

m3

##  inference instruction    lsmean       SE     df lower.CL  upper.CL
##  MP        deductive     97.2875 5.350074 126.87 86.70057 107.87443
##  MT        deductive     70.4000 5.350074 126.87 59.81307  80.98693
##  AC        deductive     61.4875 5.350074 126.87 50.90057  72.07443
##  DA        deductive     81.8125 5.350074 126.87 71.22557  92.39943
##  MP        probabilistic 77.7375 5.350074 126.87 67.15057  88.32443
##  MT        probabilistic 82.9625 5.350074 126.87 72.37557  93.54943
##  AC        probabilistic 77.3375 5.350074 126.87 66.75057  87.92443
##  DA        probabilistic 84.1000 5.350074 126.87 73.51307  94.68693
## 
## Results are averaged over the levels of: plausibility 
## Confidence level used: 0.95

pairs(m3)

##  contrast                            estimate       SE     df t.ratio p.value
##  MP,deductive - MT,deductive          26.8875 6.522147 114.00   4.122  0.0018
##  MP,deductive - AC,deductive          35.8000 6.522147 114.00   5.489  <.0001
##  MP,deductive - DA,deductive          15.4750 6.522147 114.00   2.373  0.2649
##  MP,deductive - MP,probabilistic      19.5500 7.566147 126.87   2.584  0.1716
##  MP,deductive - MT,probabilistic      14.3250 7.566147 126.87   1.893  0.5581
##  MP,deductive - AC,probabilistic      19.9500 7.566147 126.87   2.637  0.1527
##  MP,deductive - DA,probabilistic      13.1875 7.566147 126.87   1.743  0.6592
##  MT,deductive - AC,deductive           8.9125 6.522147 114.00   1.366  0.8704
##  MT,deductive - DA,deductive         -11.4125 6.522147 114.00  -1.750  0.6548
##  MT,deductive - MP,probabilistic      -7.3375 7.566147 126.87  -0.970  0.9779
##  MT,deductive - MT,probabilistic     -12.5625 7.566147 126.87  -1.660  0.7125
##  MT,deductive - AC,probabilistic      -6.9375 7.566147 126.87  -0.917  0.9839
##  MT,deductive - DA,probabilistic     -13.7000 7.566147 126.87  -1.811  0.6141
##  AC,deductive - DA,deductive         -20.3250 6.522147 114.00  -3.116  0.0462
##  AC,deductive - MP,probabilistic     -16.2500 7.566147 126.87  -2.148  0.3906
##  AC,deductive - MT,probabilistic     -21.4750 7.566147 126.87  -2.838  0.0948
##  AC,deductive - AC,probabilistic     -15.8500 7.566147 126.87  -2.095  0.4239
##  AC,deductive - DA,probabilistic     -22.6125 7.566147 126.87  -2.989  0.0644
##  DA,deductive - MP,probabilistic       4.0750 7.566147 126.87   0.539  0.9994
##  DA,deductive - MT,probabilistic      -1.1500 7.566147 126.87  -0.152  1.0000
##  DA,deductive - AC,probabilistic       4.4750 7.566147 126.87   0.591  0.9989
##  DA,deductive - DA,probabilistic      -2.2875 7.566147 126.87  -0.302  1.0000
##  MP,probabilistic - MT,probabilistic  -5.2250 6.522147 114.00  -0.801  0.9928
##  MP,probabilistic - AC,probabilistic   0.4000 6.522147 114.00   0.061  1.0000
##  MP,probabilistic - DA,probabilistic  -6.3625 6.522147 114.00  -0.976  0.9770
##  MT,probabilistic - AC,probabilistic   5.6250 6.522147 114.00   0.862  0.9887
##  MT,probabilistic - DA,probabilistic  -1.1375 6.522147 114.00  -0.174  1.0000
##  AC,probabilistic - DA,probabilistic  -6.7625 6.522147 114.00  -1.037  0.9677
## 
## Results are averaged over the levels of: plausibility 
## P value adjustment: tukey method for comparing a family of 8 estimates

Running Custom Contrasts

Objects returned from lsmeans can also be used to test specific contrasts. For this, we can simply create a list, where each element corresponds to one contrasts. A contrast is defined as a vector of constants on the reference grid (i.e., the object returned from lsmeans, here m3). For example, we might be interested in whether there is a difference between the valid and invalid inferences in each of the two conditions.

c1 <- list(
  v_i.ded = c(0.5, 0.5, -0.5, -0.5, 0, 0, 0, 0),
  v_i.prob = c(0, 0, 0, 0, 0.5, 0.5, -0.5, -0.5)
  )

contrast(m3, c1, adjust = "holm")

##  contrast estimate       SE  df t.ratio p.value
##  v_i.ded  12.19375 4.611854 114   2.644  0.0187
##  v_i.prob -0.36875 4.611854 114  -0.080  0.9364
## 
## Results are averaged over the levels of: plausibility 
## P value adjustment: holm method for 2 tests

summary(as.glht(contrast(m3, c1)), test =adjusted("free"))

## 
##   Simultaneous Tests for General Linear Hypotheses
## 
## Linear Hypotheses:
##               Estimate Std. Error t value Pr(>|t|)  
## v_i.ded == 0   12.1937     4.6119   2.644   0.0186 *
## v_i.prob == 0  -0.3687     4.6119  -0.080   0.9364  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- free method)

The results can be interpreted as in line with expectations. Responses are larger for valid than invalid problems in the deductive, but not the probabilistic condition.

Plotting

Function lsmip from package lsmeans can be used for plotting the data directly from an afex_aov object. As said initially, we are interested in the three-way interaction of instruction with inference, plausibility, and instruction. A plot of this interaction could be the following:

lsmip(a1, instruction ~ inference|plausibility)

Replicate Analysis from Singmann and Klauer (2011)

As this plot is not very helpful, we now fit a new ANOVA model in which we separate the data in affirmation and denial inferences, as done in the original manuscript and plot the data then a second time.

a2 <- aov_ez("id", "response", sk2011.1, between = "instruction", 
       within = c("validity", "plausibility", "what"))

## Warning in aov_car(formula = as.formula(formula), data = data, fun.aggregate =
## fun.aggregate, : More than one observation per cell, aggregating the data using mean (i.e,
## fun.aggregate = mean)!

## Contrasts set to contr.sum for the following variables: instruction

a2

##                                    Effect    df     MSE         F    ges p.value
## 1                             instruction 1, 38 2027.42      0.31   .003     .58
## 2                                validity 1, 38  678.65    4.12 *    .01     .05
## 3                    instruction:validity 1, 38  678.65    4.65 *    .01     .04
## 4                            plausibility 1, 38  468.82 34.23 ***    .07  <.0001
## 5                instruction:plausibility 1, 38  468.82  10.67 **    .02    .002
## 6                                    what 1, 38  660.52      0.22  .0007     .64
## 7                        instruction:what 1, 38  660.52      2.60   .008     .11
## 8                   validity:plausibility 1, 38  371.87      0.14  .0002     .71
## 9       instruction:validity:plausibility 1, 38  371.87    4.78 *   .008     .04
## 10                          validity:what 1, 38 1213.14   9.80 **    .05    .003
## 11              instruction:validity:what 1, 38 1213.14   8.60 **    .05    .006
## 12                      plausibility:what 1, 38  204.54   9.97 **   .009    .003
## 13          instruction:plausibility:what 1, 38  204.54    5.23 *   .005     .03
## 14             validity:plausibility:what 1, 38  154.62      0.03 <.0001     .85
## 15 instruction:validity:plausibility:what 1, 38  154.62      0.42  .0003     .52

Then we plot the data from this ANOVA.

lsmip(a2, ~instruction ~ plausibility+validity|what, 
      scales = list(x=list(
        at = 1:4,
        labels = c("pl:v", "im:v", "pl:i", "im:i")
        )))

We see the critical predicted cross-over interaction in the left of those two graphs. For valid but implausible problems (im:v) deductive responses are larger than probabilistic responses. The opposite is true for invalid but plausible problems (pl:i). We now tests these differences at each of the four x-axis ticks in each plot using custom contrasts (diff_1 to diff_4). Furthermore, we test for a validity effect and plausibility effect in both conditions.

(m4 <- lsmeans(a2, ~instruction+plausibility+validity|what))

## what = affirmation:
##  instruction   plausibility validity lsmean      SE     df lower.CL upper.CL
##  deductive     plausible    valid    99.475 6.01019 183.89  87.6172 111.3328
##  probabilistic plausible    valid    95.300 6.01019 183.89  83.4422 107.1578
##  deductive     implausible  valid    95.100 6.01019 183.89  83.2422 106.9578
##  probabilistic implausible  valid    60.175 6.01019 183.89  48.3172  72.0328
##  deductive     plausible    invalid  66.950 6.01019 183.89  55.0922  78.8078
##  probabilistic plausible    invalid  90.550 6.01019 183.89  78.6922 102.4078
##  deductive     implausible  invalid  56.025 6.01019 183.89  44.1672  67.8828
##  probabilistic implausible  invalid  64.125 6.01019 183.89  52.2672  75.9828
## 
## what = denial:
##  instruction   plausibility validity lsmean      SE     df lower.CL upper.CL
##  deductive     plausible    valid    70.550 6.01019 183.89  58.6922  82.4078
##  probabilistic plausible    valid    92.975 6.01019 183.89  81.1172 104.8328
##  deductive     implausible  valid    70.250 6.01019 183.89  58.3922  82.1078
##  probabilistic implausible  valid    72.950 6.01019 183.89  61.0922  84.8078
##  deductive     plausible    invalid  86.525 6.01019 183.89  74.6672  98.3828
##  probabilistic plausible    invalid  87.450 6.01019 183.89  75.5922  99.3078
##  deductive     implausible  invalid  77.100 6.01019 183.89  65.2422  88.9578
##  probabilistic implausible  invalid  80.750 6.01019 183.89  68.8922  92.6078
## 
## Confidence level used: 0.95

c2 <- list(
  diff_1 = c(1, -1, 0, 0, 0, 0, 0, 0),
  diff_2 = c(0, 0, 1, -1, 0, 0, 0, 0),
  diff_3 = c(0, 0, 0, 0,  1, -1, 0, 0),
  diff_4 = c(0, 0, 0, 0,  0, 0, 1, -1),
  val_ded  = c(0.5, 0, 0.5, 0, -0.5, 0, -0.5, 0),
  val_prob = c(0, 0.5, 0, 0.5, 0, -0.5, 0, -0.5),
  plau_ded   = c(0.5, 0, -0.5, 0, -0.5, 0, 0.5, 0),
  plau_prob  = c(0, 0.5, 0, -0.5, 0, 0.5, 0, -0.5)
  )
contrast(m4, c2, adjust = "holm")

## what = affirmation:
##  contrast  estimate       SE     df t.ratio p.value
##  diff_1      4.1750 8.499692 183.89   0.491  1.0000
##  diff_2     34.9250 8.499692 183.89   4.109  0.0004
##  diff_3    -23.6000 8.499692 183.89  -2.777  0.0303
##  diff_4     -8.1000 8.499692 183.89  -0.953  1.0000
##  val_ded    35.8000 6.877109  70.38   5.206  <.0001
##  val_prob    0.4000 6.877109  70.38   0.058  1.0000
##  plau_ded   -3.2750 3.627998  64.94  -0.903  1.0000
##  plau_prob  30.7750 4.102924  65.86   7.501  <.0001
## 
## what = denial:
##  contrast  estimate       SE     df t.ratio p.value
##  diff_1    -22.4250 8.499692 183.89  -2.638  0.0633
##  diff_2     -2.7000 8.499692 183.89  -0.318  1.0000
##  diff_3     -0.9250 8.499692 183.89  -0.109  1.0000
##  diff_4     -3.6500 8.499692 183.89  -0.429  1.0000
##  val_ded   -11.4125 6.877109  70.38  -1.659  0.6088
##  val_prob   -1.1375 6.877109  70.38  -0.165  1.0000
##  plau_ded   -4.5625 3.627998  64.94  -1.258  1.0000
##  plau_prob  13.3625 4.102924  65.86   3.257  0.0143
## 
## P value adjustment: holm method for 8 tests

As the resulting eight contrasts have different numbers of degrees-of-freedom, we can only pass them to multcomp in small batches. This gives us more powerful Type 1 error corrections but overall a reduced correction as we now control for three families of tests (i.e., overall Type 1 error probability of .15).

summary(as.glht(contrast(m4, c2[1:4])), test =adjusted("free"))

## $`what = affirmation`
## 
##   Simultaneous Tests for General Linear Hypotheses
## 
## Linear Hypotheses:
##             Estimate Std. Error t value Pr(>|t|)    
## diff_1 == 0    4.175      8.500   0.491 0.623874    
## diff_2 == 0   34.925      8.500   4.109 0.000234 ***
## diff_3 == 0  -23.600      8.500  -2.777 0.017357 *  
## diff_4 == 0   -8.100      8.500  -0.953 0.564739    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- free method)
## 
## 
## $`what = denial`
## 
##   Simultaneous Tests for General Linear Hypotheses
## 
## Linear Hypotheses:
##             Estimate Std. Error t value Pr(>|t|)  
## diff_1 == 0  -22.425      8.500  -2.638    0.033 *
## diff_2 == 0   -2.700      8.500  -0.318    0.955  
## diff_3 == 0   -0.925      8.500  -0.109    0.955  
## diff_4 == 0   -3.650      8.500  -0.429    0.955  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- free method)

summary(as.glht(contrast(m4, c2[5:6])), test =adjusted("free"))

## $`what = affirmation`
## 
##   Simultaneous Tests for General Linear Hypotheses
## 
## Linear Hypotheses:
##               Estimate Std. Error t value Pr(>|t|)    
## val_ded == 0    35.800      6.877   5.206 3.69e-06 ***
## val_prob == 0    0.400      6.877   0.058    0.954    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- free method)
## 
## 
## $`what = denial`
## 
##   Simultaneous Tests for General Linear Hypotheses
## 
## Linear Hypotheses:
##               Estimate Std. Error t value Pr(>|t|)
## val_ded == 0   -11.412      6.877  -1.659    0.192
## val_prob == 0   -1.137      6.877  -0.165    0.869
## (Adjusted p values reported -- free method)

summary(as.glht(contrast(m4, c2[7:8])), test =adjusted("free"))

## $`what = affirmation`
## 
##   Simultaneous Tests for General Linear Hypotheses
## 
## Linear Hypotheses:
##                Estimate Std. Error t value Pr(>|t|)    
## plau_ded == 0    -3.275      3.628  -0.903     0.37    
## plau_prob == 0   30.775      4.103   7.501  4.5e-10 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- free method)
## 
## 
## $`what = denial`
## 
##   Simultaneous Tests for General Linear Hypotheses
## 
## Linear Hypotheses:
##                Estimate Std. Error t value Pr(>|t|)   
## plau_ded == 0    -4.562      3.628  -1.258  0.21304   
## plau_prob == 0   13.362      4.103   3.257  0.00358 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- free method)

The pattern for the affirmation problems is in line with the expectations: We find the predicted differences between the instruction types for valid and implausible (diff_2) and invalid and plausible (diff_3) and the predicted non-differences for the other two problems (diff_1 and diff_4). Furthermore, we find a validity effect in the deductive but not in the probabilistic condition. Likewise, we find a plausibility effect in the probabilistic but not in the deductive condition.

Some Cautionary Notes

While the df of the ANOVA tables are Greenhouse-Geisser corrected per default for within-subject factors with more than two levels, this is not the case for post-hoc tests or contrasts using lsmeans. The contrasts use uncorrected degrees of freedom that are Satterthwaite approximated. This most likely produces anti-conservative tests if compound symmetry/sphericity is violated.
For unbalanced samples, aov is usually not the correct choise. This is why the test of effects is based on car::Anova. However, for lsmeans we need to use aov models. However, lsmeans offers the option to weight the marginal means differently in case of different group sizes (i.e., unbalanced data). For example, it offers the option that each group is assumed to be of equal size (i.e., weights = "equal") or proportionally (i.e., weights = "proportional"). See help of lsmeans for more information.
Choosing the right correction for multiple testing can be difficult. In fact multcomp comes with an accompanying book (Bretz et al., 2011). If the degrees-of-freedom of all contrasts are identical using multcomp’s method free is more powerful than simply using the Bonferroni-Holm method. free is a generalization of the Bonferroni-Holm method that takes the correlations among the model parameters into account and uniformly more powerful.
For data sets with many within-subject factors, creating the aov object can take some time (i.e., compared to producing the ANOVA table which is usually very fast). Those objects are also comparatively large, often multiple MB. If speed is important and one is not interested in employing lsmeans one can set return = "nice" in the call to the ANOVA function to only receive the ANOVA table (this can also be done globally: afex_options(return_aov = "nice")).

References

Bretz, F., Hothorn, T., & Westfall, P. H. (2011). Multiple comparisons using R. Boca Raton, FL: CRC Press. http://cran.r-project.org/package=multcomp
Singmann, H., & Klauer, K. C. (2011). Deductive and inductive conditional inferences: Two modes of reasoning. Thinking & Reasoning, 17(3), 247-281. doi: 10.1080/13546783.2011.572718
Lenth, R. V. (2015). lsmeans: Least-Squares Means. R package version 2.16-4. http://cran.r-project.org/package=lsmeans

ANOVA and Post-Hoc Contrasts: Reanalysis of Singmann and Klauer (2011)

Henrik Singmann

2015-10-24