Let’s have some fun and pick apart a paper (try saying “pick apart a paper” 10 times fast!)

The question is, did 9/11 impact the character of Americans? I mean that personally not just in political attitudes towards this or that. It’s a tough question to answer empirically.

“What do you mean by character?” is the obvious first hurdle. Peterson and Seligman (2003) had a go, using their VIA model. They have a website, authentichappiness.com, where people can take a VIA self-report strengths test. I’ve discussed this model previously, here’s an overview, and here’s a comparison to another model of strengths. Your interest in the rest of the article will depend on how suitable you think that tool is to answering this question. It’s pretty new and quite easy to pick it apart (see the previous posts and comments).

So, thousands of people log in to that site and take the strengths test, giving the researchers a good opportunity to compare the results pre and post 9/11. First they looked at the 30 days before compared with the 30 days after 9/11. They found an overall difference, and then narrowed down to look at individual strengths. This is a part I’m confused about. There are 24 VIA strengths in this model, but they used a p value of .01. Surely they should have used .05 / 24 = .002? I’ve looked through the paper several times and can’t see a justification for using .01, it seems pretty arbitrary.

Anyway, using .01 they found significant differences for the strengths of gratitude, hope, kindness, leadership, love, spirituality and teamwork. Kind of interesting, teamwork makes sense, maybe you’d expect a drop in things like hope straight after a terrorist attack.

When testing longer time periods, they stuck to the strengths that were significantly different in the month immediately after “For the sake of convenience”. Here are the results (this is a composite measure of the strengths identified in the first analysis which they name “Theological Virtues”):

At first glance 9/11/01 is clearly separating this increase. But how big is the difference? Effect sizes are not reported in the paper. As you can see in the graph, the difference is from about 3.7 to 3.8. What does it mean to be .1 higher in a combination of gratitude, hope, kindness, leadership, love, spirituality and teamwork? Does that mean anything in real terms, or does the difference only exist statistically? Even then, the exact p value is not given, in favour of “ps < .05". So we're back to .05 now, even though they say they tested each of the pre to each of the post time-points (12 tests), and presumably the ps are between .01 and .05 otherwise why not say < .01, or less than .001? We should also note that the sample sizes are massive - 4510 participants overall, which cannot help but contribute to lower p values, regardless of real-world effects. Anyway, even if this result was correct, maybe it isn't truly representative of the nation. As I mentioned, these results were from people who found the website (not a controlled sample). "Walk-ins" you might say. Another explanation is, after 9/11, people with more hope, leadership, gratitude etc., were more inclined to seek out and complete questionnaires of this type. I started by asking whether 9/11 affected the character of Americans. The answer is, "Who knows?" Saying that 9/11 gave Americans more gratitude, hope, kindness, leadership, love, spirituality and teamwork is a nice story, but I don't think these results really show that. Reference: Peterson, C.,&Seligman, M. E. P. (2003). Character strengths before and after 9/11. Psychological Science, 14(4), 381-384.