esting stereotype threat vs. group differences models

After the study came out, Scott Alexander wrote a terse reply to it: Perceptions Of Required Ability Act As A Proxy For Actual Required Ability In Explaining The Gender Gap. He produced this plot based on GRE math scores:

But actually, we can go further back in time. In a not very well known paper from 2002, Templer and Tomeo reported data showing this pattern:

As such, the rather obvious explanation for the correlation between field level “perceptions of brilliance” and female representation is that women are somewhat worse at math, don’t like math as much, and tend to avoid math heavy fields. A boring but accurate explanation. This doesn’t get you published in Big Journal of course. As such, academics publishing papers in top journals ignore such boring findings. In fact, they didn’t even report them, other than to misleading say:

The second competing hypothesis concerns possible gender differences at the high end of the aptitude distribution [(14, 15); but see (16, 17) for criticism]. Such differences might cause greater gender gaps in fields that, by virtue of their selectivity, sample from the extreme right of the aptitude distribution: The more selective a discipline, the fewer the women.

…

To assess selectivity, we asked faculty participants to estimate the percentage of graduate applicants admitted each year to their department. We then reverse-coded this measure so that higher values indicate more selectivity. Fields that were more selective tended to have higher, rather than lower, female representation, but this correlation did not reach significance, r(28) = 0.34, P = 0.065.

It’s a silly approach. Many university majors are highly sought after, but are not difficult. This is not just an American phenomenon, it is also true in Denmark. The most sought after fields of study are psychology, state science (sociology/political science i.e. bureaucrat studies), animal doctors, midwives. All female dominated fields. In fact, the authors should have known about this particular issue since this is the exact situation featured in the famous Simpson’s paradox:

Their paper now has 1700+ citations, and has spawned a minor industry of publishing equally stupid follow-up studies. Yawn.

Anyway, that’s the background for our study. What did we find? Well, one problem with the Leslie study is that they used students in their own fields to rate the required brilliance, instead of using 3rd party observers, or at least a representative sample:

We surveyed faculty, postdoctoral fellows, and graduate students (N = 1820) from 30 disciplines (12 STEM, 18 SocSci/Hum) (table S1) at geographically diverse high-profile public and private research universities across the United States. Participants were asked questions concerning their own discipline (table S2); responses in each discipline were averaged (tables S3 and S4), and analyses were conducted over disciplines (not in- dividuals). As our dependent measure, we used the percentage of female Ph.D. recipients in each discipline (7).

To assess field-specific ability beliefs, we asked participants to rate their agreement with four statements concerning what is required for success in their field (e.g., “Being a top scholar of [discipline] requires a special aptitude that just can’t be taught”) (table S2). Respondents rated both the extent to which they personally agreed with these statements, and the extent to which they believed other people in their field would agree with the statements. Because answers to these eight questions displayed very similar pat- terns (a = 0.90), they were averaged to produce a field-specific ability belief score for each discipline (with higher scores indicating more emphasis on raw ability).

Their numbers, then, may not entirely reflect the general stereotype, as it is possible that fields vary in their level of field-narcissism/humility. In other words, people in a given field may think it requires a lot of brilliance, whereas those not in it don’t think so. So we did what they should have (also) done and sampled a broad group of doctorate students (n = 98) to rate 55 fields (they had 26). If we correlate our results with their, it looks like this:

Given their sample size of 26 fields, the data is not exactly overwhelming. But we see that there is some correlation between their within-field brilliance stereotype and the general doctorate students’ perception, but it doesn’t reach p < .05. By looking at the outlying cases, one can see the fields that others don’t think requires brilliance but whose practitioners think so, mainly philosophy and English literature. These appear to be the pretentious fields in other words. Here’s the complete set perceptions of required talent for 55 fields:

The overall impression from the results are not so surprising — stereotype results rarely are. In the top we find tough, male-dominated fields with a lot of math. In the bottom we find leftist (gender studies, art history, library stuff, English literature) and low-tier applied fields (sports, nutrition, business administration). Maybe there is some objective way to measure this. For instance, using SAT/GREs to predict grades across fields, and seeing if different levels of SAT/GRE are required to earn the same GPA across fields.

We also compiled sex and race representation percentages among graduate students:

(In hindsight, a stack bar plot would have been much nicer here.) We see that there is substantial variation in what different races prefer to study.

To really hammer it home, we also surveyed a representative sample of 500 adult Americans to rate the scientificness of each field, as well as the proportion of conservatives among professors. The political stereotype looks like this along with some actual voting behavior data from this study:

The dotted line shows the expectation given perfect calibration and accuracy. It can be seen that people’s stereotypes about fields had some accuracy, r = .55 (p < .01) with a voting share estimate from another study. However, all the stereotypes were far too high. The general public vastly overestimates the proportion of conservatives/Republicans among academics. The study of actual voting behavior is quite noisy, so this correlation would likely be larger if we had better data.

Putting all of these together, we get this correlation matrix (pairwise complete data):

If we ignore the correlations that aren’t at least at the p < .01 level, we can note:

• Stereotypes of the need for talent in a field among PhD students correlate with how much the publicly considers the field scientific (r = 0.41). Stereotypes about the necessity of talent also correlates with the proportion of Republicans in the voting behavior study (r = 0.55), but not with the stereotype of conservatives (r = 0.02). GRE Math tilt and math scores also correlate with the talent requirement perception (r’s = .39, 0.49). Going further, the same need for talent correlates negatively with proportion female and Black (r’s = -0.39 and -0.51) but positively with proportion Asian% (r = 0.49).


• Asians are particularly attracted to high math fields, as both math tilt and math scores correlate strongly with Asian % (r’s = 0.77 and 0.68).


• Blacks and women tend to be found in the same fields (r = 0.47).


• Women are found especially in the fields considered to be and which are left-wing (r’s = -0.51 and -0.62), and those which have lower GRE’s low GRE maths and math tilts (r’s = -0.67, -0.79, -0.42).


• GRE math tilt correlates with perceived scientificness, perceived conservativeness and actual Republic voting (r’s = 0.36, 0.49, 0.71).

In general, these are sensible findings that one could figure out by spending some time on campus or looking at academic Twitter posting.

Furthermore, in a direct test of Leslie’s stereotype threat model, we find that actual GRE scores predict female representation better than their within discipline stereotype measure (though p = 0.048, n = 26). They didn’t include enough fields in their survey for us to be more certain unfortunately.

In terms of perceptions of professors’ politics, there weren’t large differences by who the subjects in the national survey voted for:

In other words, Republican and Democrat voters agree nearly perfectly in their stereotypes about the politics of professors, and both groups severely overestimate the proportion of Republicans among professors.

There is a very interesting non-linear association between political centrism and perceived scientificness:

It seems ordinary people consider centrism or apoliticalness a hallmark of good science. This pattern also holds if one looks at ratings by Republicans or Democrats alone.

Note that in each case, the distribution is tilted towards their own political side.

When it comes to perceptions of scientificness by field, there was likewise strong agreement across political parties:

Democrat voters generally perceived everything to be slightly more scientific than Republicans did (dots are above the dotted line of perfect agreement).

Note that some of these plots aren’t in the published paper, I made them for the purpose of this post. They can go into some future study.

The field level data are here for those interested in using them for other work, or making nicer figures (some data are missing because Bronski did the analyses, I will add them when I get them). More statistical details are here.