The Anatomy of a Bad Science: Reflections on Nattrass’ ‘commentary’

In its May/June 2020 issue, the South African Journal of Science (S Afr J Sci) published a two-page ‘commentary’ by Nicoli Nattrass, a professor of economics and co-Director of the Institute for Communities and Wildlife in Africa at the University of Cape Town (UCT). The ‘commentary’ is titled ‘Why are black South African students less likely to consider studying biological sciences?’ The piece reported ‘the findings’ of a survey of students at the University of Cape Town that was conducted in mid-2019. This was supposed to be an exploratory study, intended to offer insights into why Black South African students do not study biological sciences. In this article, I explore three dimensions of the research and the results reported in Nattrass’ commentary.

First, I assess the methodological framing of the 'exploratory' survey and argue that the study design is gravely deficient. Second, I examine the descriptive and inferential statistics that Nattrass reports. I argue that there is significant dissonance between the results of the regression models reported and the conclusions that Nattrass draws. Third, I examine a set of highly problematic claims that shape the survey and offer insights into the presuppositions that explain the conclusions Nattrass draws from the study. These presuppositions involve a set of racially charged tropes about 'black South African' students. The summative conclusion is that what we have is bad science hanging on the horns of prejudice.

Methodological problems in the study design
The first issue of concern in engaging with the commentary concerns the study design. The study participants were recruited 'by approaching students during the lunch break.' 1 Nattrass referred to this as 'opportunistic survey.' 1 The choice of sampling technique is puzzling, instead of more robust probability sampling methods. The claim that the study is 'exploratory' is not credible. An exploratory study seeks to address, in tentative ways, issues that might not have been previously studied. Using the students at UCT may be permissible for such studyas a prelude to a more national survey-but this would not justify the use of non-probability sampling. The reason for this is simple. The study seeks to answer a 'why' question not a 'how' question. Even in its 'exploratory' nature, the (tentative, preliminary) answers that the researchers sought involve making claims beyond the study sample. Generalisability is at the heart of the question the researcher sought to answer.
The choice of non-probability sampling is all the more puzzling because, as staff of the university, the researchers had access to a reliable sampling frame from which they could develop the probability sampling. The sampling frame would involve a complete record of all the students at the university. As researchers at the university, they could easily access such sampling frame, from the Office of the Registrar. The frame would have provided the relevant socio-demographic distribution of the student population, their degree options, and other characteristics needed for generating a credible sample.
With the sampling frame, the researchers could have employed an appropriate stratified, random sampling procedure. Race-categories, fields of study (as a proxy for career trajectory), and socio-economic status of the study participants are all critical for the research questions that underpin this study. The sampling frame would have provided the basis for a robust stratification of the population from which they can draw the study sample. Further, the researchers would have indicated, beforehand, the margin of error used in arriving at the sample size. The error margin would have helped in interpreting even the descriptive statistics since we would know the confidence interval for the reported percentile distribution of the sample.
It is unclear if a scientific committee ever considered the study proposal at the university. If this was the case, the study should have been flagged. A research ethics committee would be concerned with issues of whether the research process could open the students to harm. Still, an approval that the study is scientifically robust should have come before the ethics clearance application.
The effects of the methodological deficiencies highlighted above are that the researchers stumbled blindly into the field. While 'black South African' students'(BSA) share of the total population of UCT students was 30 per cent, they are 54 per cent of the study sample. The sample size derived with convenience sampling was 211 students. Properly designed, with a 3 per cent margin of error, we would have expected a sample size of at least 1 030 students. The use of the sampling frame would have offered the researcher access to the email addresses of the potential respondents. They could have used this to invite them to complete an online questionnaire rather than the face-to-face interviews approach they employed. With this, they could have avoided methodological problems with the race or gender of the enumerators, for instance, that are widely known in social research.
Even for a study based on non-probability sampling, there is a curious homogenisation of the BSA students at UCT-that they are all from impoverished backgrounds. A background claim to the study is that 'obviously... persisting inequalities in the school system make it less likely that they [BSA students] will meet the entrance requirements for science courses.' This clearly shows a shocking lack of appreciation for the diversity of the UCT BSA students and their school backgrounds.
The issues raised above immediately undermine the author's capacity to make any credible inferences about the study population, much less the study universe. Similarly, the study cannot make any inferences about BSA students at UCT (beyond those in the study sample), much less BSA students in the country. The title and conclusion of the 'commentary' make claims that cannot be supported by the study design-even the ones that supposedly repudiate race as a critical variable with predictive power (see further discussion below).
A study on why BSA students 'are less likely' to consider studying biological sciences is not the same as 'no BSA student' considered studying biological sciences. If the population of BSA students in UCT is 30 per cent but the share of BSA among those studying Biological Sciences was 35 per cent, the share of BSA students studying Biological Sciences would still be low relative to the other categories of students but would be higher than their overall share of the university student population. The author makes no effort to engage with this scenario.
For a study concerned with explaining why BSA students are less likely to study the Biological Sciences, a researcher would want to consider a sample stratified, at least, along the lines of those studying the Natural Sciences and those Further, what share of the surveyed BSA students (114 out of 211) is in the Humanities or the Natural Sciences, for instance, relative to the category 'Other students' would matter for the question that the author claims she sought to answer? Would a student's degree focus have some bearing on what they think of a question such as whether humans evolved from apes? Suppose a large share of the BSA students in the sample is registered in Theology, and such students consider the idea that humans evolved from apes absurd. Would that reflect their race classification or their disciplinary orientation? Would a BSA student studying Medicine or Zoology hold the same position?
For a sample that the author admits has a higher share of BSA students than the population, it is interesting that there is no attempt to allocate weight to the subcategories of the sample (BSA students and 'Other students') when reporting the findings. Even in the most rigorously designed probability sampling, sub-categories of the sample may be over-represented due to high non-response rates from other sub-categories of the sample. Researchers need not be held responsible for the non-response rate. What would be expected, however, is that the sample is properly weighted. In this case, the weight for BSA students would be 0.555 (30.0/54.0)-30 per cent being their share in the student population and 54 per cent, their share of the sample. Correspondingly, the weight for 'Other Students' would be 1.521. Even for the descriptive statistics, such weights matter.

Dissonance: The conclusion misreports the study result
While a poorly designed study may reflect limited methodological proficiency on the part of the researchers, misrepresenting the results of the survey is less easily dismissed. Nattrass acknowledges that the result of her 'exploratory regressions' is that attitudes are more important as predictors of the decision to study biological sciences, rather than 'the crude indicator of being a black South African' (cf. The pressing question to ask the author is this: if being a BSA student is less or not a predictor of whether one considered studying (zoology or) biological sciences, what is the purpose of the title of the paper? Given the low or no predictive power of race-category in the model, 'Are black South African students likely to consider studying biological sciences?' would have been a more appropriate title. In such a case, the answer would then have been in the negative: a definite 'No'.
The same diminished predictive power of the race-category of 'black South Africans' is evident in the regression model reported in

Race-based explanations by any means
Even with the low predictive power of racecategory in the model, lurking under the cover are race-based explanations. This plays out in two ways. The first concerns the pattern of attitudinal disposition ascribed to the BSA variable. The variable may not be a good predictor, but the attitudes that are the predictors are racially ascribed. The black South African students in the sample are presented as less likely to agree that humans evolved from apes than the 'Other students.' The BSA students are less likely to like having the redwing 'starlings around at UCT.' They are more likely to agree to the statement that 'I support wildlife conservation but have no interest in having a career in it.' All these are statistically significant in the Fisher's Exact Test results that the author presents. Race explanation remains; only that they are disguised as attitudes.
Despite the diminished (or non-existing) explanatory power of race-category in the model, which Nattrass conceded, nonetheless, she insists that: In short, the survey results suggest that black South African students are less likely to consider studying biological sciences than other students, and that this stance was linked primarily with career aspirations... and these were associated with materialist values and attitudes to local wildlife (p.13). No, they do not. Such 'result' is a thumb-suck. She suggests that BSA students are more likely to opt for degrees in accountancy and law because these are better paying. This is intended to underpin the claim of a materialist disposition among black South Africans. No evidence exists in the study to support this claim; no authority offered, and the study offers no such insight.
The correct, and ethically sound, interpretation of the regression statistics that Nattrass presents in Tables 2 and 3 is that regardless of the race categories of the respondents, the attitudinal variables are better predictors of whether a student considered studying conservation biology. But even with the descriptive statistics presented in Table 1, Nattrass ought to have known that Fisher's Exact Test results are for descriptive statistics. The results of the regression model that render the author's conclusions absurd are inferential statistics. Descriptive statistics merely describe the distribution of the sample (within a given confidence interval). Predictions and inferences are better, based on inferential statistics.
For a non-probability survey, Nattrass goes ahead to make generalisations that ignore the non-probability design of the study. A simple rule of quantitative research is this: never make claims about your research that it was never designed to carry. Nattrass' 'commentary' breached these simple rules of sample survey studies. She generalised from the sample not only to the study population (about BSA students at UCT) but to the study universe (about BSAs in South Africa beyond UCT).
Even so, these are all moot points. A poorly designed study will, in all probability, produce bad data. No amount of regression or other inferential statistics can fix that foundational problem. The aphorism, in statistical analysis, of 'garbage in, garbage out' would apply.
There are more minor issues to raise with the study. First, the study reduced 'biological sciences' to conservation biology. 'Biological sciences is the study of life and living organisms, their life cycles, adaptations and environment. There are many different areas of study under the umbrella of biological sciences, including biochemistry, microbiology and evolutionary biology.' 2 How many BSA students at the university and other universities in South Africa are studying other biological sciences, other than conservation biology and wildlife?
The second issue concerns data aggregation. From the reporting of the regressions for the composite index used in Table 3 of the 'commentary', one gets the impression that the questions in the survey instrument were in the form of a Likert scale. However, the reporting on Table 1 involves a binary or categorical 'Agree' or 'Disagree' rather than the fuzzy set disposition of a Likert scale. Otherwise, it is difficult to see why Nattrass reports Fisher's Exact Test results. This is not a pedantic concern since what it does is to elide the more nuanced differences between someone who indicates a preference for 'disagree' as against 'agree', compared to others who expressed a preference for 'strongly disagree' against those who chose 'strongly agree.' The 'distance' between the former respondents is much less than the distance between the latter respondents. I once heard a dean at a Faculty Board meeting say that as far as transformation is concerned, he is 'a minimum compliance person.' Is labelling Black (African) students going into the legal and accountancy profession materialistic driven by similar disposition? Is this the subversion, by other means, of a country's effort to overcome the prevailing legacy of its racist past? Consider a hypothetical situation where ALL 'black South African' students take to heart Nattrass' subliminal injunction not to be 'materialistic'-by not going into law and accountancy-would that not render permanent the apartheid footprint on the professions? As the saying goes, there are many ways to skin the cat called transformation.
In reporting the survey results, Nattrass' analysis takes a curious turn. She constructs what she calls 'an anti-conservation index' or a Fallist index! (More about this below). The 'anti-conservation index' is a composite measure drawn from three questions. The questions include whether national parks should be scrapped, to whether disciplines like conservation biology are colonial and should be scrapped, and whether the respondents like having the redwing starlings on the UCT campus.
Nattrass links the 'materialist index' to the World Values Survey. She claims that the twelve questions used in the World Values Survey as composites for the materialist index were included in the UCT survey. The distribution for the variables was not presented in the table that reports the descriptive statistics. Most significantly, the materialist index drawn from the World Values Survey has nothing to do with whether a respondent was studying accountancy or law. There is no evidence that the survey included a question of whether a student was studying accountancy or law. The 'anti-conservation index' and the 'materialist index' are attitudinal measures. These attitudes are held by BSA and OS respondents in the study. Yet, in the conclusion, Nattrass reverts to a student's choice of professions (law or accountancy?). 'Materialist values' are presented as 'a key determinant of not desiring a career in conservation' (p.13). 1 The conclusions drawn concerning 'materialist values of the BSA students is neither consistent nor derived from the study; it would seem that they derive from the author's predisposition rather than science.
In the regression analysis on Conservation is big business in South Africa. 9 In 2015 alone, the value of wildlife hunting value chain was R10.1 billion. 9 The hunting component of the value chain was valued at R5.1 billion. Trophy hunting of kudu alone was valued at R1.2 million, at R13 000 per head of the animal killed for trophy. 9 In the same year, a lion was sold for R230 000; the average price of a buffalo was R334 841 -eighty-four buf- The impression that emerges from the 'commentary' is of a researcher with a deepseated antipathy towards the Fallist movement. Such aversion may be legitimate as points of difference. It is problematic when it corrodes the scientific endeavour. The troubling part is that Nattrass seeks to render as irrational important conversations that the South African education system (not just the higher education sector) needs to have and act upon. It is legitimate to object to some of the methods employed in the campaigns that defined the protest movements without demonising the demands or rendering the demands themselves irrational. The Fallist index would seem more a product of prejudice than a legitimate effort in pursuing a scientific inquiry.
Further, one suspects a second layer of presupposition in the author's argument (possibly in the research instrument, as well): the false belief that 'Africans don't do conservation'; that conservation is alien to Africa. The larger argument is not so much about conservation, per se, but the modality of conservation. If you dispossess people of their lands and sources of livelihood to create a wildlife reserve for (European and American) tourists, those dispossessed have a right to question your idea of conservation. Thandika Mkandawire once referred to the 'eco-fascism' of those who demand nature reserves at the expense of the welfare of African people. 10 If you have no ontological link to such land dispossession, you would see the conservation area but not its origin and persisting consequences. This is a classic case of ontological disconnect-a disconnection from, and a lack of empathy for, the bearers of a collective memory of dispossession and who inhabit its aftermath. It is legitimate to argue that mass extinction of biological species, wanton depletion of wildlife, trophy hunting, and canned hunting are as colonial as one can imagine, and a marker of racial capitalism.

In lieu of a conclusion
The discussion above has sought to raise three distinct issues with Nattrass' commentary. First is the methodological deficiencies in the study design. Second is the dissonance between the results of the regression models in the commentary and the conclusions that Nattrass drew. The third is the extent of the corrosive effects of the author's presuppositions and prejudice on the premise and reporting of the study.