Comparing mathematics knowledge of first-year students from three different school curricula

FUNDING: None Mathematics forms an integral part in the training of scientists and engineers. In recent history the South African school system has experienced several changes in school curricula. In 1994 the traditional knowledge-based curricula were replaced by an outcomes-based curriculum. Owing to implementation problems which resulted in resistance from teachers and the general public, revisions followed of which the National Curriculum Statement (NCS) and Curriculum Assessment Policy Statements had the most direct effect in terms of preparation for tertiary mathematics. We report here on an investigation of the basic mathematical knowledge of three student cohorts representing three curricula, namely the last cohort that received the traditional knowledge-based curriculum, and the first cohorts that received the two outcomes-based curricula. The results indicate that changes in the mathematical content of the curricula did not impact negatively on the basic mathematical knowledge of students enrolled for tertiary mainstream mathematics. The only exception is Euclidean geometry, for which certain topics were transferred to an optional paper in the NCS curriculum.


Introduction
Role players involved in the training of scientists and engineers have a vested interest in the basic mathematics knowledge of prospective science and engineering students.Changes in school curricula invariably influence the preparedness of students for tertiary studies, especially where mathematics-intensive programmes are concerned.Substantial changes were made to the South African school system since 1994 as a result of a change in the government system. 1 The racially and provincially segregated curricula of the apartheid era were replaced by a unified national curriculum.Whereas the previous fragmented curricula were mainly specified in terms of content knowledge to be learned using a transmission teaching model 2 , the new national curriculum was a skills-based constructivist curriculum 3 , which was implemented with an outcomes-based management structure 4 .Another major structural change to the curriculum was the merging of the higher grade (HG) and standard grade (SG) curricula documents into a single curriculum document, presumably to diminish the emphasis on individual achievements of a few in exchange for a more rounded approach to education for all learners.Owing to problems experienced with the implementation of the new curriculum, revisions were introduced.Not all of these revisions were sustained up to Grade 12, but, ultimately, three outbound Grade 12 cohorts can be distinguished in South Africa: (1) learners who matriculated in the years up to 2007 who were exposed in Grades 10 to 12 to traditional knowledge-based curricula (TKC); (2) matriculants of 2008 to 2013 who were exposed to a constructivist curriculum implemented through an outcomes-based educational system (OBE) as summarised in the National Curriculum Statement (NCS) 5 ; and (3) those who matriculated between 2014 and the present who experienced a revised version of the OBE curriculum, officially documented in the Curriculum Assessment Policy Statement (CAPS) 6 .
Since the replacement of traditional, knowledge-based curricula by new, OBE-grounded curricula in countries such as the USA, the United Kingdom, Canada, Australia, New Zealand and South Africa, discourses have developed among academics and the general public on the expediency of these changes.Perceptions were that students experienced difficulty with the transition to tertiary level, especially those enrolled in mathematics-intensive programmes.For example, a survey done in the USA revealed that an increasing number of incoming students needed remedial courses in mathematics. 7A report on the preparedness of Irish students for tertiary mathematics studies refers to 'grade depreciation', implying that grades achieved in state examinations were not comparable to the same grades obtained 10 years earlier. 8However, it was not made clear to which extent the perceived lack of mathematical knowledge could be attributed to the introduction of OBE.
0][11] In the study of Engelbrecht and Harding 9 , the first-year mathematics cohorts of 2005, 2006 and 2007 were compared using a mathematics achievement test which was designed to determine the level of mathematical competency of students who did not necessarily excel in the final matric exam.The questions were set on topics such as percentages, spatial geometry, parallel lines, word sums, ratio and proportion, number concept and manipulation, functions, graphs and trigonometry.The cohort of 2005 represented students who did not experience OBE at school level, while the cohorts of 2006 and 2007 experienced OBE up to Grade 9, but reverted back to traditional teaching in Grades 10 to 12. Engelbrecht and Harding 9 concluded that, except for geometry, word sums and ratio and proportion, the student performances were on par for these three cohorts.In 2015 we published an article 12 in which we compared the mathematical knowledge and skills of the 2008 cohort at our institution (the last cohort exposed to traditional curriculum in Grades 10 to 12) with the 2009 cohort (the first cohort exposed to the full OBE curricula to reach university) in terms of a framework consisting of three mathematics knowledge types, namely procedural knowledge, proceptual knowledge and conceptual knowledge.The framework was an attempt to investigate if the changes in the curricula impacted on the way mathematics was taught and learned in Grades 10 to 12.The sample was all students enrolled for mathematics, including students from science, engineering, commerce and education.The comparison showed that the performance of the OBE cohort was not as poor as has been perceived initially; however, the expected outcome that students from the OBE curriculum would have better conceptual understanding because of the emphasis on exploration and searching for patterns only materialised for some questions on the interpretation of graphs.
In the present article we follow up on these comparisons by including the cohort of 2015, which was the first cohort exposed to CAPS in Grades 10 to 12.The focus of this study is to investigate how changes in the school curricula, especially differences in the mathematical content, influenced the basic mathematical knowledge of natural science and engineering students enrolled at our institution.We report on a quantitative empirical study based on the results of multiple-choice tests written by all new first-year mathematics students at our tertiary institution.The main intention of the study was to investigate whether the mathematical knowledge of the cohorts of 2008, 2009 and 2015, which represent three different school curricula for Grades 10 to 12 -namely TKC, NCS and CAPS -are significantly different.

Changes to mathematics curricula in South Africa
In order to gain insight into the influence of each curriculum on the development of basic mathematical knowledge, we give a brief overview of the foundations of these three curricula and their historical implementation in South African schools.The management structure of the new curriculum was grounded in the principles of OBE, where assessment is based on 'demonstration of outcomes', ranging from 'simple discrete content skills' to the 'highly complex open-ended liferole performances required by adults in the real world' 4(p.25) .An OBE system places greater emphasis on dispositions, resulting in formative, criterion-based assessment, instead of summative assessment and high-risk tests. 13Three varieties of OBE with rising hierarchical orders of complexity 14 and corresponding decreasing scales of modernism 15 can be distinguished.Traditional OBE is similar to a traditional knowledgebased curriculum in terms of the organisation of learning content in disciplines and an emphasis on academic development, but it differs from the traditional knowledge-based curriculum in terms of its assessment criteria, which are based on mastery of specified outcomes, and on its learner-centred approach and emphasis on life-long learning.Transitional OBE is more future orientated and accentuates the cultivation of higher-order competencies, such as critical thinking, problem solving and communication skills.Transformational OBE is the most complex and extreme form of OBE which defies fixed curriculum outcomes based on conventional subject areas, and strives to change the disposition of learners.In the planning stages of the new South African curriculum, a transformational OBE model was envisaged. 16,17However, the first implemented version of OBE in South Africa -Curriculum 2005 18 was described by one researcher 19(p.22) as a 'potpourri of curriculum proposals with largely unacknowledged origins'.However, with the introduction of cross-curriculum critical outcomes, integrated learning areas and integrated real-life problem settings, Curriculum 2005 could be classified as a transitional model in the OBE hierarchy.
In practice, there was tension between the critical outcomes of Curriculum 2005 and the formulation of its learning outcomes. 15At that stage, most teachers were products of schooling in the old dispensation and struggled to come to terms with the implementation of the new curriculum. 20,21This difficulty resulted in revision of the original OBE curriculum in an attempt to reduce the level of integration of subjects.The Revised NCS for Grades R-9 22 was implemented in the foundation phase from 2004, and the NCS for Grades 10-12 5 was introduced in Grade 10 in 2006.These revisions did not address all the problems teachers experienced, as complaints about implementation issues and administration overload suffered by teachers persisted. 23Further revisions were necessary to address these problems, and, in 2012, new assessment criteria, referred to as the CAPS, were introduced in Grade 10. 6 In the revised document it was emphasised that the basic philosophy of the curriculum remained unchanged and that the adjustments only related to 'what to teach and not how to teach' 23 .The content organisation of the CAPS is similar to a traditional knowledgebased curriculum, but the critical cross-curriculum outcomes were retained.Some researchers labelled the CAPS curriculum as too prescriptive and restrictive 24,25 , arguing that the most dramatic change brought about by CAPS has been its shift in focus from assessment of learning to learning for assessment.Signs are already there that South Africa has moved away from OBE towards the US model of a standards approach to education, with an overemphasis on external assessment of learners in all the school phases. 26It is therefore not clear, and will be difficult to determine, whether the changes envisaged by the OBE curriclula had an effect on teachers' practices in their classrooms.
In this article we rather focus on the effect of changes in the structure and content of the curricula.Structurally, the differentiation between HG and SG mathematics has been removed, with the unintended consequence that fewer marks are available to differentiate among students who want to enrol for mainstream mathematics.In order to better prepare learners for their future role in society, themes of a statistical nature such as data handling, descriptive statistics and financial mathematics -were added to the core curriculum of mathematics.To make room for these inclusions, absolute value theory and the remainder theorem were omitted from the curriculum and Euclidean geometry was transferred to a separate optional paper, to which a few new topics such as recursive sequences, bivariate data and probability were added. 5One of the main consequences of this move was that students in the first-year cohorts of 2009 to 2014 were only partly exposed to Euclidean geometry in Grades 10 to 12. Furthermore, the scope of some topics was reduced by limiting assessment of formal proofs and definitions and the focus shifted to the application of rules and theorems in problem-solving situations.For example, the factor theorem was used to find the roots of higher-order polynomials, and the logarithmic rules were applied to solve for the time period of an investment.
The main objective of the OBE mathematics curriculum was to deliver learners who are able to 'transfer skills from familiar to unfamiliar situations' 5(p.5) .The emphasis shifted from knowing mathematics as facts, rules and principles, to the interpretation and application of conceptual representations such as graphs and algebraic patterns.Learners were encouraged to use graphical representations to solve problems and to search for connections between different topics.For example, in the topic of functions, more emphasis was placed on the connection between algebraic equations and the subsequent shifting of graphs.Another example is the topic of series and sequences in which the general recursive formulation in terms of geometric and arithmetic sequences were reformulated in terms of linear, quadratic and exponential sequences, which could be linked to functional graphs.
With the introduction of the CAPS, the content of the optional paper, which included topics of Euclidean geometry, was reintegrated into the core curriculum, while only the single topic of linear programming was omitted.These additions resulted in an overall increase in the mathematical content of the curriculum.In the CAPS document, the specific mathematical content to be learned was more clearly defined and examples were given for each specific curriculum statement.

Research question
The empirical study was undertaken to address the research question: Are there practical significant differences in the basic mathematical knowledge of first-year natural science and engineering students who matriculated from traditional knowledge-based curricula (TKC), the new outcomes-based curriculum (NCS), and the recently revised outcomesbased curriculum (CAPS)?

Research design
The research can be categorised as a quantitative investigation using a comparative analysis of variance (ANOVA)-test analysis of the results of a diagnostic test written by first-year mathematics students in order to quantify the differences in basic knowledge between students from three different school cohorts.

Sample
The sample was conveniently selected and consisted of all bona fide first-year students enrolled for mainstream mathematics in engineering and natural sciences programmes at our tertiary institution in 2008, 2009 and 2015.Only students who wrote the relevant matriculation exam in the previous year were considered in the analyses.For Grades 10 to 12, the 2008 cohort was exposed to traditional knowledge-based curricula (TKC), the 2009 cohort to the new outcomes-based curriculum (NCS), and the 2015 cohort to the recently revised outcomes-based curriculum (CAPS).The admission requirement for enrolment into mainstream mathematics modules for the 2008 cohort was 60% for SG or 50% for HG mathematics in the final matriculation examination.For both the 2009 and 2015 cohorts, the requirement was Level 4 (50%) for mathematics.The sample sizes for the respective cohorts were n=287 for the 2008 cohort, n=357 for the 2009 cohort and n=455 for the 2015 cohort.
The demography of students enrolled for 2008, 2009 and 2015 did not differ considerably.This similarity gave us an opportunity to investigate the effect of changes in the school curriculum on a relatively homogeneous sample of students regarding academic achievement at school level, but with exposure to different school curricula.

Ethical considerations and data collection
Permission was obtained from the director of the School for Computer, Mathematical and Statistical Sciences to conduct the research.Ethical clearance was obtained from the Research Ethics Committee of the Faculty of Natural Sciences (reference number NWU-0007-14-S3).The first-year cohorts of 2008 and 2009 wrote the diagnostic test in the first week of their respective first semesters.The participation was voluntary and individual results were not made public.Since 2011, all first-year students enrolling for mainstream mathematics on campus have been required to attend a mathematics refresher course before the commencement of classes.The first-year cohort of 2015 wrote the diagnostic test on the first day of the mathematics refresher course.Only data from bona fide first-year students who matriculated in South Africa from the relevant curricula were considered for the data analysis.

Selection of test items
The mathematical knowledge test used in the empirical study for the 2008 and 2009 cohorts consisted of 40 multiple-choice questions: 24 items were selected from a test developed to predict students' success in tertiary studies in mathematics 27 , and 16 items extracted from previous question papers were added to the core test to broaden its scope in terms of specific topics in mathematics.The 2015 cohort wrote the pre-test of the mathematics refresher course; this test consisted of 35 questions.A total of 25 items appeared in both tests and could be used in the comparative analysis.

Classification of test items in terms of topic areas
The 25 items were grouped into different topics of mathematics, namely algebraic knowledge, functions and graphs, trigonometry, geometry and differentiation.Because of the multidimensionality of mathematical knowledge it is not always possible to classify certain items.For example, the simplification of can be classified as algebraic knowledge, but it also requires an understanding of the function concept.For this reason, the items were compared on an individual basis.

Comparison criteria
The differences between the means of the individual questions of the three cohorts were compared using the ANOVA test for Cohen's effect sizes, as given by Ellis and Steyn 28(p.51) .Effect sizes (d-values) are independent of the sample size and provide a measure of the practical significance of the differences between the means.According to Cohen, d-values are interpreted as follows: • Small effect: • Medium effect: Large effect: Although the sample used is a convenience sample, p-values giving the statistical significance of the differences between the cohorts are reported for the sake of completeness.A p-value smaller than 0.05 indicates that the difference between the cohorts is statistically significant.

Results
The results are presented in Tables 1

Algebraic knowledge
The results for algebraic knowledge are listed in Table 1.The cohorts all performed well (>55%) on most of the questions, with the exception of Question 4 (<51%) on simplifying a surd and Question 25 (<36%) on applying rules of logarithms.When comparing d-values for the rest of the questions, five other questions (Q1, Q3, Q13, Q16 and Q17) yielded small (*) to medium (**) practical differences between cohorts.The TKC cohort performed slightly better than the other two cohorts in solving general linear equations (Q1), and when applying long division to determine the remainder of a polynomial (Q16); better than the NCS cohort when simplifying fractions (Q3), and better than the CAPS cohort when solving fractional equations (Q13).Questions 2 and 17 are both set on the application of exponential rules.Question 2 was a direct application of the rules where the expression (2x 3 ) -2 had to be simplified, whereas in Question 17 students first had to use higher-order thinking skills to analyse the question before applying the appropriate rule.The question stated that they had to determine a third of 3 15 .This higherorder thinking could explain the lower performance of all cohorts on this question in relation to Question 2. The CAPS cohort performed best in both these questions.

Functions and graphs
The results for functions and graphs are given in Table 2.The two questions on the quadratic function (Q6 and Q21) were answered well by all the cohorts (>68%), with the NCS group receiving the lowest scores.The CAPS cohort performed the best in determining the inverse of a function using algebraic manipulation (Q19), while the NCS cohort performed best in the question on linking a graph of an absolute value function to the given equations (Q9).All cohorts performed poorly in linking an inequality to its graphical representation (Q32), with the CAPS cohort performing the worst.

Trigonometry
The results for trigonometry are given in Table 3.The questions on trigonometry tested the application of right-angled trigonometry (Q12) and finding the period from trigonometric equations (Q27).There was no practically significant difference in performance on the question on right-angled trigonometry (Q12).The CAPS cohort performed best in the question on the period of the tangent function (Q27).

Geometry
The results for geometry are given in Table 4.For the items listed in Table 4, the students had to apply the named theorem to perform numerical tasks.In the geometry section, the performance of the NCS cohort was the poorest.The practical difference between the scores for the NCS and the other two cohorts, for the questions on angles in the same segment (Q29), angles in the centre and circumference (Q30), and a line perpendicular to a chord (Q31), ranged from a small (0.40) to medium (0.60) effect.In the question on the theorem of similarity of triangles (Q35), the CAPS cohort performed best with a small effect (0.27) in practice.

Differentiation
The results for differentiation are given in Table 5.The questions on differentiation included two questions on limits (Q26 and Q33) and two questions on application of differentiation (Q24 and Q34) (Table 5).
Although determination of maxima and minima of parabola is included in all curricula, all cohorts performed poorly on the question based on the maximum of a horizontally shifted parabola (Q24).The TKC cohort performed best in determining the limit of a hyperbole (Q26), and the CAPS cohort performed best in the question based on the interpretation of the derivative as the slope of a tangent line (Q34).

Overall performance
The results for the average performance on all the questions are given in Table 6.There are two sets of comparisons: one including geometry results and one without the geometry results.The former shows a decline in the overall performance of the NCS curriculum with a medium effect of 0.56 with respect to the TKC curriculum, and of 0.55 with respect to the CAPS curriculum.When disregarding the effect of the shift of Euclidean geometry to an optional paper in NCS curriculum, the effect becomes smaller, namely 0.38 and 0.29, respectively.

Discussion of results
The statistical comparison of responses to the individual questions in general did not yield considerable differences for the three cohorts.
Responses for some questions were statistically significantly different according to the p-values, but most of these differences were not significant in practice according to the d-values.There was a slight drop in performance for the NCS cohort in relation to the TKC and CAPS cohorts, but the practical difference was small.The comparison of the questions testing algebraic knowledge showed small differences in practice for questions set on application of exponential rules, solving linear equations, working with different forms of quadratic equations, and solving and simplifying fractional equations and expressions, respectively.This was also the case for functions and graphs, trigonometry and differentiation.Noticeable exceptions were the questions set on the topic of geometry (with the NCS cohort performing poorest), the question on finding the remainder after division (with the TKC cohort performing best), and the question on finding the inverse of a function (with the CAPS cohort performing best).The poor performance of the NCS cohort in the geometry questions can be attributed to the transfer of some topics of Euclidean geometry to an optional paper.In our sample, only 35% of the students of the 2009 NCS cohort wrote the optional paper in the NCS examination of 2008.In the NCS curriculum, the remainder theorem was omitted and students from this cohort had to rely on the algorithm for long division to find the remainder.Although the remainder and factor theorems are briefly mentioned in the CAPS curriculum, this cohort also struggled more than the students from TKC with the application of the remainder theorem.The high score of the CAPS cohort in the question on finding an inverse function with algebraic manipulation is difficult to explain, as finding an inverse function is included in all three curricula.
There were other notable results obtained for some questions, albeit not in terms of a practical difference in results.Although the absolute value function was omitted in the NCS and CAPS curricula, students from these cohorts could apply their knowledge of functions to identify the correct shifted graph of the given absolute value function and surprisingly the NCS cohort obtained the highest score for this question.Despite a scaling down in the rules of logarithms in the NCS and CAPS curricula, all three cohorts performed poorly in the question on logarithms.A low score was also obtained for the question on the simplification of a surd.A common denominator of these two questions is knowledge about algebra of functions, namely that in general f(x+a) f(x)+f(a) or that f(xa) a f(x).All three cohorts failed to intuitively identify the maximum of a quadratic function of the form y=-(x-2) 2 as zero.All these questions required a conceptual understanding of the mathematics involved and in spite of the intention of the OBE curricula to foster higher-order cognitive skills, all the cohorts performed poorly on these type of questions.
The comparison of the total scores of the three cohorts indicates a lower performance of the NCS cohort relative to the TKC and CAPS cohorts.This can partly be attributed to the omission of Euclidian geometry from the NCS curriculum, and the reintegration of geometry in the core curriculum of the CAPS curriculum.After the exclusion of the geometry results from the overall scores, the NCS cohort still performed slightly poorer.

Conclusions and recommendations
In this study, the mathematical knowledge which is mostly learned in Grades 10 to 12, and which we presume will have the greatest impact on success in tertiary mainstream mathematics, was compared for three cohorts representing three different exit-level school curricula.The comparisons of the overall results and the results of individual questions, which reflect topics of basic mathematical knowledge of Grades 10 to 12, show that in general there was little or no difference in practice for these cohorts.The only mentionable difference was in the domain of Euclidean geometry, in which the NCS cohort performed poorer, which can be directly attributed to the transfer of some topics of Euclidean geometry to an optional paper.The results signal that the omission of certain basic topics can be detrimental to the preparation of learners for tertiary studies where knowledge of these mathematical topics is important.If learners were not exposed to the gradual build-up of basic knowledge of a domain, it would be difficult to remedy the situation within a short duration.Developers of school mathematics curricula should be sensitive to the requirements of tertiary educational institutions regarding the basic mathematical knowledge needed by natural science and engineering students.
The introduction of new outcomes-based curricula led to the perception that students from these curricula enrol at tertiary institutions with poorer basic mathematical knowledge than those from traditional knowledgebased curricula.The results of this study indicate that this perception is not necessarily true.The samples in our study were fairly homogeneous in terms of demography, schooling and selection criteria, and the main difference was in their exposure to different school curricula.We suggest that other factors, such as school management or general societal changes or technological innovations, should be considered as an explanation for so-called grade depreciation.Finally, we want to point out that the results of the study do not reflect on changes envisaged by OBE curricula regarding teaching practices or the development of higher-order thinking skills.More qualitative studies will be needed to investigate these factors.
to 6 for the different topic areas.The abbreviations 'T', 'N' and 'C' indicate the cohorts from the TKC (2008 cohort), NCS (2009 cohort) and CAPS (2015 cohort) curricula, respectively.The abbreviation 'T/N', for example, indicates the practical difference between the scores for the TKC and NCS curricula.An asterisk in the p-column indicates an acceptable statistically significant difference (p<0.05), and asterisks in the d-column indicate a small (*), medium (**) or large (***) effect in practical difference.A positive d-value in a column indicates that the group mentioned first performed better, while a negative d-value indicates a better performance by the group mentioned second.

Table 1 :
Question by question comparison for algebraic skills Note: An asterisk in the p-column indicates an acceptable statistically significant difference (p<0.05), and asterisks in the d-column indicate a small (*) or medium (**) effect in practical difference.A positive d-value in a column indicates that the group mentioned first performed better, while a negative d-value indicates a better performance by the group mentioned second.

Table 2 :
Question by question comparison for functions and graphs Note: An asterisk in the p-column indicates an acceptable statistically significant difference (p<0.05), and an asterisk (*) in the d-column indicates a small effect in practical difference.A positive d-value in a column indicates that the group mentioned first performed better, while a negative d-value indicates a better performance by the group mentioned second.

Table 3 :
Question by question comparison for trigonometry Note: An asterisk in the p-column indicates an acceptable statistically significant difference (p<0.05), and an asterisk (*) in the d-column indicates a small effect in practical difference.A positive d-value in a column indicates that the group mentioned first performed better, while a negative d-value indicates a better performance by the group mentioned second.https://doi.org/10.17159/sajs.2019/4652

Table 4 :
Question by question comparison for geometry An asterisk in the p-column indicates an acceptable statistically significant difference (p<0.05), and asterisks in the d-column indicate a small (*) or medium (**) effect in practical difference.A positive d-value in a column indicates that the group mentioned first performed better, while a negative d-value indicates a better performance by the group mentioned second.

Table 5 :
Question by question comparison for differentiation An asterisk in the p-column indicates an acceptable statistically significant difference (p<0.05), and an asterisk (*) in the d-column indicates a small effect in practical difference.A positive d-value in a column indicates that the group mentioned first performed better, while a negative d-value indicates a better performance by the group mentioned second.

Table 6 :
Comparison on overall performance Note: An asterisk in the p-column indicates an acceptable statistically significant difference (p<0.05), and asterisks in the d-column indicate a small (*) or medium (**) effect in practical difference.A positive d-value in a column indicates that the group mentioned first performed better, while a negative d-value indicates a better performance by the group mentioned second.