Standardisation arrangements for summer 2020 in Wales
Following the Minister for Education’s announcement to close schools and colleges and cancel the 2020 summer exam series to help fight the spread of COVID-19, we have worked to develop a process that fairly recognises learners’ work and makes sure they get their grades in time to progress.
In May, we consulted on introducing aims to underpin the statistical standardisation model for the issuing of grades for the summer 2020 exam series, and the responses showed that we had support for these aims.
For this summer’s awards, schools, colleges and other exam centres have been asked to provide centre assessment grades (CAGs) and a rank order for their learners. This has never been done before, but the unprecedented circumstances meant taking extraordinary action.
It’s important that the grades awarded in August are as fair as possible, and that there is consistency across centres and across years. For this reason, a standardisation process will be applied to the information provided by centres. This is to ensure that learners across Wales are treated in the fairest possible way.
Who is responsible for what this summer?
We have worked with Welsh Government and WJEC to help learners get their grades in GCSE, AS, A Level and Skills Challenge Certificate qualifications in summer 2020. We each have a different, but complementary, role to play this summer.
Welsh Government’s role is to set education policy and oversee the education system, providing policy direction and guidance. Through engagement with key stakeholders, it promotes confidence in, and the credibility of, the qualifications system.
Qualifications Wales regulates qualifications in Wales. We have put regulations in place to allow grades to be awarded this summer. We have approved the standardisation models developed by WJEC and monitored its processes for determining the grades awarded. Before results are issued, we will review them with WJEC to ensure that national outcomes are broadly similar to previous years, in order to protect qualification standards and support fairness for learners, past, present and future.
WJEC’s role is to provide approved qualifications for learners in Wales. Every summer, it awards grades through established methods. For this summer, it has collected CAGs from centres and developed and implemented the standardisation approach to calculate learners’ qualification grades from a range of evidence, including the information provided by centres.
Why do we need to use a process of standardisation?
Centres were asked to make fair, objective and carefully considered judgements of the grades their learners would have been most likely to achieve if they had sat their exams, considering the full range of available evidence. Our analysis shows the CAGs are generous and that, on average, centres have submitted higher than expected grades. We have produced a set of slides on our CAG analysis.
At AS and A level, our analysis shows that the CAGs would produce atypically high outcomes not seen in the past ten years for the higher grades and a typically low outcomes for the lower grades.
At A level, the difference in the percentage of learners given an A* CAG compared to those awarded an A* in 2019 is 6.3 percentage points (pp) and for A is 7.1pp. So overall, had the CAGs been awarded this year for A level, 40.4% would have had a grade A* or A, compared to 27.0% in 2019 – a difference of 13.4pp.
At AS, there is a similar difference. If the CAGs had been awarded this year for AS, 29.3% would have had a grade A, compared to 20.3% in 2019 – a difference of 9.0pp.
There is a similar picture at GCSE. If the CAGs had been awarded this year for GCSE, 24.5% would have had a grade A* or A, compared to 17.9% in 2019 – a difference of 6.6pp. And 73.4% would have had an A* to C grade, compared to 62.4% in 2019 – a difference of 11.0pp.
Given the exceptional circumstances and pressure that exam centres were operating under this summer, it is not surprising that the CAGs they submitted were optimistic; and similar CAG inflation that has been found in England. But this clear difference between this year’s CAGs and exam results in previous years highlights the need for a standardisation process to secure fairness for learners.
The inconsistency and overall generosity in the CAGs means that after standardisation:
- a small percentage of learners will receive a grade that is higher than their CAG;
- a majority of learners will receive a grade that is the same as their CAG;
- the remainder will receive a grade that is lower than their CAG. A small percentage of calculated grades will be two or more grades lower than the CAGs.
How will the standardisation work in Wales?
Centres provided CAGs and a rank order for each learner, for each subject, to WJEC. This will be standardised to improve consistency in grades, and to avoid extreme under or over estimations.
Step 1: WJEC calculates a set of grades for each subject in each centre.
Step 2: WJEC allocates these grades to learners using the rank order provided by centres.
In Wales, GCSE, AS and A level and Skills Challenge Certificate qualifications have different designs and different structures. Some have units that are typically sat earlier in the course (unitised qualifications), and others are designed so that units are all sat at the end of the course, in the same exam series (linear qualifications).
For unitised qualifications  where there is performance in earlier units (which provides consistent evidence of prior attainment across most of the cohort in the same qualification) and where these units make up a substantial part of the course, Model 1 will be used. Model 1 will be used for A levels, GCSEs in biology, chemistry, physics, double award science and English literature and Skills Challenge Certificate qualifications. Model 1 calculates a set of grades for the centre based on learners’ performance in previous units of the qualification. Model 1 is only used in Wales and Northern Ireland.
For linear qualifications  and unitised qualifications where there is insufficient evidence from units already taken or where the pattern of taking units earlier in the course is inconsistent, Model 2 is used. Model 2 will be used for AS levels and most GCSEs. Model 2 calculates a set of grades for the centre based on its previous performance, taking into account learners’ prior attainment. For AS levels, evidence of prior attainment comes from average performance in GCSEs for those learners. For GCSEs, evidence of prior attainment comes from learners’ performance in the key stage 3 national tests and teacher assessments for those learners.
 Unitised qualifications are designed so that assessment units can be taken by learners at different points during the course, with a final grade ‘cashed-in’ at the end of the programme of learning once a learner has completed all units.
 Linear GCSEs are designed so that all assessment in a subject is completed at the end of the programme of learning.
Will missing data and small subject entries be accommodated by the models?
Both models being used to standardise this summer will accommodate learners and centres with missing data. For a centre where there is no prior unit performance data available to be used in Model 1, the process will move to using Model 2 for that centre. For a centre where there is limited prior performance data available to be used in Model 1, the rank order of learners will be used to ‘slot in’ missing grades.
The size of the subject entry in a centre does not affect Model 1. Model 2 is designed to accommodate centres with small subject entries by balancing the weight given to the statistical evidence with the CAG evidence. There are also rules in place for Model 2 to accommodate centres with no historical performance evidence in the qualification.
What effect does standardisation have on learners with protected characteristics?
One of our aims for this summer was to ensure that groups of learners are not disadvantaged relative to previous years. There are many different factors that can contribute to differences in results across groups of learners (with unconscious bias being one of them) so it is important that the analysis and interpretation of these differences is done carefully. Model 1 has been developed by WJEC and undergone testing on historic data; as part of this process WJEC analysed differences by age and gender and found no evidence of systematic bias. Model 2 has been used across all jurisdictions and undergone testing on historic data that shows that there is no evidence that the model will disadvantage groups of learners. Given that the KS3 data used in the GCSE standardisation model has not been previously used in this way, we have performed our own analysis to determine if there was any evidence of bias and again found none.
Despite this rigorous testing there is still the possibility of bias coming through in the final grades, so we have monitored gender gaps at the qualification level (i.e. across all A levels, etc.) before signing off the awards. We will publish an in-depth equalities analysis after the summer once more detailed data is available.
Are the standardisation approaches the same across Wales, England and Northern Ireland for GCSEs and A levels?
Although the standardisation methods considered across England, Northern Ireland and Wales are similar, differences in education systems and qualification designs mean that the data available to calculate grades varies and the approaches reflect that. For example, in Wales and Northern Ireland, AS results can be used in the standardisation model as evidence of prior attainment because the AS and A level are coupled – the AS contributes to the A level. This also applies in Northern Ireland but does not apply in England. Further information on the England and Northern Ireland standardisation models can be found on the Ofqual and CCEA Regulation websites.
A PowerPoint presentation summarising Centre Assessment Grades