Standardised tests aren't perfect but they do warn us there is something wrong in our school system. If we don’t like the results, we shouldn’t ignore the test data
Opinion: It has recently come to the public’s attention the Ministry of Education (MoE) and the New Zealand Qualifications Authority (NZQA) disagree why so few students are passing the recently created compulsory tests in reading, writing, and mathematics.
The squabble, as it has been reported in the media, is about whether the tests are the problem. The MoE suggests the tests may be too hard, poorly constructed, or badly marked. NZQA disagrees, and says it has extensive experience with crafting, administering, and marking high-quality statistically validated tests and examinations.
Few of us outside the MoE and NZQA have seen the tests so we cannot run an independent validity or reliability analysis, which might point out any deficiencies within the tests run by NZQA. However, it is possible the tests are a good measure of what students need to know at these critical junctures in their schooling – and there are clear deficiencies in achievement levels.
A key mechanism for determining whether the NZQA results are believable is whether they align with previous research into the performance of New Zealand students. If the results resemble other sources of data about student writing competence, then we have good grounds for thinking that their results are correct.
We are told by the NZQA that in a pilot run in the middle of 2022 with mostly year 10 students there was a pass rate of just 34 percent in writing. We don’t know if the test was calibrated to NCEA level 1, which is meant to be level 6 of the English curriculum. But assuming passing the test means that the students had met the minimum expectation for NCEA level 1, this could be seen as an astonishingly and disturbingly low rate.
READ MORE:
* Education that doesn’t add up
* There are no easy fixes for NZ’s growing truancy problem
Fortunately, there are some easily accessible sources of information against which we can evaluate its results. I managed the Assessment Tools for Teaching and Learning (asTTle) research and development team from 2000 to 2005. As part of that project, we tested about 20,000 students from years 5 to 12 on their ability to write, on-demand, an approximately 40-minute test which included a piece of writing linked to one of the major purposes for writing, that is persuade, instruct, narrate, describe, explain, recount and analyse.
Student writing was scored against the eight curriculum levels and given a numeric score. We broke the curriculum levels into three sub-levels – basic, proficient, advanced – to account for the time it took for students to progress through those levels.
Our results showed that at years 11 and 12, the sample we tested scored mostly in levels 3 and 4, with very few scoring above that. Their average score was 658, which equated to curriculum level 4 Basic. The average for year 8 students was level 3 Basic, and only 30 percent scored in or above curriculum level 4. This was quite different for reading comprehension and mathematics, where students scored much higher.
Much more recently the National Monitoring Survey of Student Achievement run by the University of Otago and the New Zealand Council for Educational Research reported on the performance of students in years 4 and 8 in writing as well as other aspects of the English curriculum. The 2019 report indicates only 35 percent of year 8 students scored at level 4 or above. That’s similar but not much better than the asTTle project reported in 2005, in which 30 percent scored at level 4 or above.
These two data points suggest the NZQA result is consistent with contemporary data from the 21st century concerning how well New Zealand students write – or, as the research suggests, how well they don’t. In general, our young people do not write well despite all our best efforts to improve the curriculum, the teaching of writing, and the teaching of future teachers. There are multiple potential causes for a lack of progress in writing. I will leave it to others to advance possible explanations. However, I am convinced that replication of previously reported results is a strong basis for thinking the tests are correct, and our children don’t know how to write.
Standardised tests may cause some students to be overly anxious, they may not reflect well the curriculum in its entirety and complexity, and they may be a poor tool for judging the value and quality of the school and its teachers. Nonetheless, they do act as an important canary in the mine warning us there is something wrong in our school system. If we don’t like the results, we shouldn’t ignore the test data. Instead, we need to consider what has gone wrong in how we prepare young people in this country to be proficient writers.
