Analysis of NCEA results highlights arbitrary nature of exam results

An analysis by Auckland's MacLeans College of three years of NCEA results shows its faults not to be teething problems but endemic to the system.

MacLeans College principal Byron Bentley said after that three years of NCEA Level 1 and two years of Level 2 the variations in failure rates were still unacceptably large:

• In 2004, Level 1 failure rates for the External Achievement Standards ranged from 74.5 percent to 3.1 percent.
• In 2004, Level 2 failure rates for the External Achievement Standards ranged from 74.6 percent to 17.9 percent.
• There was little consistency in the distribution of Excellence and Not Achieved grades from 2002 to 2004 for the same Achievement Standards across a range of subjects.

"The analysis shows that the excuse that the unfairness to students of this inconsistent, arbitrary system is just 'teething problems' does not stack up when weighed against the evidence," Mr Bentley (pictured) said. The Level 1 and 2 results should have stabilised by now if the standards-based model was a viable one, he said.

In the high-stakes Level 3 qualification introduced for the first time in 2004, the analysis also highlighted a large variation in failure rates:

• The failure rate ranged from 86.6 percent to 10.6 percent.
• In 19 of the 85 standards over 60 percent of the students failed, but in 16 other standards the failure rate was less than 30 percent.

Level 3 Excellence grades (and Merit grades) were critical for students wishing to gain a place in select entry courses at university, as they carried a greater weighting than Achieved grades when universities calculated grade point totals.

There was a large variation in Excellence grades:

• The percentage of Excellence grades awarded by Achievement Standard ranged from 31.9 percent to 0 percent.
• Eight standards had over 10 percent of the candidates gaining Excellence.
• Seven standards had 0 percent of the candidates gaining Excellence (six of these were in Technology).
• The percentage of Excellence grades ranged from 14.8 percent to 0.4 percent by subject when aggregated.

Very few students in a large number of standards were achieving at the highest level:

• Eighteen of the 85 standards had less than one percent of the students gaining Excellence.
• Sixty of the 85 standards had less than five percent of the students gaining Excellence.

Many of the percentage variations were very large, meaning that thousands of students were either greatly advantaged or greatly disadvantaged depending on which year they sat a particular standard or group of standards.

Past research indicated that cohorts of NZ students did not vary greatly in ability across a range of learning areas from year to year but the variation in NCEA occurred in nearly every subject with no consistent pattern between subject or year.

The variation was therefore due to the unreliability of the NCEA model, Mr Bentley said.

"The NCEA system was untried, unproven and untested with a lack of credible research when implemented, and the shambles that the experts forecast has occurred.

"Unfairness to students is the result, matched by the debilitating effects of this on students, their parents and their hard-working and conscientious teachers. Drastic surgery is needed on the NCEA," Mr Bentley said.

He said the call from the president of the Auckland Secondary Schools Principals' Association for an "entirely independent, preferably overseas education team" to review the system should be strongly supported.

Meanwhile, the Post-Primary Teachers' Association has released a report calling for urgent reviews of NCEA's external moderation, management and communication. The report is the result of two months of focus groups of more than 100 teachers at nine schools late last year.

The Macleans College analysis.

The PPTA report: 'Teachers Talk About NCEA'.