The New Zealand
Qualifications Framework
Prepared by Alan Smithers
Centre for Education and Employment Research, Brunel University
for the Education Forum
November 1997
First published in 1997
by The Education Forum,
PO Box 38218, Howick, Auckland, New Zealand
ISBN 0-9583540-3-0
© Text: as acknowledged
© Edition: Education Forum
Printed by Astra DPS, Wellington
AUTHOR AND ACKNOWLEDGEMENTS
Professor Alan Smithers BSc MSc MEd PhD (Plant Physiology) PhD (Education) CPsychol
FSRHE is Director of the Centre for Education and Employment Research (CEER)
which in 1996 moved from the University of Manchester to Brunel University in
London. CEER undertakes research for industry, government departments and other
bodies, and receives grants from research councils and charitable organisations.
It specialises in providing accessible numerical accounts of education, particularly
in relation to employment. Recent projects have included: vocational education
and qualifications, teacher supply, education 16-19, graduate employment, flows
into science and engineering, technology in schools, single sex education, further
and higher education, and industry and education links.
Professor Smithers is currently Professor of Education (Policy Research) at
Brunel University. Before that he was Professor of Education at the University
of Manchester from 1977-96. He was originally a research scientist and lecturer
in plant physiology at Birkbeck College, University of London. He subsequently
retrained as a psychologist while a Research Fellow at Bradford University where
he became a senior lecturer in education.
In 1986 Professor Smithers was elected as one of the first three Fellows of
the Society for Research into Higher Education. He has been a member of the
National Curriculum Council and the Beaumont Committee on Vocational Qualifications.
He has served on committees of The Royal Society, the Engineering Council, and
the Business and Technology Education Council. He is currently Special Adviser
to the House of Commons Education and Employment Committee.
Professor Smithers has published a number of influential books and reports,
and over a hundred papers in biology, psychology and education. His research
often leads him to be asked to comment on the educational issues of the day.
Professor Smithers wishes to acknowledge with thanks the assistance of Douglas
Blackmur, Peter Coolbear, Terry Crooks, Jim Doyle, Greg Dwyer, Warwick Elley,
Don Griffin, Michael Irwin, Roger Kerr, David Lythe and Mike Woods for providing
many valuable comments and suggestions on the various drafts of this report.
Dr Pamela Robinson gave unstinting support and advice, and Mandy-Diana Coughlan's
information-technology skills helped to make this publication possible.
The interpretations, conclusions and recommendations in this report are solely
those of the author and should not be ascribed to any whose assistance is acknowledged
above.
CONTENTS
PAGE
FOREWORD BY PHILIP RAFFILLS ix
EXECUTIVE SUMMARY xiii
Chapter 1 INTRODUCTION 1
Chapter 2 THE EMERGENCE OF NEW ZEALAND'S NEW APPROACH
3
Chapter 3 THE QUALIFICATIONS FRAMEWORK 7
Chapter 4 IMPLEMENTATION 11
4.1 Creating the structure 11
4.2 Participation 13
4.3 Performance 14
4.4 Qualifications awarded 16
4.5 Qualifications outside the NQF 17
4.6 Funding 18
4.7 Overview 19
Chapter 5 THE NQF AND SCHOOLS 21
Chapter 6 THE NQF AND DEGREES 25
6.1 Universities 25
6.2 Polytechnics 26
Chapter 7 THE NQF AND INDUSTRY 29
Chapter 8 BRITAIN'S EXPERIENCE WITH VOCATIONAL QUALIFICATIONS
33
8.1 Criticisms 33
8.1.1 Precision 34
8.1.2 Fragmentation 34
8.1.3 Prioritisation 34
8.1.4 Knowledge and understanding 35
8.1.5 Assessment 35
8.1.6 Output-related funding 36
8.2 Beaumont Review 36
8.3 Qualifications and Curriculum Authority 37
8.4 The present situation 37
8.5 Relevance to New Zealand 38
Chapter 9 CRITIQUE 39
9.1 Can learning always be expressed appropriately as outcomes?
39
9.2 Can standards be expressed unambiguously? 40
9.3 Can the candidate's own teachers make dispassionate judgements?
43
9.4 Does the aggregation of components amount to overall capability?
43
9.5 Can levels and credits be assigned meaningfully? 44
9.6 Is the split between 'competent/not yet competent' sufficient?
45
9.7 Can a highly fragmented system be manageable and affordable?
46
9.8 Can teachers cope with the workload? 46
9.9 Justifications for change 46
9.9.1 Academic-vocational divide 46
9.9.2 Norm-referencing 48
9.9.3 Scaling 49
9.9.4 Partial learning 49
9.10 Conclusion 50
Chapter 10 THE GREEN PAPERS 51
10.1 Green Paper on qualifications 52
10.1.1 Outcome 53
10.1.2 Level and credit 54
10.1.3 Quality 54
10.2 Green Paper on tertiary education 55
10.3 Is a quality threshold feasible? 56
10.4 Governance of qualifications 57
10.4.1 Qualifications authority 57
10.4.2 Approval agencies 57
10.4.3 Developers and providers 58
10.5 Operation of a quality threshold for programmes
and providers
58
10.6 Conclusion 59
Chapter 11 THE WAY FORWARD 61
11.1 Is a qualifications framework needed? 61
11.2 Qualifications reform 63
11.2.1 Schools 63
11.2.2 Vocational education and training 64
11.2.3 Higher education 65
11.3 Designing a qualifications structure 66
11.3.1 Schools qualifications 66
11.3.2 Occupational qualifications 68
11.3.3 Advanced qualifications 69
11.4 Qualifications network 70
11.5 Format of qualifications 71
11.6 Registration of providers 73
11.7 Qualifications in the senior secondary school 73
11.8 Costs 77
11.9 Recommendations 77
11.10 Conclusion 80
NOTES 81
FOREWORD
It has been evident for several years that the National Qualifications Framework
(NQF) is not performing to its original high expectations. Several of the Education
Forum's publications have identified serious problems with some of the key assumptions
on which the Framework is based. In view of these problems, the Education Forum
welcomed the government's announcement last year of a review of the NQF, since
this clearly indicated that the government also recognised that all was not
well and that some rethinking was required.
We also welcomed the opportunity to contribute to the review of the NQF by way of a submission on the government's Green Paper A Future Qualifications Policy for New Zealand: A Plan for the National Qualifications Framework. We sought expert assistance and invited Professor Alan Smithers to prepare a report for us. Professor Smithers was Professor of Education at the University of Manchester prior to his current appointment as Professor of Education (Policy Research) at Brunel University, London. He has written widely on education and employment policy issues and has recently been appointed as Special Adviser to the House of Commons Education and Employment Committee. He has contributed to the policy debate on the United Kingdom's vocational qualifications which, importantly from a New Zealand perspective, have much in common in terms of structure and underlying philosophy with our unit standards.
Professor Smithers visited New Zealand early in 1997 and discussed the NQF with a wide range of people in the education and employment sectors. His report went through several stages during which significant feedback was provided by New Zealand academics, education officials and others. In some important respects, Professor Smithers' recommendations take us back to those of the 1988 report of the Working Group on Post Compulsory Education and Training convened by Professor Gary Hawke which was the starting point for the NQF before it took on its much more ambitious role.
Professor Smithers' report formed the basis of the Education Forum's submission to the government on the Green Paper. We consider that his analysis of the situation is robust; and we concur with the recommendations. It is with pleasure, therefore, that we make it available to a wider audience through publication.
The history of the NQF hitherto has been the search for a single concept that would give coherence to a comprehensive qualifications framework and at the same time transform education and training. I think it has become widely acknowledged that unit standards cannot achieve this. The danger now is that reliance will be put on the concept of quality, in particular a 'quality threshold', to provide what unit standards have failed to deliver. Needless to say, everyone can be in favour of quality; it is an attractive notion, but that in itself does not guarantee that it is going to be any more successful as a unifying concept and transforming agent than unit standards.
The more productive approach is to work through the issues from a first principles basis recognising that qualifications vary considerably in their purpose, in the content to be covered and in assessment methodologies. If we seek to impose on all qualifications a similar structure we will end up with serious distortions and increasing costs as layer upon layer of quality controls procedures, such as moderation, are put in place to ameliorate the effects.
To avoid educational deformations and excessive costs, we must start with the curriculum content, then decide on the appropriate assessment methodology, and only then determine what linkages are feasible between the qualifications. The result will be, to use Professor Smithers' words, more of a 'network' or a 'map' with a variety of connections than a rigid framework. It will be less ambitious than the comprehensive, seamless qualifications framework that has been sought hitherto. But it will be educationally sound, flexible, open to innovation and hence enduring.
Professor Smithers' report leaves some detailed work to be undertaken. For example, more work needs to be done in the area of senior secondary school qualifications, though his report does very clearly and helpfully point the way ahead. Further, his report outlines new administrative arrangements which will need to be developed. These take account of the diversity of qualifications and the need to separate out various functions, for example those of setting the content to be covered and of designing the assessment.
Professor Smithers' report makes it clear that no single remedy such as the establishment of a 'quality threshold' is going to resolve the multiple problems that have accumulated over several years and which we now face. To achieve an enduring, educationally sound qualifications system, there is no alternative to identifying carefully where we are, analysing where we have gone wrong, establishing sound principles on which to proceed, and restructuring the system accordingly. The Education Forum believes Professor Smithers' report addresses all these issues and points the way forward. I strongly commend it.
Philip Raffills
Chairman, Education Forum
EXECUTIVE SUMMARY
The recent publication of two Green Papers by the New Zealand government, one
specifically concerned with qualifications and the other considering them in
the context of tertiary education, suggests that the government is not entirely
happy with the progress of qualifications reform. In this report, we review
how the qualifications structure came to be as it is (Chapters 2 and 3), report
on its implementation (Chapter 4), consider the experiences of schools, universities
and industry as they have sought to come to terms with it (Chapters 5, 6, and
7), and summarise how similar arrangements have been received in the United
Kingdom (Chapter 8). A conceptual analysis of the present approach (Chapter
9) and a critique of the Green Papers (Chapter 10) are offered. It is concluded
that the basic mistake New Zealand is making is to search in vain for some magic
formula (the philosopher's stone comes to mind) - first, 'unit standards', now
'quality threshold' - which would transform its entire education and training
system at a stroke.
In fact, neither 'unit standard' nor 'quality threshold' can achieve what is asked of it. The construct of 'unit standard' has many weaknesses (Chapters 3 and 9). But the crucial one is that it is not possible to state the standards with sufficient precision to ensure that they unambiguously convey what a qualification is about. Thus if it is the only information to pass to teachers and assessors there will be major inconsistencies. Attempts to rectify this through moderation lead to an insupportable workload. In order to specify a qualification clearly, information needs to be provided about at least three things - purposes, content and assessment. The idea of unit is separable from that of standard. While it can sometimes be an advantage for a qualification to consist of components, they too need to be expressed in terms of purposes, content and assessment.
Between the two Green Papers there is a shift in emphasis to 'quality threshold', but this is no more of a panacea than 'unit standard'. Quality is an elusive notion which is, in large part, tacit in the sense that it cannot easily be articulated (Chapter 10). In their struggle to define quality the Green Papers invoke no fewer than 39 extrinsic attributes without ever pinning it down satisfactorily. If there is ever an attempt to implement the 'quality threshold', there will thus be major inconsistencies at first but one version is likely to harden up through case law into a rigid bureaucracy.
Rather than seeking some transforming principle, the New Zealand government, the report argues, should be carefully working through a number of issues from first principles. The report identifies (Chapter 11) three clusters:
o the failure of schools qualifications to keep pace with the wider range of abilities and aspirations now represented in the senior secondary school;
o the need to have a first-class system of employment-related qualifications covering both preparation for work and upskilling in work; and
o the desirability of opening up tertiary education by allowing degrees to be offered by providers other than universities.
Conceivably it could be left to the market to provide solutions, but the report favours an appropriate balance between regulation and competition. It takes the view that the New Zealand government of the day was wrong to allow the recommendation of the Hawke Committee for three distinct sub-agencies co-ordinated by a Qualifications Authority to lead to the establishment of an all-powerful qualifications monopoly which has sought to impose the same solution across a wide variety of situations. Many of the present difficulties stem from that decision.
The report proposes reactivating and strengthening Hawke's ideas. It is suggested that each of the three areas identified above should be the responsibility of a separate body - called respectively the Schools, Occupational, and Advanced Qualifications Boards - with their activities co-ordinated by a Qualifications Co-ordinating Council. This would represent a significant change in the balance of power from that which exists at present, especially since in 1991 the NZQA disbanded its standing committees, including those for secondary qualifications, vocational qualifications and academic qualifications. While there will be those who will groan at the thought of a multiplication of government bodies, there is every prospect that, being more focused, they will be leaner and fitter in total than the present unwieldy New Zealand Qualifications Authority. Each would be free to pursue the approaches it found most suitable, working to a common qualifications format of purposes, content and assessment.
In deciding the content of qualifications the Boards would be advised by specialist groups, but the details would differ from domain to domain - which is one of the reasons for proposing separate Boards. In the case of the Schools Qualifications Board it is envisaged that subject groups would be convened from among those with particular expertise. In order to avoid confusion of responsibility, it is suggested that the national curriculum as determined by the Ministry of Education should run to the end of year 10 (Form 4). In the senior secondary school (years 11-13) qualification prescriptions would be the basis of the curriculum. To ensure a smooth transition from lower to upper secondary school the Ministry should be represented on the subject groups.
The domain of the Occupational Qualifications Board would be much larger and more varied than that of the Schools Qualifications Board and setting the requirements for qualifications would be a much bigger task. It is suggested that this should be devolved to Occupational Councils, successors of the present Industry Training Organisations, but fewer in number, perhaps 20, as a result of encouraging mergers.
In higher education it is the tradition for the provider to take decisions about the content; in validating degrees outside universities, it is suggested the Advanced Qualifications Board would receive and evaluate submissions.
Assessment is a major task in its own right. It is suggested that once the Schools and Occupational Qualifications Boards have settled on the purposes and content of qualifications they should let contracts to awarding bodies, perhaps for five years at a time, to devise and administer the assessments. The Boards would be responsible for ensuring that the assessments were consistent with the purposes and content. The approach of the Advanced Qualifications Board is likely to be different - with it appointing external examiners to the providers' own examining processes.
Each of the domains has particular issues which need to be addressed. In the case of schools there is a need for a review of the curriculum, assessment and qualifications for the senior secondary school in the light of the raising of the school leaving age and increasing participation post 16. In particular, there is a need for a new qualification, or set of qualifications, for the exit point at year 12. This should embrace a broader range of studies than at present including the academic, applied and occupational. Whether these would be best represented by different qualifications, or different pathways within the one qualification, would be for the Schools Qualifications Board to decide.
The occupational domain is diverse and complex and the Occupational Qualifications Board would have to decide how best to bring some order and pattern to it. In particular, it would have to ensure that there were good qualifications in place both in preparation for work and upskilling in work for the employers who wanted them. A major issue for the Advanced Qualifications Board would be to decide how it is to judge whether a proposed qualification is of degree standing in national and international terms.
The role of the proposed Qualifications Co-ordinating Council would be, as its name emphasises, to co-ordinate, ensuring, for example, that the Schools and Occupational Qualifications Boards liaised to provide attractive pathways from school to employment. The Council would also be responsible for maintaining an up-to-date database of the qualifications available, from information supplied by the Boards, to assist the choices of students, parents and employers.
It is suggested that the qualifications structure as it is developed under these arrangements could be better regarded as a network, with a variety of connections, rather than a rigid framework of levels and equivalences. Registration on the network - via the Boards - would be voluntary, but would be a means by which emerging qualifications could gain credibility and reputation. Awarders of qualifications of already established reputation, like university degrees, would be encouraged to provide information to enable the qualifications map to be as complete as possible. The government is urged to treat the credibility of qualifications and eligibility for public funds as separate issues and not run them together as is proposed in the Tertiary Green Paper.
The thrust of these proposals is that, in undertaking a necessary reform of qualifications, New Zealand has been looking for a simple tidy structure which in practice has become a straitjacket. The way forward, it is suggested, is a more bottom up approach beginning with the desired learning and appropriate assessment, and only then moving on to qualifications and possible linkages between them. Qualifications can be an important way of creating expectations for an education and training system. But they are only a means to an end, not the end itself - which is the quality of the learning which is engendered.
Recommendations
Recommended approach to a qualifications framework
1. The government should shift the emphasis of its qualifications reform from
seeking a single formula applicable to the whole system - first unit standards,
now 'quality threshold' - to working through a variety of issues and problems
from first principles.
2. A national qualifications system is desirable to give recognition to qualifications,
and to provide a map to assist choices by students, parents and employers.
3. Any qualifications structure should derive from deciding first how desired learning could be represented in qualifications, and only then seeing what linkages can be made.
4. It is suggested that it would be better to think in terms of a national qualifications network rather than framework, since this implies a more flexible and open set of arrangements.
5. Experience has shown that unit standards cannot be stated with the precision necessary to ensure the fairness, consistency and validity of assessment, and therefore the attempt to use unit standards as the common currency for a qualifications structure should be abandoned.
6. Qualifications should be stated in terms of their purposes, content and assessment in order to clearly convey what they are about, and to facilitate co-ordination between them.
7. While qualifications can sometimes with advantage consist of separable components, for simplicity and practicability it should normally be the whole qualification that is logged on the qualifications network.
8. Registration of qualifications should be voluntary but would carry the entitlement to be called 'national'. This is likely to appeal most to employers and private training establishments, and to providers without degree conferring powers wishing to offer degree courses.
9. Awarders not seeking registration for their qualifications (perhaps the universities) would be encouraged to supply information to enable a comprehensive qualifications database to be maintained.
10. Links between qualifications and opportunities for cross-crediting and credit accumulation should be identified where possible.
11. It would be open for qualifications to specify other qualifications as an entry requirement so that some would be considered higher than others, but issues of level and equivalence would not loom so large in a qualifications network as in a framework.
Recommendations for administrative arrangements
12. A Qualifications Co-ordinating Council should be established to set up and maintain the qualifications network, with the functions outlined in Box D, page 70.
13. A Schools Qualifications Board, an Occupational Qualifications Board and an Advanced Qualifications Board (see Boxes A, B and C, pages 67, 68 and 69) should be established to oversee qualifications in those areas.
14. The requirements for national qualifications in schools should be set by subject working groups convened by the Schools Qualifications Board.
15. The requirements for national qualifications in, and for, employment should be set by Occupational Councils, derived from the present Industry Training Organisations, but there would be ultimately fewer of them and their composition would be different.
16. The requirements for national degrees, and advanced diplomas and certificates, would be agreed by the tertiary providers with the validating committee of the Advanced Qualifications Board, in consultation, if appropriate, with professional bodies.
17. The Schools and Occupational Qualifications Boards would put out to tender contracts for the devising and administering of assessments for the qualifications for which they were responsible.
18. The Advanced Qualifications Board would establish a validating committee for degrees and other higher academic awards offered outside the universities.
19. The Qualifications Boards would establish that the assessments are appropriate to the requirements set and use this as the main criterion for registering the qualifications.
Recommendations concerning school qualifications
20. It is recommended that a review be conducted of the curriculum, assessment and qualifications in the senior secondary school.
21. It is suggested that up to the end of year 10 (Form 4) existing arrangements for deciding the curriculum should continue, but for senior secondary schooling, years 11-13 (Forms 5-7), qualification prescriptions would be the basis of the curriculum.
22. It is proposed that serious consideration should be given to the establishment of a new qualification, or set of qualifications, for the end of year 12 (Form 6) which is presently the main school exit point.
23. Any review of qualifications in the senior secondary school should take into account that consistency across schools is best achieved through assessment on common tasks independently marked. This would suggest there is an important place for appropriate external assessment (which should not be equated with just written examinations).
Recommendations concerning the registration of providers
24. It is recommended that the credibility of qualifications and eligibility for public funds be treated separately.
25. The proposed Qualifications Boards could make available registration and accreditation services to providers on a voluntary basis where the providers saw it as offering some advantage, or where a Board considers some such process is a necessary prerequisite to the registration of its qualifications.
26. The Ministry of Education should be responsible for the approval of providers and their students as suitable to receive money voted by parliament.
CHAPTER 1
INTRODUCTION
A key issue for any country seeking to encourage the learning of all its people
- at school and lifelong - through a qualifications system is how to have something
worthwhile for everyone to aim at while, at the same time, providing the differentiation
that employers and those at the next stage of education and training often need.
It is, of course, a moot point whether qualifications are the best way of promoting
learning - they can take the eye off the ball of the curriculum and training
- but, if it is accepted that they are an important influence, the dilemma has
to be faced.
Traditionally, qualifications in education and training have been used as a selection device. In England, for example, there were, and still are to some extent, tough hurdles at age 16 ('O' levels, now GCSEs) and age 18 ('A' levels) - and originally also at age 11 - to identify students who could be educated at university to a high level in three years with few dropouts.1 Over time employers became used to using degrees as a way of recruiting people to train for the senior positions in their organisations. There were also craft and technician qualifications, usually taken from the ages of 16 or 18 (after the school leaving age had been raised to 16), to train people for skilled supporting roles. This approach worked well in selecting and developing a range of talents which carried the United Kingdom to the forefront of many fields.
Qualifications became a powerful incentive for learning because of the doors they opened. They were for something. Competitive qualifications, however, reject as well as select, and indeed tend to be valued in proportion to the difficulty of obtaining them. Accountancy examinations have, for example, acquired a cachet in the United Kingdom, not only because of where they lead but because the failure rate often exceeds 40 per cent.2
Failing to make the grade at any level can, however, be deeply discouraging. On the other side of the coin from those motivated by the prospect of being selected are those who do not feel they have any realistic hope of passing and do not really try. During the 1980s a number of English-speaking countries became increasingly concerned that this was not only damaging to those affected, but a waste of talent in an increasingly competitive world.
New Zealand has been attracted, along with those countries, to an alternative form of qualification - the badge of achievement. A qualification in something. The essential idea here is that performance is compared with a standard, not with how well other people do. This is analogous to checking a weight or a measure against a reference held for this purpose. Once an educational or training standard has been set there is, in principle, no limit to the number of people who can successfully achieve it. With standards at different levels everyone has a chance to show what they can do. Under this system, it is argued, the higher fliers will emerge by completing more units at a higher level in a shorter time. Everyday examples are scout badges, swimming certificates and graded tests in music. The driving test also depends on comparing performance with an explicit standard.
The yardstick approach to assessment goes under a number of names - criterion-referencing, competency-based, standards-based - each implying slight differences in emphasis, but all embodying the same idea.
The approach was first used extensively by the United States military to train recruits in specified tasks such as being able to strip down and re-assemble a gun. The United States of America has also experimented with it in teacher education.3 It crossed the Atlantic to Scotland and England, first as a way of chalking up credit for unemployed school-leavers on youth training schemes, on small isolatable tasks such as being able to get to work on time, but it was then expanded into a philosophy for reforming the whole of vocational education. New Zealand heard about these developments through its interest in the Scottish system, but has gone further than any other country in trying to establish a standards-based framework, embracing not only occupational training but also the upper years of secondary schooling and university education.
The key question, however, is does the new approach have 'a balance of advantage'
over what it is replacing? Does it provide what students, teachers, employers
and the nation as a whole want in terms of reflecting capabilities and recording
achievement? New Zealand's boldness - or perhaps its foolhardiness - has meant
that it has become the world's testbed for this type of qualification system.
CHAPTER 2
THE EMERGENCE OF NEW ZEALAND'S
NEW APPROACH
Relative economic decline in New Zealand was a major spur to educational reform.
The loss of the preferential trade agreements with the United Kingdom exposed
an over-reliance on agricultural products and caused the government to think
very seriously about how the country was going to pay its way in the world.
Like many other advanced industrial nations which were re-appraising their economies,
it began to focus on high value-added goods and services as a way of competing
with countries with low labour costs. A high value-added economy depends crucially
on the skills of the workforce.
New Zealand had inherited from Britain a highly selective and specialised education
system. In 1984 a third of school-leavers left with no formal qualifications,
while 18 per cent went on to university. Only 11 per cent of the age group graduated.
Out of every 100 fifth formers, only nine went on to get technology-related
qualifications, four at degree-level and five in technology-based trade certificates
or technician certificates.4 Those without skills or qualifications were particularly
likely to find themselves unemployed.
The New Zealand government embarked on an intensive period of review, consultation
and policy formation.5 The move towards internal standards-based assessment
was first signalled in June 1986 in a report, Learning and Achieving,6 from
the Committee of Inquiry into Curriculum, Assessment and Qualifications. Reviewing
education in the senior secondary school in the light of the growing proportion
of pupils who were staying on but without the intention of going to university,
the Committee recommended, among other things:
o that there should be a national curriculum incorporating achievement-based
assessment, and offering a wider choice of options from Form 5 onwards; and
o that internally assessed certificates replace the current external examinations, and that all students receive a National Leaving Certificate (pp. 14-16).
At this time also a policy paper was being prepared on vocational education
and training by the Departments of Education and Labour. They produced a draft7
in 1986 which took the view that occupational training was "unsystematic,
inefficient and unresponsive to change" (p. 12). The policy thrust was
to create easier movement between different vocational paths, assisted by open
credit transfer and the recognition of prior learning. It also came out in favour
of a national certification system based on criterion-referenced assessment.
The ideas emerging in Learning and Achieving and the draft policy paper were
carried forward in the report of a Ministerial Working Party under the joint
chairmanship of Dr Mervyn Probine and Ray Farghar.8 Originally asked to look
at the funding of the technical institutes, the Working Party's brief was extended
to include the whole of post-compulsory education and training. In order to
give some shape to what it saw as haphazard and incomplete provision, it proposed
that a national validation authority be established to examine and moderate
courses and qualifications, except those in universities. It also argued for
a more flexible system of credit transfer and recognition of prior learning
which would link all sectors including the universities and teachers' colleges.
In 1987 and 1988 there were a number of other reports on various aspects of
post-compulsory education and training.9 There was also an influential lecture
tour (reciprocating a visit to Scotland in 1985 by New Zealand's then Minister
of Education and a senior Education Department official) by Tom McCool, Chief
Executive of the Scottish Vocational Education Council (SCOTVEC).10 The features
of the SCOTVEC system - modular programmes of study, a single national certificate,
assessment based on specified performance criteria, a range of entry and exit
points, credit accumulation, and closer links between providers - became central
to the discussions11 of the Achievement Post School Planning Committee (1987).
While supporting the concept of a national certification system, this Committee
also flagged some issues that remain unresolved to the present day, for example
how to marry school examinations with standards-based assessment and the problem
of recognising superior performance in a pass/fail system.
In 1988 the government established a Working Group on Post Compulsory Education
and Training under the chairmanship of Professor Gary Hawke, an economic historian,
to draw together the findings of the previous reports. The Hawke Committee12
noted that the terms of reference did not distinguish between education and
training, but accepted that "Throughout PCET, there is a tension over the
appropriate balance between different but interrelated components of the whole
process of education and training". While advocating:
… a system of national educational qualifications … to give assurance
about educational standards and competencies … [and an] 'across the portfolio'
approach to qualifications [which would] help to reduce barriers to access and
to movement between institutions, and allow appropriate recognition of a course
irrespective of the institution in which it is taught …
it did recognise a distinction between "the stimulat[ion] of the students'
imagination … through subjects" and "the development of directly
useful capabilities" (p. 15).
The differences between the types of learning at different levels were embodied
in the Hawke report's proposal for a National Educational Qualifications Agency
(NEQA). This "would be a small body of part-time members, and would be
essentially a co-ordinating body for three distinct sub-agencies" (p. 54):
o the National Vocational Qualifications Board (NVQB) for vocational qualifications;
o a Secondary Education Qualifications Board (SEQB) for qualifications in secondary schools; and
o a National Academic Awards Body (NAAB) for degree level courses.
Accreditation and validation by the Boards would be voluntary. To make way
for NEQA, the report endorsed the disestablishment of the University Grants
Committee (which had curriculum responsibilities), the Vocational Training Council
and the National Council for Adult Education.
Following consultation on the Hawke Report, and clearly influenced by the SCOTVEC
model, the government brought out a policy paper, Learning for Life: Two,13
in August 1989. It accepted the proposal for a National Education Qualifications
Authority and described its role as "to co-ordinate national secondary
school qualifications, national vocational qualifications and national advanced
academic qualifications" (p. 44). It also said that the qualification system
was to be based on a student-centred approach which stressed the competency
of students to understand and apply their acquired knowledge (p. 44).
A Ministerial Working Party14 under the chairmanship of David Hood was convened
to pave the way for NEQA. During the consultations the universities recorded
their opposition to close involvement with NEQA, other than the monitoring of
course approval procedures. The body was nevertheless formally established as
the New Zealand Qualifications Authority (NZQA) under the Education Amendment
Act 1990. It was given powers to develop a framework for national qualifications
in secondary schools and post-school education and training in which:
o all qualifications (including pre-vocational courses provided under the Access
Training Scheme) have a purpose and relationship to each other that the students
and the public can understand; and
o there is a flexible system for the gaining of qualifications, with recognition of competency already achieved [section 253 (1) (c)].
This is, of course, capable of being interpreted narrowly or broadly. How Gary Hawke's proposal for a light-touch qualifications authority came to lead to an attempted qualifications revolution, taking New Zealand further than any other country, we now explore.
CHAPTER 3
THE QUALIFICATIONS FRAMEWORK
Towards the end of 1990 the newly-established New Zealand Qualifications Authority
(NZQA) issued a discussion document, Towards a National Qualifications Framework.15
This offered essentially two options. The first built on the existing arrangements
and retained some separation between awards for school, university and employment.
The second was for a unified system which broke down what it said was an artificial
distinction between academic and vocational education and which was claimed
to provide a more viable structure for credit transfer and the recognition of
prior learning.
Encouraged by a favourable reaction to the second option - perhaps unsurprising
given the preamble - the NZQA issued a second consultation document, Designing
the Framework.16 The submissions were interpreted as indicating widespread support
for the proposed unified framework, but even at this stage questions were being
raised about keeping competitive examinations and whether assessment should
be graded (achievement-based) or pass/working towards a pass (competency-based).
The NZQA nevertheless felt that it had a clear mandate to be ambitious. Much
of the impetus came from David Hood who had chaired the Ministerial Working
Party on the establishment of NEQA (page 5) and who emerged as the chief executive
of the NZQA. In an extended interview17 just before his retirement in 1996 (under
the clever headline "Has the NZQA hoodwinked a nation?"), he revealed
his "burning sense of outrage" at the consequences of failing the
11-plus for his 30 classmates in primary school in Newcastle, England. His passionate
desire for a qualification system running on success rather than rejection met
Dr Lockwood Smith's, the then Education Minister's, "vision of a seamless
education system",18 and the touchpaper was lit. Ambition fuelled ambition,
and by 1996 the burgeoning framework had acquired19 no fewer than 16 objectives:
o to create a single, co-ordinated framework of qualifications;
o to provide a consistent basis for the recognition of educational achievement wherever that achievement occurs;
o to extend recognition to a wide range of achievements;
o to encourage the integration of "academic skills" with applied skills, and to bring together theory and practice;
o to enable and encourage diversity among providers of education and training, and to recognise academic freedom;
o to reform assessment practices in education and training;
o to raise progressively the standards of educational achievement;
o to shift the practice of teaching from the didactic to student-centred;
o to provide quality assurance for qualifications;
o to enable qualifications to evolve and develop;
o to recognise the principles of the Treaty of Waitangi;
o to provide a rational system of nomenclature;
o to provide a system of credit accumulation and transfer;
o to enable qualifications that are flexible;
o to encourage a wider range of educational settings; and
o to provide incentives to increase individual and collective investment.
This is an agenda which runs considerably beyond the remit of creating a transparent
and flexible qualifications structure, taking a pre-emptive stance on, among
other things, the epistemology of knowledge and ways of learning.
Crucial to the NZQA's plans was the idea of common building blocks, analogous
perhaps to atomic particles in the periodic table. All learning, whether academic
or occupational, and wherever it took place, was to be organised into units
described in terms of outcomes. Each unit was to be assigned to one of eight
levels ranging from Form 5 to postgraduate, and reflecting the amount of work
thought to be involved. The modules - eventually to be called unit standards
- were to be logged by the NZQA on a national database. Qualifications were
to be obtained by the aggregation of modules - national certificate (with units
mainly at levels 1-4), national diploma (with units mainly at levels 5-7), first
degrees (assigned to level 7) and higher degrees and higher certificates and
diplomas (assigned to level 8). The minimum number of credits has been set at
40 for the national certificate, 120 for the national diploma, and 360 for a
degree.
Although the unit standards were said to be time-free - indeed that is intrinsic
to the concept - incongruously a credit has been taken to be the equivalent
of ten hours learning (from the assumption that a typical student would undertake
1200 hours learning a year and be capable of achieving 120 credits). Since units
are frequently small, carrying low credit weightings, many may have to be accumulated
to achieve a qualification. These ideas were formally adopted by the Board of
the NZQA20 in 1991 and disseminated to the various sectors of education and
training through a series of booklets.21
At first sight, the NZQA would seem to have come up with a brilliant solution
for creating a seamless qualifications structure. All conceivable qualifications,
from drainlayers to doctors, could be built up from unit standards registered
on the national framework. But as the Strategic Review22 by Foulkes, Kyrke-Smith
and Laking pointed out:
Development of the framework has thus been shaped by differing policy priorities
and contesting of political forces. It received an early push from the government's
desire to provide meaningful training for the unemployed and widespread concern
about the problem of failure in schools. These pragmatic policy concerns did
not necessarily support a universal seamless qualifications framework integrated
by the single currency of the unit standard. At the political level this vision
was largely the inspiration of one Minister, with the committed support of the
Authority. However at least some Ministers had expected rather less than a total
learning revolution and rather more of a vocational reform through the Industry
Skills Strategy (p. 14).
The grand design for the National Qualifications Framework (NQF), in fact, rests
on a number of largely untested assumptions:
o that outcomes are an appropriate way of expressing all kinds of learning;
o that it is possible to provide clear, unambiguous standards;
o that the candidate's own teacher or trainer can be relied upon to make dispassionate judgments in 'high stakes' summative assessment;
o that the aggregation of the bits will amount to the overall capability implied by the qualification;
o that it is possible to assign level and credit values accurately to units;
o that a unit-based pass/fail system will meet all the requirements of a qualifications structure including being able to distinguish between performances;
o that a system based on many thousands of units will be manageable and affordable;
and
o that teachers and trainers will be able and willing to cope with a greatly increased assessment workload.
If one or more of these assumptions should not hold, the system will be unstable.
If it is not possible to provide clear, unambiguous standards, assessment will
differ in different settings, across the schools for example, and any qualification
based on them will be fundamentally unreliable and therefore invalid. If credits
and levels cannot be assigned meaningfully, the qualification will, in effect,
be adding apples and pears to come up with oranges. If a unit-based system cannot
differentiate accurately, it will not provide the information that students,
teachers and employers often need.
Not all the assumptions are necessarily linked. Modules do not have to be outcome-based;
whole qualifications - not units - can be registered; and there can be comparisons
between performances rather than against benchmarks. Unit standards do not have
to be internally assessed. But in the NQF they come as a package. In many ways
it is a gigantic leap of faith. If it works, it will be copied throughout the
world. If it flops, New Zealand will be left, metaphorically and indeed literally,
picking up the pieces. In the next chapters, we look to see how the NQF has
been received in schools, industry, universities and elsewhere. As background,
we take a quantitative overview of its implementation.
CHAPTER 4
IMPLEMENTATION
The introduction of the NQF is still at an early stage, but already some patterns
are emerging.23 On looking at the numbers, it is possible to get some idea of
how far the proposed system has been put in place, where the take-up is, and
what credits and qualifications are being awarded.
4.1 Creating the structure
A record-of-learning database has been established by the NZQA to keep track of learners' achievements. Everyone from Form 5 (year 11) in schools is eligible, as the expression goes, to "hook on" (that is, pay a registration fee and receive an identity number). Each year all active learners receive an updated record of learning. The aggregate statistics of the database also provide useful indicators of the progress of the system.
CHART 1: Growth in NQF
19961 19971
Learners Hooked On 62385 161511
Registered Unit Standards 5513 9161
Registered Qualifications 161 300
Schools Accredited 264 369
Other Providers Accredited 282 480
Registered Training Establishments 816 777
Industry Training Organisations 52 52
National Standards Bodies/Standards-setters 223 230
National Qualifications Awarded 1600 4138
1 Figures as at 1 May.
Chart 1 shows that we are in the midst of an explosion in unit standards. In
the year to 1 May 1997, there was an increase from 5513 to 9161, with 14,800
estimated to be ready by 1998. The units to date can be assembled into 300 different
qualifications. The sheer volume makes for a very difficult logistical exercise,
and already the cracks are beginning to show, for example in the occasional
stories in the media of blunders like trainee signwriters receiving certificates
for tractor driving.24
There are two quality checks on potential providers of unit standards - registration
(to ensure "the fundamental standards of delivery that safeguard their
clients" are met)25 and accreditation ("formal recognition that a
provider is capable of delivering units to the standards specified").26
Educational institutions are eligible to apply for accreditation without registration.
So far, as Chart 1 shows, 369 of the 445 secondary schools have been accredited
by the NZQA on the advice of the Education Review Office. The New Zealand Polytechnic
Programmes Committee and the New Zealand Council for Teacher Education have
delegated authority to accredit providers in their sectors, and all 25 polytechnics
and three of the five colleges of education have been accredited to deliver
some unit standards to prescribed levels. None of the seven universities, however,
has moved towards formal NQF involvement.
Private training establishments (PTEs, defined as "an establishment, other
than an institution, that provides post-school education or vocational training")27
have to seek registration as well as accreditation if they wish to deliver unit
standards. As at 1 May 1997 there were 765 (the other 12 shown in Chart 1 are
approved government training establishments, for example the police). Of the
PTEs, a third (243) were Maori owned and operated. The sector has grown rapidly
since 1990 partly through existing PTEs becoming registered (there may be over
1,000 still unregistered)28 and partly through new ones coming into existence.
There is continual movement at the margins with a slight fall from 1 May 1996.
The burgeoning of registered PTEs has been attributed by Guerin29 to the increasing
use of contracting models by government agencies, the PTEs' flexibility and
low tendering prices in meeting the Training Opportunities Programme purchasing
criteria (as compared with polytechnics), the recognition given to them by the
NZQA's quality assurance procedures, increased subsidies, and the availability
of allowances and loans to PTE students. Of the 765 registered PTEs, 446 had
been accredited for the delivery of unit standards together with five of the
12 government training establishments (GTEs).
Another important element in the NQF is the standards-setting process. For industry
this is handled by the Industry Training Organisations (ITOs) set up under the
1992 Industry Training Act. Any sector is able to establish an ITO so long as
it can demonstrate support. This voluntaristic approach has led to 52 ITOs coming
into being under the auspices of the Education and Training and Support Agency
(ETSA). A polytechnic can find itself having to deal with many ITOs, each with
their own requirements, and even some of the small providers may be dealing
with five or more. In some sectors there is potential duplication which is leading
to mergers like that between Telecommunications Education and Skills Standards
Council and the Electrotechnology ITO. At the other end of the scale, some sectors
like health and education have remained outside the NQF. In areas where there
has been no ITO, the NZQA has maintained advisory groups which are currently
being consolidated into National Standards Bodies (NSBs) of which four have
so far been established. There are 226 standards-setters in the form of advisory
groups, whakaruruhau and forums. It is claimed that the ITOs cover three-quarters
of industry and taken together with the other standards-setters there is 90
per cent coverage of the workforce.
The ITOs are also responsible for administering training and 33 have been accredited
by the NZQA as having "a quality system that supports the work of registered
assessors operating in the workplace or through industry training".30 The
'Other Providers Accredited' of Chart 1 therefore comprise 446 PTEs, 33 ITOs,
25 polytechnics, 5 GTEs, 3 colleges of education and l wananga.31 Much of the
development so far has been generated by the Training Opportunities Programme.
This is borne out by the participation figures.
4.2 Participation
Chart 2 shows the number of learners hooked on to the NQF has grown rapidly in the last three years from a handful in June 1994 to 164,796 by 29 May 1997. Their distribution across education and training is shown in Chart 3. Over a third were in PTEs, mainly second-chance learners. About a quarter were in schools, and nearly a quarter in polytechnics. Employees hooked on by ITOs accounted only for some 15 per cent (23,534). As we have already noted, the universities are absent.
CHART 3: Hook-ons1 by Education and Training Sector
Sector N Per cent
Private Training Establishments 55356 34.3
Schools 42246 26.2
Polytechnics 36180 22.4
Industry Training Organisations 23534 14.6
Government Training Establishments 1973 1.2
No Provider 1078 0.7
Wananga 532 0.3
Colleges of Education 59 0.0
Other 553 0.3
Total 161511 100.0
1 1 Jan 1993 to 30 April 1997.
Chart 4 shows that two-thirds of the learners were 25 years or younger and nearly 30 per cent of school age. This is in the direction anticipated, since it is planned that by the year 2000 almost all New Zealanders will hook on while still at school - upwards of 50,000 a year. Of the hook-ons to the end of May 1997, 33,888 were Maori (20.0 per cent) and 10,528 (6.2 per cent) Pacific Islands people. Of the Maori, about half had been hooked on by PTEs.
CHART 4: Hook-ons1 by Age
Age N Per cent
18 and younger 48418 29.4
19-25 57634 35.0
26-35 27073 16.4
36-45 19260 11.7
46-55 10127 6.1
56-65 2284 1.4
Total 164796 100.0
1 1 Jan 1993 to 29 May 1997.
Over half the hook-ons were male 56.0 per cent (87,856) and 44.0 per cent were female (69,482). Most of the women have been hooked on by providers - only 5 per cent came through the ITOs, compared with 24 per cent of men.
4.3 Performance
The NZQA expresses achievement in terms of the number of credits awarded, which results in some very big numbers. Chart 5 shows that three-quarters of the awards have been at levels 1 and 2, reflecting the early thrust to help the unemployed and those who had missed out at school. In terms of sub-fields, it is forestry and the motor industry that have the most awards (Chart 6). Quite why they should have come to the fore is not clear. It may be that they are more amenable to the unit standards approach or that there were particular enthusiasts in those sectors (see Chapter 7).
CHART 5: Credits1 Awarded by Level
Level N Per cent
1 326850 31.0
2 463790 44.1
3 184743 17.6
4 69665 6.6
5 6777 0.6
6 773 0.1
7 111 0.0
1 1 Jan 1997 to 29 May 1997.
In terms of domains, 220 out of 335 (62 per cent) each account for less than
0.1 per cent of credits awarded, that is, have as yet been taken by hardly anyone.
This is a problem which has been encountered in England and Wales where a somewhat
similar approach to unit standards has been adopted in framing vocational qualifications.
By April 1997, of the 878 National Vocational Qualifications (NVQs) on the books
(whole qualifications are accredited), 330 had not been taken by anyone and
a further 50 had been completed by only one person.32 The funding mechanism
has encouraged the creation of awards whether they were wanted or not, and up
to about a third are now being phased out.
CHART 6: Credits1 by Sub-field
Sub-field N Per cent
Forestry 122360 11.6
Motor Industry 118606 11.3
Core Generic 113271 10.8
Mathematics 82689 7.9
Computing 56236 5.3
Service Sector 55484 5.3
Bus Admin 51301 4.9
Beauty Service 46656 4.4
Health Studies 44808 4.3
Communication Skills 34738 3.3
1 1 Jan 1993 to 29 May 1997.
4.4 Qualifications awarded
Although talking 'credits' takes us into telephone numbers, relatively few National
Certificates have been awarded. We saw in Chart 1 that to 1 May 1997 a total
of 4,138 had been obtained. There have been 1230 in computing and 1,134 in the
motor industry, but to arrive at a top 10, as in Chart 7, we have to go all
the way down to the 88 awards in hairdressing.
CHART 7: National Certificates Awarded1
Computing 1230
Motor Industry 1134
Food Handling 603
Bus Admin and Comp 533
Carpeting 335
Dairy Manufacturing 176
Horticulture 165
Travel 131
Drainlaying 100
Hairdressing 88
1 By 29 May 1997.
Using 'credits' as the currency can lead to an over-optimistic impression of
the amount of activity. England also tended to take too rosy a view of participation
in NVQs. But a detailed analysis by Robinson33 showed that:
o 660,000 people, not three million as was claimed, were working towards NVQs;
o only 2 per cent of the workforce were working towards NVQ Level 3 by Spring 1995 against a target of 50 per cent by 1996;
o the NVQs that are taken tend to be in the internationally-sheltered service occupations - clerical, secretarial, personal service and sales;
o over two-thirds of vocational qualifications currently awarded are the old-style pre-NVQ awards; and
o NVQs do not appear to have added to total training but have increased the
complexity of provision.
These stark findings and other criticisms have led the Department for Education
and Employment in the United Kingdom to institute more careful monitoring procedures.
They have also added to the clamour for the reform of NVQs, because it is becoming
increasingly apparent that employers have not taken to them in the way that
was hoped. In New Zealand too a large number of qualifications remain outside
the Framework. These are of two kinds - provider qualifications recognised by
the NZQA but not part of the NQF, and qualifications completely outside the
NQF.
4.5 Qualifications outside the NQF
As at 6 May 1997 there were 1789 Provider Courses approved and accredited by
the NZQA.34 Of these, 692 were local and 223 national. Although these carry
the NZQA's seal of approval they are not registered on the NQF because they
do not consist of unit standards. They do, however, become eligible for public
funding. In addition, there are 158 degree courses approved outside the universities.
Most were in polytechnics but some colleges of education, wananga and PTEs are
also accredited to offer degrees.
The Sixth Form Certificate is accredited separately from the NQF and this accounts
for 716 of the courses. The NZQA also runs a number of examinations outside
the NQF, including the two secondary school examinations. The School Certificate
is usually taken in year 11 and is awarded in individual subjects. It is mainly
by external assessment although in some subjects there is an increasing amount
of internal assessment. In 1996, 38,759 of 63,749 entries (60.8 per cent) achieved
a grade C or higher.35 The University Bursaries examination, usually taken in
year 13, is by external assessment (except for practical art and physical education)
although some subjects also include internal assessment. In 1996, 10,546 of
25,474 entries (41.4 per cent) achieved at least a B, including 892 (3.5 per
cent) awarded scholarships.36
There are also a diminishing number of trades, advanced vocational awards and
typing examinations run by the NZQA which are being progressively replaced by
unit standards. In 1995 there were 241 trades certificates, but by the year
2000 this is expected to be down to 26.
University degrees, where the Vice-Chancellors' Committee has statutory powers
to accredit universities and approve degree programmes, have remained completely
outside the NQF. In addition, there will be many other awards of PTEs that are
not registered or accredited, and an unknown number of qualifications offered
by employers and others, which do not appear on a central database.
4.6 Funding
Although the NQF is eventually intended to provide the basis for an education marketplace running on learners choosing between providers, its present shape and development is much influenced - both positively and negatively - by the availability of public funding. Foulkes, Kyrke-Smith and Laking37 (p. 29) suggest that in 1996 the main sources were:
Education and Training Support Agency
o TOPS funding ($186 million) which purchases training mainly from PTEs, and
also polytechnics (8 per cent). Most (70 per cent) is for unit standards based
training;
o Industry Training Fund ($52 million) which is available to ITOs to purchase
on- or off-job training to level 4 of the Framework. Ninety per cent of the
$32.8m for off-job training goes to the polytechnics;
o Skills Enhancement ($9.3 million), about half of which goes to the polytechnics,
for young Maori and Pacific Islands people between 16-21 (replacing Maori Trade
Training and Maori Vocational Training).
Ministry of Education
o Equivalent Full-Time Student (EFTS) funding (about $1.1 billion) for courses
offered by universities, colleges of education, and polytechnics to all levels
including postgraduate degrees; and
o Secondary Tertiary Alignment Resource (STAR) funding ($6 million) which can
be used by schools to purchase standards-based training from PTEs and polytechnics.
These mechanisms have provided the incentives for the setting up of ITOs and
PTEs, and the development of unit standards at the lower levels. The PTEs have
usually been able to offer lower tender prices for TOPs than the polytechnics
(which can often get a higher return from EFTS funding - based on estimates
of the amount of training to be undertaken, not a buyer/seller relationship).
The lack of financial encouragement for polytechnics, or indeed colleges of
education or universities, to throw themselves into unit standards was a source
of concern to Foulkes et al. who recommended that the NZQA should seek to have
funding arrangements brought into line with policy objectives. This hangs as
something of a threat over the current review of unit standards.
4.7 Overview
The availability of public money is such a succulent carrot and the prospect of its withdrawal such a big stick that it is not always easy to disentangle reactions to the NQF itself from the funding aspects. From its origins in the Training Opportunities Programme, the conception of the NQF has grown to encompass all of education and training, from secondary school awards upwards. It has, however, posed special problems for schools, and it has not taken off at all in the universities. The response from industry is also less than might have been expected.
CHAPTER 5
THE NQF AND SCHOOLS
The response of the schools to the introduction of unit standards has been mixed.
Some secondary school teachers have welcomed them. At Cashmere High School,
Christchurch, I met a group of teachers38 who were largely in favour on the
grounds that:
o they provided something for all students to aim at;
o they created qualifications more oriented to employment; and
o they were based on internal assessment.
I also met the principals and staff members of Epsom Girls' Grammar and St Cuthbert's,
Auckland,39 who favoured the approach as a diagnostic tool revealing the strengths
and weaknesses of high-achieving girls who were then able to go on, they said,
to get better results in the Bursaries examination.
On the other hand, a number of schools have been fierce in their opposition.
A report40 from 80 secondary school principals argued that "unit standards
as currently constituted will impose an unacceptably narrow philosophy and methodology
of teaching, learning and assessment on the New Zealand education system".
They argued for the retention of school examinations as part of "a dual
pathway".
The Post Primary Teachers' Association41 (PPTA) was initially enthusiastic,
but during 1992 and 1993 imposed a moratorium on its members' involvement in
NQF development, not because of concerns with the NQF itself but as part of
industrial action over the bulk funding of teacher salaries. During this time
the NZQA moved from achievement-based (graded) to standards-based (competent/not
yet competent) assessment which swung the PPTA towards opposition, though it
wanted to go further than the NZQA in phasing out examinations. However, as
the trials of the mathematics and geography unit standards got under way the
PPTA membership began to express serious reservations. This led the PPTA to
commission, in 1995, a Qualifications Framework Inquiry. In its report the Inquiry
members make a number of severe criticisms of the way the unit standards approach
is being implemented, and emphasise, "the Inquiry has no desire to see
unit standards or the Framework used extensively in the senior secondary school
unless modifications are made" (p. 119).
The Inquiry report (pp. 71-72) records the results of a survey of PPTA members,
which attracted 832 responses from 202 schools, indicating a number of design
features found unacceptable including:
o time and workload problems (1398 mentions);
o fragmentation of teaching and learning (1330 mentions);
o no recognition for excellence (1330 mentions);
o inefficient or inadequate moderation structures and processes (1221 mentions);
o very difficult for weaker students to gain credit - leading to lack of motivation (818 mentions); and
o forcing diverse courses into a single assessment model (741 mentions).
These echoed the mounting criticism of the unit standards approach from a number
of individual teachers and academics. The chief problems identified were:
o the difficulty of specifying school subjects as unit standards (maths - Neyland,42
1994; history - Campbell,43 1995, Childs,44 1995; science - Austin,45 1996,
Batchelor,46 1996, New Zealand Institute of Physics,47 1996);
o concern about the appropriateness of standards-based assessment (Morris,48 1993; Croft,49 1993, 1994; Irwin,50 1994; Campbell, 1995, Elley,51 1995, Brace,52 1996, Milne,53 1996);
o feasibility of moderation (Morris, 1993; Campbell, 1995; Irwin, Elley and Hall,54 1995; Mann,55 1997);
o manageability and practicability (Coutts and McAlpine,56 1996; Keown,57 1996; Haskell,58 1995).
Much of this criticism arises from analysis and personal experience, but it
is supported by research (Gilmore,59 1997). The Report on the 1996 School Trials
Surveys60 in biology, science, history, chemistry and physics found that "most
students and teachers are struggling with the paradigm shift" (p. 3). "For
teachers there are big issues around workload and class-room management"
(p. 45). The students were even more critical:
Students believe that the current examination system gives them a qualification
(even a very low grade in school certificate) and low achievers are concerned
that as they are not achieving unit standards they will leave school with no
qualifications. There needs to be lower level unit standards so that this is
not the case (p. 46).
Keown (1996) in his study of assessment practices in six schools trialling unit
standards in geography and maths found: "Much of the data reported in this
study suggests that most teachers and classrooms have tended to slide off the
ridge (metaphor for balancing teaching and assessment) on the over-assessment
side". Difficulties have also been experienced with moderation. Coutts
and McAlpine's report on the 1995 maths and geography trials found: "the
majority of teachers indicated they did not know or were unsure about the effectiveness
of moderation between schools" (p. 111). Some of the moderators expressed
concerns: "there was apparently still a large gap between the quality of
assessments between schools and the quality of the judgements made". Moderators
reported feeling unsupported and isolated and were wondering how they could
cope when more subjects and more schools come on stream.
At whatever evidence we look - comment, analysis, research - it seems that the
introduction of unit standards into schools has not gone smoothly. Whether this
is merely the teething troubles of adjusting to a new 'paradigm', or because
of fundamental weaknesses as Elley (1996)61 has argued, we will attempt to answer
in a critique of the NQF in Chapter 9. It is already evident, however, that
achieving consistency in assessment practices for the unit standards approach
across nearly 450 secondary schools is looking dauntingly difficult.
CHAPTER 6
THE NQF AND DEGREES
6.1 Universities
Perhaps not surprisingly the strongest resistance to the NQF has come from the
universities. Not only is the approach philosophically and pedagogically alien
to them, but it poses a threat to their independence as conferers of degrees.
In a sequence of papers which might collectively be called by the title of one
of them, Why Universities Do Not Want Unit Standards, Hall62 has persuasively
put the case from the universities' point of view. There have also been trenchant
criticisms from, among others, Codd (1994)63 and Elley (1992)64. The major objections
are:
o the atomised nature of unit standards militates against the depth and integration
of understanding that is part of the raison d'etre of degree courses;
o this is exacerbated by registering units rather than whole qualifications;
o the difficulty, if not the impossibility given the nature of knowledge, of specifying university studies as unit standards;
o the unnatural separation of unit standards from course design - course development proceeds as an iterative process;
o incompatibility of university procedures and the registration requirements of the NZQA;
o the unit standard methodology fails to give sufficient recognition to the importance of excellence;
o redeveloping 500 qualifications and 9,000 individual courses or papers would impose an unacceptable workload and have major resource implications for the universities;
o the framework is too coarse to deal with the detail and complexity of advanced
university qualifications and, for example, lumps together at Level 8, PhDs,
diplomas and certificates; and
o different philosophies on credit transfer and the recognition of prior learning,
including the apparent failure of the NZQA to appreciate research findings on
the transfer of generic skills.
Many of these criticisms were taken up by the New Zealand Vice-Chancellors'
Committee (advised by Hall) in its 1994 paper, The National Qualifications Framework
and the Universities.65 The NZVCC express:
… grave doubts as to whether it is practicable or desirable for the Framework
proposed by the NZQA to encompass tertiary degrees. In particular, the NZVCC
believes that the model for developing and registering unit standards upon which
the whole Framework depends is incompatible with the aims of most university
degree courses (p. 2).
The Committee argued for a dual structure:
… in which degrees are handled according to the procedures that are appropriate
to their aims, nature and complexity and in which the relationship between degrees
is determined according to a range of stated principles. Specifically, the NZVCC
believes that the procedures of the Committee on University Academic Programmes
and the New Zealand Universities Academic Audit Unit are the most appropriate
way of satisfying the gazetted criteria for the approval and accreditation of
university courses of study (p. 19).
A Tertiary Lead Group66 was appointed in 1994 to advise the Minister on the
relationship between universities and the NQF, and it recommended that approved
degrees should be registered on the NQF but as whole qualifications not necessarily
composed of unit standards. In addition, it allowed for standards-based degrees
which could be developed by other bodies and registered as national qualifications.
It in effect drew a distinction between 'provider' and 'national' degrees.
The Tertiary Action Group67 was established to make recommendations to the Board of NZQA on what should be required of registered degrees. It reported in April 1996, since when the Board and the NZVCC have been negotiating over the details. It remains to be seen whether the Vice-Chancellors will eventually sign up.
6.2 Polytechnics
The NZQA has been perceived as a threat by the universities. However, a 1990
amendment to the Education Act 1989 made it possible for the first time for
other providers to award degrees. This mainly affected the polytechnics. As
we saw in Chapter 4, the NZQA approves at present 158 degree courses outside
the universities.
It is likely that in the future the polytechnics will be looking to become more
involved in degree-level work, as will other institutions. Making it possible
for them, in effect, to offer their own degrees (instead of being the franchisee)
has left polytechnics with an ambivalent attitude to the NQF. On the one hand,
it represents an opportunity but, on the other, in some of their traditional
fields they are having to bow to the demands of the ITOs and employers, and
finding it hard to compete on cost grounds with the PTEs.
They have also commented publicly that:
… the rigid implementation philosophy adopted by the New Zealand Qualifications
Authority has been over-bureaucratic, inflexible and unlikely to be responsive
to New Zealand's rapidly changing educational needs (p. 1).68
One of the issues that must be addressed in any review of the NQF is how degrees
both in universities and outside are to be validated.
CHAPTER 7
THE NQF AND INDUSTRY
In their objections to standards-based assessment, the schools and universities
have generally conceded that it might be appropriate for the development of
practical skills. It could be expected, therefore, that the NQF would have received
its warmest welcome from industry, and that generally has been the case. Industry
has relished being put in the driving seat and being able to treat education
institutions as contractors instead of, as it saw, having to take what it was
given (hence some of the polytechnics' discomfort with the NQF).
There has been vigorous promotion of the NQF by the various agencies involved
in Skill New Zealand,69 by ETSA, and by the NZQA itself. It is supported by
the Employers Federation70 and the trade unions.71 Both ITOs and the PTEs have
opposed what they see as the 'dilution' of the NQF if, as has been proposed,
'provider qualifications' (i.e. those not made up of unit standards) are allowed
to be registered on it.
Given the considerable political and financial investment in the success of
Skill New Zealand, and the public relations exercise surrounding it, it is difficult
at this stage to assess its impact. Although the annual Stocktakes72 are producing
some impressive numbers, as we saw in Chapter 4, they have been magnified by
using 'credit' as the coinage, and boosted by public money. As Smelt73 has observed,
some tensions are already emerging:
Each ITO has, in effect, monopoly rights in its designated area to set skill
standards and develop training arrangements which can connect to the national
qualifications framework. By various means ITOs can extend this to training
delivery. The control and use of their property rights are proving a source
of tension between firms, between employer and employee interests and between
industry and provider interests. Moreover, the discipline of market forces in
resolving these tensions is lacking (from Executive Summary).
There is no doubt that some employers are very enthusiastic. I had the opportunity
of meeting during my fieldwork Stephen Wickens, the Executive Director of the
Motor Industry Training Organisation and Allan Bloomfield, the Managing Director
of Mitsubishi Motors, where it is based. They left me in no doubt how much better
they felt training was now that they, and not the providers, had control of
it. They also showed me the impressive training manuals for the National Certificate
in Automotive Electrical Engineering and the National Certificate in Automotive
Engineering, in which the bald and generalised elements and performance criteria
of the unit standards had been turned into detailed training programmes. The
manuals listed the units that had to be obtained and to what levels and credit
values, and for each unit addressed a number of relevant questions:
What is the objective of the unit?
What is this unit about?
What unit standards must I have done first?
What safety precautions must I take?
What tools and equipment will I need?
What do I need to be able to do to pass this unit?
What do I need to know to be able to pass this unit?
On the same occasion, I met Rod McGowan of the Forest Industries Training and
Education Council (FITEC) who reported enthusiastically on how the unit standards
approach was motivating forestry workers (who are often isolated) to improve
their skills by, in effect, having stamps put in 'passports' issued by FITEC.
I was also able to visit DB Breweries at Otahuhu where a large scale trial74
is being conducted into the feasibility of basing the Company's staff development
policy on unit standards. Specifically, it has used general units developed
for the National Certificate in Manufacturing Processing (Level 1) and Food
Processing (Level 2), in the packaging departments. There were a number of positive
outcomes:
For the breweries the trial has shown up some procedure problems and gaps in
our quality systems. It has provided us with a way of solving these problems.
For staff the trial has started to give recognition for existing skills that
have previously not been formally recognised. This has major personal worth.
Perhaps the greatest to benefit at this stage are the assessors themselves.
But there have also been problems:
Time has proved the biggest stumbling block for the trial. There is a need to
balance the needs of production against the time needed for assessment.
Staff have been overwhelmed by the number of units available and prefer to have
a clear idea of the exact standards to take for their job.
The trial was aimed at recognising existing competency … to minimise the
threat of assessment (fear of failure) … it has proved to be frustrating
for some as they want the chance to get on and do something new.
There were also concerns about the costs:
The charges have been heavily subsidised by the ITO under the Skill New Zealand
development project. The company is picking up all costs for the trials, however
whether we continue to pay for NZQA charges for all staff at full implementation
has yet to be decided.
Mitsubishi and DB Breweries provide good examples of how industry is working
to incorporate unit standards into training programmes and staff development.
But it is too early to say whether the approach will bed down once industry
has to bear more of the costs. It is possible to look abroad, however, to infer
what the response might be.
Countries within the United Kingdom have so far attempted to introduce a competency-based
approach only into vocational education, but the criticisms of it in this sphere
are remarkably similar to those made by schools and universities in New Zealand.
It may be that only the most obvious weaknesses are held up to public scrutiny,
and New Zealand will have to work through the layers affecting schools and universities
before those affecting industry are fully aired. It is to the British experience
with vocational qualifications that we now turn.
CHAPTER 8
BRITAIN'S EXPERIENCE WITH VOCATIONAL QUALIFICATIONS
The New Zealand Qualifications Authority has worked closely with its UK counterparts,
the National Council for Vocational Qualifications (NCVQ) and the Scottish Vocational
Education Council (SCOTVEC), and this is reflected in the very similar structure
of unit standards and National Vocational Qualifications (NVQs) and their Scottish
equivalents (SVQs). It is important to see, therefore, what can be learned from
the somewhat longer UK experience.
NCVQ was established in 1986 with the aim of creating a national framework for vocational qualifications to rival the well-established academic ladder of O-levels, A-levels and degrees. However, instead of bringing together existing awards into a coherent pattern and building on these, it adopted a radically new approach based on what it called 'functional analysis' - a route which New Zealand subsequently followed. Every job was to have a qualification, with requirements derived from breaking down the work into units, elements and performance criteria. This was a method more suited to accrediting prior learning in the workplace - in effect qualifying people for jobs they already had - but it was extended to become the basis of all vocational qualifications. Instead, therefore, of there being training programmes with appropriate assessment, the obtaining of a vocational qualification became a matter of evidence-gathering in relation to hundreds of performance criteria.
8.1 Criticisms
At first, criticism was muted, because so many of those who could see the weaknesses were financially dependent on NCVQ and its qualifications, and it was ruthless in slapping down adverse comment. But a television programme in December 1993, All Our Futures: Britain's Education Revolution,75 opened the floodgates. A number of significant flaws were brought to light, among the most serious of which was that setting out the requirements of qualifications in a fragmented fashion means that the content of the qualifications is not clearly specified, nor is there any assessment of overall performance. More specifically, as regards content, it was pointed out that the approach lacks precision, is fragmentary, does not prioritise, and devalues knowledge and understanding. Furthermore, assessment is atomised, internal and bureaucratic, and is not robust enough to withstand payment by results. Some of those criticisms are expanded below.
8.1.1 Precision
NCVQ's analytical approach becomes in practice a search for the elusive irreducible building blocks of competence. The NVQ Level 2 Care (Residential/Hospital Support), for example, is set out as 11 units, 39 elements and 338 performance criteria. Although this detail is assumed to give precision, in fact the performance criteria come out as very generalised. In the unit '"Enable clients to eat and drink", the first element is "Enable clients to choose appropriate food and drink" of which the first performance criterion is, "The support required by the client is established with him/her". Similarly, in the NVQ Level 3 in Engineering Assembly, the unit, "Produce assembled output by joining and fastening operations" contains the element "Process materials to produce assembled output", which has as a performance criterion, "Materials presented to the assembly operation are completely compliant with operational specification".
Not only are the performance criteria generalised, but they hang in the air addressing no one in particular. If they had been written for candidates they would have said something like: 'In order to get this qualification you will have to show you can … ' . Or if for employers, 'A person holding this qualification is able to … .' Furthermore, they have been shown to be written in a language which breaks the rules of grammar and is therefore very difficult to read.76
8.1.2 Fragmentation
The fault is, however, more than technical; it is fundamental. There is no guarantee that numerous individual competencies - even if they could be identified and be simply and unambiguously stated - would amount to skilled overall performance. Being able to dribble and do headers are separate accomplishments, but not enough to make a footballer. They must be used appropriately, and integrated with other skills.
8.1.3 Prioritisation
NCVQ's lists of performance criteria are neither integrated, nor prioritised. In the NVQ Level 2 Bus and Coach Driving and Customer Care, for example, the minutiae of customer care are treated on a par with keeping the vehicle safely on the road. NVQs take no account of time, and thus difficult decisions as to what is essential as opposed to being merely desirable do not have to be faced. This is claimed as a virtue,77 getting away from the time-serving basis of old apprenticeships. But leaving time out of the equation altogether conveniently side-steps difficult decisions about what can be fitted in.
8.1.4 Knowledge and understanding
The atomistic approach also has profound consequences for the way 'knowledge and understanding' (used by NCVQ as a compound noun) is treated. It was first considered to be embedded in the performance criteria and implied by them. But even in later revisions,78 where the intention has been to lay greater stress on knowledge and understanding, it comes out as itemised and disparate 'knowledge specifications'. This means that NVQs are virtually useless as qualifications for 14 to 19 year-olds, or indeed adults, preparing for work, since there is no coherent statement of content. Nor is there a reservoir of knowledge and understanding to enable people to cope with a changing working world or to provide a platform for progression.
8.1.5 Assessment
The unsuitability of NVQs as a qualification for work, as opposed to accrediting prior learning in work, is underlined by the way they are assessed. NVQs were said to have been devised on the driving test model. That is, it does not matter how you have learned to drive - through a driving school, or from a spouse, or friend, or in any other way - what is important is that you can satisfy an independent examiner that you can do it. However, as they have emerged, NVQs are not equivalent to the driving test, but consist of long lists of performance criteria that have to be signed off. Given that there may be several hundred of them, the only practicable way of achieving this is to leave it to the teachers/trainers whose main task can then become signing their names.
There is some check through external verification, but that is based on inspecting portfolios of evidence rather than observing the candidate in action. Moreover, because the requirements are so loose, the external verifiers are to some extent able to invent the qualifications by insisting that in the portfolio the candidate does or does not include this or that, uses 'I' or 'we', and so on. Some private providers, for the sake of their candidates, are having to deploy a member of staff specifically to get to know the foibles of particular verifiers.
NVQ assessment, instead of being based on tests of skilled overall performance, depends on collecting evidence in relation to checklists. It lacks the fairness, reliability and authenticity that would make it credible. If it is not trusted, the qualification cannot act as a passport between training provider and employer. Increasingly, the only training in which an employer can have confidence is that provided in-house. Far from enabling education and training to work together, NVQs have tended to drive a wedge between them.
8.1.6 Output-related funding
The assessment of NVQs is not strong enough to bear the weight of the payment by results that is increasingly being adopted by the funding agencies. As the instances of malpractice that surface from time to time in the press illustrate,79 the assessment arrangements leave a lot to be desired. There is ample scope to, in effect, sell NVQs (or give them away if the state is paying). What NVQ assessment amounts to, in practice, is a bit like your driving being examined by your trainer with each item - gear changing for example - signed off as you achieve it, and the trainer only being paid if he/she passes you. No wonder most employers do not automatically accept NVQs.
8.2 Beaumont Review
Mounting criticism led the government to set up a review of National Vocational Qualifications under the chairmanship of Gordon Beaumont who had been the chairman of the Training Panel of the Confederation of British Industry (a strong supporter of NVQs). In the circumstances the report was surprisingly hard-hitting.
o It suggested that setting standards and designing qualifications should be separated: "It is proposed that standards are written for employers. Qualifications, training and the development of assessment needs should be separately specified" (p. 5).
o It recognised that NVQs as presently framed are all but incomprehensible: "Standards are marred by complex jargon ridden language" (p. 13); "The complex and jargon ridden language used is universally condemned" (p. 28).
o It identified a number of problems with assessment: "Assessors and verifiers
are unsure of the standards they are judging and their views differ" (p.
13); "External verifiers suffer from combining incompatible roles"
(p. 19).
o It found the assessment system is not robust enough to withstand output-related
funding: "Funding programmes and policy should be harmonised with qualifications
systems" (p. 7); "Funding by outputs brings potential conflicts of
interest" (p. 40).
o It is recognised that NVQs do not provide a training route for unemployed young people or adults: "The fact that NVQs are work-based prevents those not in work from obtaining the qualifications" (p. 26).
8.3 Qualifications and Curriculum Authority
NCVQ itself was subject to a review, and the government responded by deciding to merge it with the School Curriculum and Assessment Authority (SCAA) to form a new body to be called the Qualifications and Curriculum Authority (QCA), with responsibility for the curriculum in schools and post-school qualifications except degrees. QCA came into being on 1 October 1997, with the chief executive and six of the seven senior posts appointed from SCAA, so it is more of a takeover than a merger. This will hopefully let in some fresh thinking.
Curiously, as almost its last act before giving way to QCA, NCVQ has had the task of recommending new criteria for what should count as a national vocational qualification. True to form it has adopted a narrow rigid definition and proposed that alongside NVQs there should be Related Vocational Qualifications (RVQs) to credit training undertaken in preparation for work rather than in the workplace itself, and Other Vocational Qualifications (OVQs) that employers want but which do not conform to the strict criteria. The UK government which took office in May 1997 has accepted the new criteria as a step along the road to reform but has remitted RVQs and OVQs to the new Authority for further consideration.
8.4 The present situation
The present situation in the United Kingdom can be summarised in the findings of two research reports. Eraut et al.81 from an empirical investigation of competence assessment in the NVQ system came to the view that:
NVQs as currently awarded cannot be said to indicate achievement of nationally consistent, explicit standards of competence; and the claim that the assessment of NVQs is valid and reliable cannot be sustained (p. 66).
It is impossible to have a national system of qualifications based on current competence at work; because we do not have a national system of working practice (p. 67).
There is also a danger that the functional analysis approach … will create an illusion of easy transfer from one context to another. There is little evidence about the transferability of many NVQs, so the concept of them being national qualifications may turn out to be another delusion (p. 67).
Senker82 in a Strategic Review of Construction Skills Training concluded:
Those who secure Level 2 NVQ may sometimes meet employers' short-term needs in practice. However, research studies have shown that the knowledge, skills and experience, especially underpinning knowledge, are inadequate to form the basis for development and future progression. To achieve competence requires a level of input of teaching, breadth, achievement of skill level and ability to achieve quality on a repetitive basis.
8.5 Relevance to New Zealand
How relevant is this experience to New Zealand? Both countries decided to go
in an untried direction, rather than building on what was there. The structure
of the units is very similar, although the United Kingdom opted to register
whole qualifications. In both countries analytic schemes have been turned directly
into the qualification requirements, without giving much thought to the impact
on the design of the curriculum or training programmes. In both countries the
approach has not been greeted with the enthusiasm that was anticipated.
The overall conclusion of Eraut et al. was that the problem:
… cannot be addressed only by improving assessment procedures, not even
by improving the qualification. Changes in the policy framework which sets the
conditions under which NVQ training and assessment occurs are equally necessary.
If New Zealand's experience does turn out to be the same as that of the United
Kingdom, and there are sufficient similarities to suppose that is likely, then
this is good advice. No amount of tinkering will put right something that is
fundamentally flawed. Only a conceptually sound strategy will succeed. We turn
now to consider whether New Zealand's National Qualifications Framework is,
in fact, conceptually sound.
CHAPTER 9
CRITIQUE
As we saw in Chapter 3, the National Qualifications Framework rests crucially
on a number of assumptions which were largely untested at the time it was devised.
The subsequent experiences of schools, universities, industry and other providers,
as they have grappled with the problems of implementation, suggest that the
difficulties may be more than just teething problems. There appear to be basic
structural weaknesses. Let us now examine each of those crucial assumptions
in turn before going on to look at some of the windmills at which the NZQA has
been tilting.
9.1 Can learning always be expressed appropriately as outcomes?
At first sight, since change is at the heart of learning - "to accelerate change, to promote change that would not otherwise have occurred, control the direction of change"83 - it seems perfectly reasonable to want to set out what a learner should be able to do at the end of a period of learning that he or she could not do before. But that is to assume we are always looking for predictable change. In fact, ultimately life itself cannot be stated in outcomes terms. Science presents us with a very bleak view of human existence - three score and ten on an insignificant lump of rock knowing not from where we came or where, if anywhere, we go. Each of us has to develop a sense of meaning and purpose and there is no one right answer.
Much of the justification for a compulsory school curriculum is to give children a basis for developing that sense of meaning and purpose. School subjects are essentially different ways of making sense of the world.84 Science, for example, depends on continually re-checking pictures of external reality, maths on logical deduction from axioms, and history on sifting the documentary evidence. Other subjects, for example literature, art and music, ask students to engage in experiences that have brought meaning to many people's lives to see if they bring meaning to their lives also.
The point of engaging with Shakespeare, Picasso or Mozart is to have shared their insights, not to achieve defined outcomes. It is intrinsic to the experiences that people should take away different things from them. This, of course, poses difficulties for assessment which can find itself reduced to recognising the character associated with particular phrases. But the most authentic way would seem to be to set common tasks such as discussing the meaning of the play/work of art/piece of music, or how the author/artist/composer achieves particular effects, and by comparisons between the responses (rather than against pre-set outcomes) judge what the candidates have made of it.
Not all learning is organised as subjects. In his review of qualifications for 16 to 19 year-olds in England and Wales, Dearing85 distinguished:
- Academic Study: "where the primary purpose is to develop knowledge, understanding and skills associated with a subject or discipline".
- Applied Education: "where the primary purpose is to develop and apply knowledge, understanding and skills relevant to broad areas of employment".
- Vocational Training: "where the primary purpose is to develop and recognise mastery of a trade or profession at the relevant level" (pp. 15-16).
Essentially these differ in their organising principles: subjects are ways of establishing different kinds of truth about the world; applied education consists of practical applications of knowledge (for example, engineering and medicine); and occupational training is defined by specified tasks. The NZQA, and David Hood in particular, have tended to play down these differences - and we will take up later whether there is an academic-vocational divide.
Given its organising principle, it might be thought feasible to express occupational training solely in terms of outcomes. Much of the skill of an accomplished performer, however, comes from painstaking practice and repetition. It may be important, therefore, even in this sphere, to specify what is required not in terms of outcomes alone, but also as relevant experiences - which in this case will be the training. This is not to suggest that assessment should be based on mere attendance; it is to argue that outcomes alone are inadequate for specifying qualifications.
9.2 Can standards be expressed unambiguously?
The standards-based qualifications framework which New Zealand has been aiming to introduce goes even further than attempting to express learning in terms of outcomes. It seeks to do so as 'standards'. The term 'standard' has been most recently defined in this context as:
A nationally registered, coherent set of learning outcomes and associated
performance criteria, together with technical and management information that
supports the delivery and assessment processes. Standards-based assessment is
the process of judging learner achievement against defined standards
(p. 14).86
Crucial to the practicability of the NQF is that standards can be expressed clearly, unambiguously and transparently. It is a requirement that goes beyond any other qualification system in the world, because they become the sole means of communicating what a qualification is about. In fact, defining standards is no easy task. Irwin87 has wryly commented that the whole enterprise seems to need something like C.S.Lewis's88 satirical "pragmatometer" - a device for defining the indefinable, identifying the unknowable, and connecting the unconnectable.
Standards for human performance, in essence, consist of an activity and a specification of achievement. For example, we can set a standard, say, that a person should be able to jump over a bar set at a metre above the ground. Given some modest preparation any unbiased observer would be able to judge whether the standard had been achieved. However, there are relatively few occasions where one can be that precise. Even in competitions like gymnastics, ice dancing and diving, the marks awarded by judges show there is scope for wide variation in the interpretation of performances; and controversy over refereeing decisions is an everyday part of the sporting scene. This is in spite of the great care that has been taken to spell out the rules of the game or, in assessment language, the criteria.
When one is trying to codify and judge complex behaviour like learning, it becomes even more problematic. For example, the first element in a unit standard for history is "Locate and gather historical information". The performance criteria are given as:
1.1 Sources located are appropriate to the topic of the investigation.
1.2 The gathered information is in accordance with the topic of the investigation.
These are very broad statements which do not pin down the subject at all.
Similarly, the first element in the Training Manual for Automotive Electrical Engineering is "Demonstrate knowledge of workplace security risks". The performance criteria are:
1.1 The effect of stock loss on company profitability is identified.
1.2 Methods of displaying stock in secure surroundings are identified.
1.3 Procedures to handle money in a secure manner are identified.
1.4 Company procedures to take against personal attack are identified.
1.5 Insurance security requirements for business premises are identified.
Again we have very generalised statements which only begin to take on meaning when the content and assessment of the training programme are spelled out in the Motor Industry Training Organisation's training manual. The performance criteria themselves are capable of being interpreted in a variety of ways, and different judgements can be made in relation to them.
Even the NZQA now accepts that the performance criteria do not define a transparent standard. Its current position is that the standard is an amalgamation of (i) the registered standard, (ii) the assessment activity and assessment schedule, and (iii) moderation. This is represented as a process of 'triangulation':89
Assessment Activity
Registered and
Standard Assessment Schedule
THE STANDARD
Moderation
The NZQA has, in effect, accepted that standards are as much in people's heads
as on paper. But it does not seem to appreciate that if the building blocks
of the qualifications framework cannot be clearly and unambiguously stated as
outcomes, then the whole structure fails. Desperate attempts to shore it up
through ever-more costly bureaucracy and ever-more elaborate moderation will
only delay the inevitable.
9.3 Can the candidate's own teachers make dispassionate judgements?
Judging human performance is difficult even when those doing so are being as objective and dispassionate as possible. But if there is a vested interest in the outcome, as with NVQs in the United Kingdom where, on some occasions, assessors are, in effect, paid to pass the candidates, there can be little confidence in the qualification. The increasing accountability of educational institutions in terms of results, including the publication of league tables, puts pressure on internal assessment practices.
Even where the pressures are not so overt, it may be difficult for an assessor to judge the performance of a candidate who is well known to him/her personally. We all have personalities which may clash or gel with those of others. Someone who has rubbed up a teacher/trainer the wrong way may be looked at differently from someone with whom they have got on well, irrespective of the quality of the work. If the outcome of the assessment is very important to a candidate's future, a caring teacher/trainer can easily become overly sympathetic.
Teachers and trainers may also simply be unfamiliar with the range of performance that can be expected. If they are in a small remote school, for example, they may apply different standards from assessors aware of what is possible from the spread of candidates encountered in a city. This can be ameliorated, to some extent, by bringing assessors together, but doing so can be time consuming and costly.
9.4 Does the aggregation of components amount to overall capability?
Another fundamental assumption is that learning history or physics, or training to be an automotive engineer or hairdresser, can be broken down into a number of bits which when accumulated amount to skilled overall achievement or performance. This may be the case on some occasions. In learning to service a car, for example, it seems reasonable to learn to clean the carburettor, balance the wheels and check the ignition separately. Successfully accomplishing the parts will fully service the whole. It is the car itself which brings them together. This may be one of the reasons why the motor industry has taken the lead on unit standards in New Zealand.
But the reliance of the NZQA on analysis is more likely to lead to ex-post facto taxonomic description than be a basis for setting out the requirements of qualifications. Itemising a Shakespearean play, for example, does not tell you how to write it, nor does analysing the acting tell you how to perform it. Rather in the same vein, an analysis of what a plumber, hairdresser or teacher does cannot, of itself, provide a specification of skilled performance.
Often, attempting to set standards becomes a fruitless search for coherent components from which to build a qualification. Early estimates suggested, for example, that 120-150 unit standards would be required for the pre-service diploma in teacher education and by October 1995 over 40 had been registered on the NQF. But a growing sense of unease has prompted efforts to boil them down to 23. In part, this was an attempt to learn from the American experience:
Atomisation was, however, the approach of the American competency-based teacher education programme of the 1970's. These programmes generated hundreds and even over a thousand competencies. It was soon apparent the sum of these competencies may not equal the whole of the teacher. They failed (pp. 5-6).90
The essence of subjects, applied education and occupational training almost seems to disappear in the attempt to express them as numerous performance criteria. It would perhaps be more realistic to try to embody the overall experience or performance in a whole qualification.
9.5 Can levels and credits be assigned meaningfully?
The feasibility of the NQF also rests on being able to assign level and credit so that there is a credible currency for transactions of various kinds. The idea of level has an intuitive appeal, and most of us find it recognisable. It is not difficult to accept, for example, that there is an academic ladder ranging from Form 5 to Ph.D. Similarly we can envisage a series of levels from low-skilled work to professional and managerial positions. The trouble comes when you try to put them together and combine a scale of academic depth, complexity and coverage with one based on autonomy and responsibility in work. Wagner and Sass (1992)91 attempted to do so in a report for the NZQA, but rather than finding a simple pattern they only seemed to discover further difficulties.
We should, therefore, pause to ask whether the effort is worth it. Why not have a qualifications map which reflected qualifications and qualifications ladders working in their own terms, rather than worrying too much about the relationships between them.
It would mean less interchangeability, but present claims for seamlessness are, to a large extent, spurious. The maths being assessed at DB Breweries Limited was very different from that in Form 5 at Epsom Grammar, yet the unit standards carried the same level and credit values. They were important in their own terms but are not interchangeable. Similarly there is no point in accepting someone part way through a university course on the basis of equivalence unless that person knows enough to be able to make a go of it. Neither can one expect someone with an academic qualification to step directly into employment on the basis of credits at a certain level, and expect such a person to make a go of it straightaway.
9.6 Is the split between 'competent/not yet competent' sufficient?
It is the bold claim of the NQF that it is possible to set out criteria so that performance can be judged objectively against standards, rather than comparing people with each other. This is claimed to be fairer and more motivating because obtaining a qualification depends on reaching a defined level rather than competing with others. It is suggested students can be protected from a sense of failure because one is either competent or working towards competence. As noted in Chapter 5, some of the students in the trial schools were not so sure, fearing they could end up with nothing if they did not meet all the performance criteria.
Arriving at standards, as we have already seen, is difficult. When it comes to descriptions for different levels there is often an implied or even explicit comparison with other people's performance, or expected or typical performance, so that criterion-referencing is not as objective as is sometimes assumed. This is even more the case if the standards are illustrated by exemplars of what candidates have achieved. Such comparisons are tacitly assumed by the NZQA in its answer to the question of how the NQF differentiates. It takes the view that the more able students will achieve more units at higher levels faster.
More recently the NZQA has accepted the need to differentiate within units.
How excellence is to be demonstrated has yet to be determined, but rigid adherence
to the standards philosophy will mean that the complications of standards-setting
will be duplicated if 'excellent' and 'pass' are to be distinguished, or triplicated
if there is to be 'excellent', 'merit' and 'pass'. How much simpler to admit
that comparisons between people are involved and to have achievement-based assessment.
How much fairer to standardise as far as possible the conditions of the assessment
through an appropriate examination.
9.7 Can a highly fragmented system be manageable and affordable?
A report from Coopers and Lybrand92 to the Ministry of Education revealed the very considerable costs of the enterprise. In part, these reflect the scale of what is being attempted. Already over 9,000 unit standards have been registered, and it is estimated that 14,800 will be by 1998. If the universities are brought on board that could involve some 9,000 more.
The costs and workload derive in part from the effort required to keep in place a fundamentally unsound structure.
9.8 Can teachers cope with the workload?
The Qualifications Framework Inquiry93 commissioned by the PPTA highlighted the enormous workload falling on teachers arising from the attempted introduction of the new structure. Some of the recently proposed modifications, like excellence within units, will add to those demands.
9.9 Justifications for change
As well as the soundness of the assumptions on which the NQF is based, we also need to consider some of the claims for making the changes in the first place. Chief among these are the academic-vocational divide, norm-referencing, scaling and partial learning.
9.9.1 Academic-vocational divide
Much of recent qualifications reform, not only in New Zealand but throughout the world, has been about de-emphasising differences between academic and vocational learning, and promoting transferability between them. Some, like David Hood,94 deny there is any difference at all:
If you take subjects that are often deemed to be academic, like physics or maths, they are what everyone recognises now as being perhaps the most vocational of all subjects. And where do you put medicine? Is that vocational?
This reveals that there has been confusion at the very top of the NZQA during its formative years, the results of which are still evident in some of the policy development and implementation work now taking place. As we have seen from Sir Ron Dearing's analysis (page 40), it is possible to distinguish different types of learning - which may be called academic, applied and vocational - according to their organising principles. The quote from David Hood implies that he, himself, senses a difference between physics and medicine.
It is not usually, however, organising principles which are at stake but status differences. It is often assumed that the vocational is undervalued in relation to the academic, and that 'parity of esteem' can be achieved by fudging the differences. But this is to forget that the value of a qualification depends on what you can do with it. There is no necessary linkage between academic/vocational and status, as is sometimes assumed. Some of the most highly regarded areas of study and qualifications are vocational. Medicine is full of students who cannot stand the sight of blood, but were attracted to it for prestige reasons. Similarly law, accountancy and engineering, in some countries at least, are high status vocational studies. On the other hand, some academic subjects have lower status than others and certainly lower than law and medicine. (In Tom Sharpe's95 Porterhouse Blue, for example, the Master bemoans the fact that the college has had only one 'first' in 10 years - and that was in geography!)
The widespread impression that there are status differences between the academic and the vocational comes largely from looking at the stage immediately beyond compulsory schooling. The education and training system is a bit like a tree; it carries people of very different abilities, personalities and interests towards occupations that differ considerably in pay, prestige and pleasantness. After the 'trunk' of compulsory schooling, young people progress along the branches to varying heights. Those making their way towards the most sought after occupations, including those involving professional studies at university, frequently continue with academic studies in the period immediately after compulsory schooling; those going directly into work or aiming at support roles will by definition become involved in occupational training at an earlier age.
Thus, among 16-18 year-olds, academic study tends to carry more prestige than vocational study. But that is not intrinsic to the type of learning; it is determined by where the people are heading and what they can do with the qualification. The differences in occupational opportunity will still exist whatever is done to the qualifications structure. Surgeons will still need a higher level of understanding than butchers although some of the techniques they use may be similar.
Education and training should be directly linked through qualifications to employment opportunities, and attempting to fudge real differences would seem an unnecessary distraction. This is not to argue for tramlines; there should be as much transferability and flexibility as possible, with the academic, applied and occupational studies combined as appropriate. Qualifications should, however, represent clear and coherent pathways that mean something and lead somewhere.
9.9.2 Norm-referencing
Another bête noire of the NZQA is norm-referencing. Various straw men of pre-set failure rates, people obtaining qualifications on partial understanding and the inequities of scaling are put up. It is important, however, not to stigmatise a whole approach through the ways it has sometimes been used. Where the coverage of a qualification cannot be reduced to a finite set of clear and transparent standards - and we have argued that this is rarely the case - another frame of reference for assigning achievement levels is required. The typical performance of a large cross-section of an age or grade group, on a standard set of tasks, can provide such a frame of reference against which to judge each candidate's efforts.
It is, in any case, almost impossible to avoid implicit comparisons - even setting the bar in the high jump depends on an expectation of what it is reasonable to achieve; it would be no good, for example, putting it at 10 metres.
Where the purpose of a qualification is to differentiate -- and employers, as well as university admissions tutors, do frequently want to tap into a particular band of the ability spectrum - the fairest way of deciding who had achieved most would seem to be through open competition on a particular set of tasks under standard conditions. It is not only fairer but it also means that the massive load of pre-setting expectations through stating standards can be dispensed with or much reduced. There is no need to guess within rigid limits what a person can do; it is enough to see how well he or she actually does. Comparisons between people are the way we identify talent in sport, music and people, and ultimately in education and training.
In practice in education hybrid approaches are frequently used, with comparisons between people made within the context of attempts to specify grade criteria. There is no guarantee therefore that coming top in an examination will automatically lead to a 'first' or an 'A'. We are not therefore arguing for 'pure' normative assessment; only pointing out that the NZQA has been too negative in its total rejection of it.
9.9.3 Scaling
Those who would discredit norm-referencing sometimes make play with scaling. In fact, scaling is used by most examination authorities around the world and is intended to protect students from the injustices due to the vagaries of examiners. It is a technique by which grades are awarded in set proportions, often with reference to the normal distribution. It can be used to secure apparent consistency between markers, subjects, and performance over time.
If crassly used, however, it can lead to inequities. In England, for example, in spite of O-level being a high hurdle at age 16, the failure rate at A-level was initially set at 30 per cent. Thus even students who had revealed real promise at age 16 could find themselves failing at 18.96 Grades are now awarded in relation to grade-descriptors rather than being subject to quotas, and the pass rate has risen from 68 per cent in 1982 to 87 per cent in 1997. Since entries have also gone up from about 600,000 to 776,000 this has led, in turn, to concerns that standards were not being maintained over time.97
Scaling is nevertheless a very useful tool in the examiner's armoury. Even where it is said to have been abandoned it still persists. Irwin98 cites Elley as revealing that in the marking of the School Certificate, the NZQA prepares a provisional marking guide, marks a random selection of scripts, checks the distribution of marks, and then adjusts the marking guide so the distribution of marks is more or less the same as in previous years.
9.9.4 Partial learning
Traditional approaches to assessment are frequently chided by advocates of unit standards for allowing candidates to pass when they are only partly right, for example only obtaining 50 per cent of the marks. Unit standards, it is said, ensure complete coverage and mastery learning. Credits are awarded if, and only if, the learner has demonstrated competence in relation to all of the criteria.
But is this necessary, appropriate or even remotely achievable? Whereas with astronauts, electricians and the London taxi driver's knowledge it is possible to delimit a domain of learning in which everything has to be known and, in some cases, life itself depends on it, most learning is not like that. It is open-ended. To pretend that everything can be encompassed and assessed by the performance criteria is to ignore the implicit sampling of the student's knowledge and understanding which inevitably occurs. On occasions, the advocates of mastery learning fool themselves. For example, in an art and design qualification in the United Kingdom, essentially on unit standards lines, the whole of European art history was tested by just one question. This was claimed to demonstrate understanding of the whole field, completely ignoring the extent to which countries and periods had had to be sampled to arrive at that question.
A mastery learning approach can have a place in diagnosing what a student can or cannot do, or does and does not know (vide the views of teachers at Epsom Girls and St Cuthbert's Schools noted in Chapter 5), but it is unnecessary and impractical for many qualifications. It is more realistic to acknowledge that there has to be sampling.
Norm-referencing, scaling and partial learning have all been used as arguments against comparative assessment:
A standards based system was introduced to improve upon the worst features of an examination system founded on a selective mechanism of assessment leading to: