The contribution of metaevaluation to the development of evaluation theory and practice : an exploratory study

The thesis arises from a belief that more attention needs to be given to methods of improving evaluation theory and practice. Metaevaluation - or the evaluation of evaluation - would seem to be a potentially significant element in the determination of evaluation quality, which is accepted as a major metaevaluation criterion. The study is concerned, therefore, with demonstrating the extent of this potential by evolving an approach to evaluation based on insights emerging both from the research literature and from the situation in which the study is conducted. This approach, implemented through a case study of a primary evaluation which is in turn evaluated, produces a range of conclusions and implications of relevance to the betterment of evaluation. The study is significant because it is the first formal metaevaluation, in an Australian post-secondary context at least, of an evaluation of a course innovation, using the Joint Committee Standards as a basis for assessment. As such, the study initiates an approach to metaevaluation, and in so doing contributes to knowledge of how metaevaluation relates to evaluation in a practical setting. It also derives a number of conclusions and implications for the conceptualisation of metaevaluation, and suggests directions for further research. The review of research indicates the uneven development of evaluation theory and practice in recent decades, partly as a result of proponents of 'traditional' and 'alternative' paradigms defining or redefining their respective positions in reaction to each other. This led to the proposition that one's epistemological assumptions in some way determined the research method or evaluation techniques available; further analysis revealed that one's methodology is not pre-determined in this manner, because of the absence of a sustainable - logical or causal - link between ideology, paradigm, and method-type. Acceptance of this point of view facilitated the adoption of a case study approach to the design and implementation of the evaluation and metaevaluation which formed the case study element of the overall study. Evidence of the need for an exploratory study of the type reported on in this thesis, was found in the literature and particularly in requests by the Joint Committee on Standards for Educational Evaluation, concerning the need for research efforts which would test the validity and utility of current metaevaluation standards in field settings. In general terms, the problem examined in the study was proposed as follows: 1. What is the contribution of metaevaluation to the improvement of evaluation theory and practice? 2. What is the most appropriate method of facilitating this investigation? The case study method suggested in the review of The case study method suggested in the review of literature, and suited to the context of the overall research study, employed the techniques of questionnaire, interview, and checklisting to gather data. The analysis of these data in turn produced insights of relevance to the development of metaevaluation procedures, as well as to their role in the improvement of evaluation. The results of the case study were presented and analysed with the outcomes of the formative evaluation being recorded by means of a summary of the periodic evaluation reports prepared during the three years of the case study. At the end of this 'formative' period, an external metaevaluation assessed the impact and validity of the primary evaluation, and this was supplemented by an internal metaevaluation using a checklist based on the Joint Committee Standards referred to above. Finally, the results of this analysis were related to the focus provided by the literature review and the research design for the study. As a result, it was concluded that metaevaluation had demonstrated its potential for monitoring and assessing the quality of a particular evaluation, and as well, had indicated its contribution to the development of evaluation theory and practice. Insofar as this had been established, the purpose of the study was achieved.