USING THE KIRKPATRICK EVALUATION MODEL FOR PRE-SERVICE TEACHER EDUCATION COURSE

Preparing future teachers for teaching practice is a way of professional selfdetermination and realization of a beginner career path. Successful completion of the preservice teacher education course is important for both teacher candidates and stakeholders. In our quantitative study, we used the Kirkpatrick evaluation model to determine the effectiveness of the course targeting curriculum shifts in digital education. The data show that positive student attitudes and good learning outcomes demonstrate a high probability of wellorganized course content and trainer self-efficacy. Introduction The stage of preparation for work is aimed at forming the professional identity of teachers [1]. Student years are a period of intensive formation of professionally important qualities, consciousness, and identification. Conversely, studying at a university may be accompanied by a rethinking of professional intentions, adaptation and life plans, or disappointment and loss of interest in a choice made once. Studies of the problems of first-year teachers [2] show that pre-service teacher training is observed since more theoretical and practical experience during training coincides only with specific school conditions for actual learning. Future teachers may encounter problems and obstacles during their practice due to a contradiction in their understanding of rapidly changing educational reforms, attitudes and perspectives. In addition, as mentioned by Brock & Grady (2007), after graduation, the novice teacher is expected to act as an experienced teacher who knows about classroom management and modern, effective teaching methods. According to Veenman’s study in 1984 [3], following consequences of preservice teacher education shortcomings are manifested for employed teachers as to how to manage classroom, effective ways to motivate students for learning, adequate assessment of student learning performance, and good communication skills with parents. It seems as though aforecited concerns of the XX century are relevant in contemporary time in terms of the digitalization of learning and teaching environment. Therefore, there is food for thought related to pre-service teacher training education where a crucial question emerges towards how well student teachers are trained and prepared for teaching in digital world. Some studies [4] reveal a December 11, 2020  Amsterdam, The Netherland  69


Introduction
The stage of preparation for work is aimed at forming the professional identity of teachers [1]. Student years are a period of intensive formation of professionally important qualities, consciousness, and identification. Conversely, studying at a university may be accompanied by a rethinking of professional intentions, adaptation and life plans, or disappointment and loss of interest in a choice made once. Studies of the problems of first-year teachers [2] show that pre-service teacher training is observed since more theoretical and practical experience during training coincides only with specific school conditions for actual learning. Future teachers may encounter problems and obstacles during their practice due to a contradiction in their understanding of rapidly changing educational reforms, attitudes and perspectives. In addition, as mentioned by Brock & Grady (2007), after graduation, the novice teacher is expected to act as an experienced teacher who knows about classroom management and modern, effective teaching methods.
According to Veenman's study in 1984 [3], following consequences of preservice teacher education shortcomings are manifested for employed teachers as to how to manage classroom, effective ways to motivate students for learning, adequate assessment of student learning performance, and good communication skills with parents. It seems as though aforecited concerns of the XX century are relevant in contemporary time in terms of the digitalization of learning and teaching environment.
Therefore, there is food for thought related to pre-service teacher training education where a crucial question emerges towards how well student teachers are trained and prepared for teaching in digital world. Some studies [4] reveal a technological lack of advanced proficiency of teacher candidates to work in digital future classrooms. The characteristics of a teacher in terms of digitalization requirements today include concepts such as digital skills, digital literacy, digital competencies, and digital culture. In the case of teacher candidates preparation is the vital segment to see the effectiveness of learning process. In educational studies, a variety of methods are used to assess the effectiveness of course completion. An important step in the process of determining the effectiveness of pre-service teacher training is the establishment of assessment metrics. As a classical model of appraising training courses D. Kirkpatrick model has been used to assess the whole system of training effectiveness.
Theoretical framework In 1954, Donald Kirkpatrick's proposed a formula for describing the learning cycle: reaction -learning -behavior -results in his Ph.D. thesis "Evaluating the effectiveness of program management" at the University of Wisconsin (USA). He discovered four-level-evaluation model which includes vital components for appraising any training courses from participants' satisfactions to beneficial aspect of the efficiency training program.

Pic. 1. The New World Kirkpatrick Model, Kirkpatrick Partners, 2011
Kirkpatrick's first papers focused further research in the area of learning effectiveness evaluation (mainly at levels 1 and 2). In the 1970s, the four Kirkpatrick levels were already widely used by many organizations around the world, over time they were formalized into the Four Levels TM Evaluation Model and adopted as the standard for evaluating professional training. Throughout the 1980s, many different assessment methods and tools were developed, but practitioners continued to focus on levels 1 and 2 [5].
It was not until 2005 that Donald Kirkpatrick proposed a Level 3 Assessment Tool (Behavior), which he described in Transferring Learning to Behavior, coauthored with his son, Dr. James D. Kirkpatrick. Ensuring the effective application of learning outcomes in real activities (transfer of knowledge and skills) continues to be one of the most important tasks for many training organizations today.
In 2006, in the third edition of Evaluating Training Programs, Kirkpatrick greatly expanded the scope of the four-level model. It now focuses on Level 4 (Results), so the model can be used to measure the effectiveness of both training programs and the change management process, and also to demonstrate the business value of the learning function as a whole [6].  According to the developers of the new concept, in order to guarantee results, it is necessary to determine before the start of training: -Expected results; -Key indicators; -Measurement and evaluation methods. The first level of evaluation model lies on students' reaction to the program, their attitude, perception and satisfaction, what they feel related to course procedure, objectives, settings, atmosphere, and the significant point, its relevance [7] to further job satisfaction. It is criticized to be unlikely to show great accountability of the course effectiveness but it may highlight student needs, decision making in successful course completion [8]. This level embodies one of the main issues of human resource (HR) is student needs in learning contentment and assessing Abraham Maslow's hierarchy of needs (1943) can provide a clue to the dominant trend in intrinsic and extrinsic factors of their behavior. It can be used standardized questionnaires.
The second level of evaluation model is Learning. Learning is defined as changing attitudes, improving knowledge and i skills of participants as a result of their completion of the training program. Kirkpatrick argues that a change in the behavior of participants as a result of training is possible only when learning occurs (attitudes change, knowledge or skills improves). Level 2 assesses the following elements: Knowledge and skills (I know it and I can do it), Attitudes (value orientation), Confidence (I know how to do it, high self-efficacy), Commitment (willingness to apply job-related-skills).
The third level is Behavior. At this level, an assessment is made of the extent to which the behavior of the participants has changed as a result of the training. Kirkpatrick points out that the absence of changes in the behavior of the participants does not mean that the training was ineffective. Situations are possible when the reaction to the training was positive, learning occurred, but the behavior of the participants did not change in the future, since the necessary conditions were not met for this. Following elements: Critical behavior (critical skills); Required Drivers (reinforcement, support, remuneration, monitoring); On-the-job training.
The fourth level Results includes the changes that have occurred due to the participants being trained. As examples of results, Kirkpatrick cites increased productivity, improved quality, fewer accidents, increased sales, and decreased turnover. Assessment at level 4 "Results" determines: Desired results.
Kirkpatrick believes that in most cases the assessment is limited to the use of post-training questionnaires -examining the participants' immediate reactions to the training. He calls these questionnaires "smile-sheets", meaning that most often participants use a questionnaire to express gratitude.

Methodology
In 2019 when we talked with student teachers enrolled in professional oriented foreign language class about their field experiences in actual teaching most relayed frustration of "Reality Shock" at school settings in the light of curriculum changes however they expressed their positive attitudes toward new teaching system. This feedback leaded to a big discussion with colleagues about student attitudes to the present situation in contemporary Kazakhstani education and how it determines their further decision making in professional identity. So, we have decided to create a . teacher training course targeting core idea of curriculum shifts in digital education Kazakhstani context. Branch claims that entirely well-organized course brings more effective output paradigm in learning environment [9]. However, designing a course is an insufficient level for completely transparent and adequate accountability of the course evaluation.
The focus of the current study is timed to insight into final-year student teachers perception of curriculum changes respectively in digital education. Measuring explicit understanding of curriculum policy from teacher students' perspectives depicts transparent evaluation of pre-service teacher training program (PSTTP) as determination of relevance graduate preparation accountability.
Therefore, designing effective courses in preparation pre-service teachers oriented toward relevant issues in education is a big challenge for university academicians, scholars, and university staff.
The questionnaire included a number of questions in order to evaluate the effectiveness of the course. It was carried out using the Donald Kirkpatrick's fourlevel training evaluation model, adapted to the realities of the course. Aside from students' attitude to the digital study, the questionnaire included a question about the problem of so-called professional burnout or disappointment in the chosen work by future teachers. The peculiarity or limitations of this study was that the objects of the study were students who are still doing their Bachelor degree on education. However, they were in practicum in school settings and had the opportunity to put into practice the knowledge they gained during the course. Based on this information, a questionnaire was designed and this study was conducted.

Materials and Methods
This research is conducted by using the quantitative design because a study on the concept "attitude" refers to positive or negative scales and shows premises of individuals' feelings, perceptions throughout their behavior [10]. Therefore, quantitative measures are utilized to collect essential data. Participants of this study are teacher education candidates aged 20-22 of foreign languages department in Buketov Karaganda State University.
The evaluation model of participants' understanding educational policy was selected in compliance with K.Phillips's tips [11] on designing valid data of Evaluation Level 1. As mentioned by him the validity of the survey questions should be concentrated on the following principles such as actionable insights at least for decision making to the further development of effectiveness training courses; learnercentered approach rather than trainer-centered perspectives to identify adequate participant's needs and to provoke unseen data; applying mixed both qualitative and quantitative measures for cross-validation; thorough approach for designing Scaling questions.
The survey carries out conformity of teacher training education at faculty of foreign languages and the main issues which are leaded by basic tenets of updated secondary curriculum. Students were asked to complete attitude questionnaire of five dimensions which are key issues in updated curriculum such as The Spiral Curriculum Design, Educational Curriculum Framework based on Bloom's Taxonomy, Leadership and Management Development Framework, Goal-Setting and Task Performance, and Assessment in Curriculum Framework.
In this study, the effectiveness of the course was expressed in measuring the percentage change in the correct answers to questions regarding the knowledge of innovative teaching methods as a part of the modern content of school education.
. The hypothesis in this study is to check whether the percentage of correctly answered data is the same before and after completing the course, whether the difference between the means and medians in the two related samples is statistically significant, or these deviations can only be attributed to random deviations of the means. For these purposes, a nonparametric mean comparison test for paired samples of the Wilcoxon test was applied.
Statistical methods of analysis in the research Nonparametric test comparing the average percentage of correct answers for paired samples Wilcoxon test, this test measures the change in average ranks in two related samples.

Wilcoxon Signed Ranks Test
The Wilcoxon test for related samples is a nonparametric statistical criterion is utilized for comparison two related (paired) samples according to the level of a quantitative indicator, measured on a continuous or ordinal scale. The Wilcoxon Ttest is used to assess the differences between two series of measurements made for the same set of subjects, but under different conditions or at different times. This test can identify the direction and severity of changes -that is, whether the indicators are more shifted in one direction than in the other. A classic example of a situation in which the Wilcoxon T-test for related populations can be applied is the "before-after" study, when the pre-and post-treatment parameters are compared. For example, when studying the effectiveness of an antihypertensive drug, blood pressure is compared before taking the drug and after taking it. The Wilcoxon criterion is a nonparametric criterion; therefore, unlike the paired Student t-test, it does not require a normal distribution of the compared populations.

Authors' design
As can be seen from the table 1 with the results of the Wilcoxon Signed Ranks test, the percentage of correctly specified answers increased statistically significantly after completing the course, which indicates the effectiveness of the course in introducing innovative methods into the work of the teacher.

Conclusion
Quality teaching takes a fundamental basis in pre-service teacher preparation. Understanding teacher education reforms plays an essential factor in quality teaching and student learning. According to this motive, course evaluation should play a significant role in the learning process to achieve desirable outcomes and remarkable results. However, it is perceived as a trivial component and mostly neglected. By controlling the course effectiveness, trainee can monitor the transparent way of expenses on successful goal and objective achievements, student engagement and performance, teaching quality and productivity and others. When we talk about an educational institution, it is usually focused on the first two levels of performance assessment, while the customer of training is interested in assessing the effectiveness at the third and fourth levels. Therefore, the reorientation of educational organizations in assessing the effectiveness from the first and second levels to the third and fourth levels, i.e. from consumptive metrics to impact metrics, is a key task of aligning stakeholder interests. Studies show that there is a positive correlation between level 1 and level 2. That is, if a participant is satisfied with the educational program, it can be said that the level of acquisition of knowledge and skills in accordance with the results of the program increases with a high probability. There is a similar relationship between the third and fourth levels: the more the program graduates apply their knowledge in practice, the greater the impact on the overall results of the training program. However, research does not find a significant correlation between levels two and three: training does not always lead to targeted employee behavior.