RETROSPECTIVE EVALUATION OF ELT COURSEBOOK EFFECTIVENESS

The right choice of a coursebook makes both teaching and learning experiences effective and enjoyable. However, the real strengths and weaknesses of the chosen teaching material become apparent only over a period of continuous use. This article looks into the way coursebook effectiveness can be evaluated by the participants of the educational process. The importance of retrospective assessment of a coursebook is emphasised, considering the fact that it enables to evaluate the work done and make the appropriate decisions for the future courses. The evaluation should be performed not only by teachers, but also by students. For this purpose, two different assessment forms have been developed. The author stresses that learners and educators view coursebooks from the various standpoints, that’s why their evaluation forms should include different sets of parameters. The principles for creating the questionnaires for teachers and students are outlined in the present work. The article suggests that the most effective way to conduct a coursebook evaluation is to use free data mining web tools which not only streamline the evaluation process, but also enable to effectively process the results of the survey. The author outlines the process of creating interactive questionnaires using one of such web tools. The last part of the article features description of the practical implementation of the suggested evaluation approach. The developed questionnaires have been used to evaluate Speakout Intermediate coursebook with which a general course of English was taught to a group of adult learners. The results of the survey and possible implications thereof are discussed in the article.


Introduction
Coursebooks are often considered by teachers and learners to be key elements of the language tuition.According to Mukundan (2011), "as a part of the materials used in the language classroom, the textbook can often play a crucial role in students' success or failure" (p.100).The learning outcome very much depends on the quality of a coursebook.
The scientists have expressed various opinions as to the factors that determine the quality of a coursebook and ways to evaluate it.In Ukrainian academic literature, the analysis of coursebooks has been presented (e.g., Burynska, 2003;Kodluk, 2003;Lukina, 2004;Prokopenko, 2003;Savchenko, 2013;Shamelashvili, 2008;Zhuk, 2009, and others).Burynska (2003) defines content, structure, design, polygraphy, ergonomic factor as the key parameters for evaluation of the coursebook (p.8-10).Kodluk (2003) characterizes descriptive, comparative and matrix methods of coursebook analysis (p. 74).Prokopenko (1999) showcases the method of lexical quality measuring for assessment of deductive materials.Zhuk (2009) outlines 3 models for quality assurance of a coursebook: 1) expert evaluation of the manuscript; 2) experimental approbation of several test coursebooks in the real teaching process; 3) fullscale use of coursebooks in educational activity (p. 7).The analysis of Ukrainian scientific resources shows that the research into the methods of a coursebook assessment has hardly had any practical implications in the form of checklists or questionnaires, has been mainly conducted in the context of secondary education and without a special focus on ELT field, which proves the relevance of the present article.
In foreign linguistics researchers have looked into a wide array of theoretical and practical aspects of ELT coursebook assessment that can be conducted in various teaching contexts and stages of a coursebook use, as well as developed a comprehensive terminological apparatus which we are going to apply.In this article, we are going to use the term 'evaluation' as a synonym to 'assessment' and opposite to 'analysis'.According to Tomlinson (2014), evaluation focuses on the users of the materials and makes judgments about their effects, whereas analysis has an emphasis on the materials, and results in a description of what the materials contain.Unlike analysis, which aims to be objective, evaluation is always subjective and is determined by the concrete context and requirements (p.16).Ellis (1997) discusses the differences between the following types of evaluation: 1) predictive and retrospective (done before or after the use of a teaching material); 2) micro and macro (depending on the number of participants involved); 3) impressionistic and empirical (based on teachers' impressions how effective and enjoyable activities have been, or on users' feedback) (p. 36-37).

Actuality and objectives of the article
The present article deals with the retrospective empirical evaluation performed at the micro level.In our opinion, retrospective evaluation is especially important, since a coursebook's adequacy can be most effectively assessed after the course has already been completed.Moreover, it "provides the teacher with information which can be used to determine whether it's worthwhile using the materials again, which activities 'work' and which do not, and how to modify the materials to make them more effective for future use" (Ellis, 1997, p. 36).We agree with Mukundan (2010) who points out that there is the lack of the literature devoted to retrospective evaluation as opposed to predictive one and maintains that the first should be an essential part of teachers' professional development (p.349).Unlike impressionistic evaluation, which results in favouring materials that have face validity (meet people's expectations as to how the materials should look like) and are visually appealing (Tomlinson, 2014, p. 5), the empirical one gives a more versatile and objective picture of users' experience.Doing a research at micro-level enables to look into the problems and requirements of concrete users and is, therefore, more practical and meaningful.
In order to facilitate effective and delightful language instruction, a coursebook should meet both teachers' and students' expectations.However, the analysis of scientific sources has revealed that most surveys are conducted among teachers, whereas learners' opinions are hardly ever taken into account.We believe that students' feedback cannot be neglected and should play a crucial role in decision-making regarding the future use of a coursebook.Therefore our first objective is to develop forms for both educators and students to evaluate their teaching and learning experiences, as well as stipulate the different principles underlying their creation.Secondly, both in Ukraine and abroad very little attention has been paid by researchers to the way a coursebook assessment process can be managed.Due to that reason, we would like to suggest a means of doing it in an efficient and interactive way, namely using Google forms.Thirdly, we would like to present the results of the analysis of Speakout Intermediate effectiveness carried out using the above-mentioned online tools.

A wide variety of evaluation criteria
In our opinion, a good questionnaire is supposed to help establish the level of correspondence between a coursebook and the needs of its users, as well as point to the fields where this level is low.There are numerous parameters by which the adequacy of a coursebook can be evaluated, starting from the communicative intent and skills contextualization to the quality of print and availability of the supplementary resources.
It is widely agreed in the scientific literature that a coursebook evaluation should touch upon both physical characteristics, such as design, layout, visual attractiveness, durability, organisational characteristics, quality of teaching materials, etc., and methodological ones, namely aims, approaches, integration of skills, communicative intent, how teachable the material is, promotion of critical thinking, instructional appropriacy, etc.It is also crucial what syllabus is covered by a coursebook, what skills are developed and how they are practised.That is why questionnaires may include the items dealing with vocabulary, grammar, communicative output, balance of activities, provision of review and testing.Other important parameters that should be incorporated in a questionnaire assess how interesting the topics in a coursebook are, hence the following criteria: the level interest, meaningful contexts, fulfillment of students' objectives, ability to motivate, being up-to-date, etc.The questions related to a coursebook authenticity, additional resources, cultural and gender characteristics are also included by some researchers but are way less common.
At the same time, we agree with Cunningsworth (1995) that the number of criteria assessed should be restricted to manageable proportions, "otherwise we risk being swamped in the sea of details" (p.5).Having analysed various checklists suggested by Cunningsworth (1995), Harmer (1991), Miekley (2005), Mukundan (2010), Rubdy (2003), Skierso (1991), Tok (2010), Ur (1996) and other researchers, we have come up with the lists of criteria that fit our specific context, which is as follows.The evaluated coursebook (Speakout Intermediate) has been used to teach the general course of English to adults aged from 23 to approximately 35.The main aim of the tuition was to harmoniously develop all the language skills with a particular emphasis on speaking.The coursebook was supposed to streamline teaching and learning by providing structured and actual content.Besides, the interactive workbook (MyEnglishLab) was used by teachers and students, therefore a criterion assessing this experience was included in both evaluation forms.

The two different questionnaires
Since teachers and learners view a coursebook from different perspectives, the questionnaires evaluating their experiences cannot be the same and should include different sets of parameters.Teachers who have worked with the coursebook are supposed to carry out its comprehensive evaluation from a pedagogical standpoint.The form should enable them to analyse a wide variety of factors determining the quality of a coursebook.That's why the suggested evaluation form for teachers includes more factors of assessment, which are quite detailed and feature the relevant teaching terminology (Table 1).The teachers' questionnaire comprises 30 criteria split into 5 categories.

Table 1. Teachers' questionnaire Categories Criteria
Aims and approaches 1.
The coursebook is based on communicative approach and facilitates attaining communicative competence.

2.
The target language is contextualized.
The coursebook facilitates various forms of group work.

5.
The content is logically sequenced and teachable.6.
The material in the units is logical and well-organised.
Design and technical quality 7.
The quality of paper and printing is acceptable, the size and binding of the coursebook are appropriate.8.
Illustrations in the coursebook are qualitative and relevant.9.
The quality of the audio and video is acceptable.The students' questionnaire (Table 2) differs from the teachers' form both in the quantity of the items and in their content.On the one hand, the criteria in students' form are less specific than in the teachers' one and are worded with mostly general vocabulary.On the other hand, learners' questionnaire comprises half as many items as the above-mentioned form, which is rational considering the fact students are less likely to fill in too long forms.Besides, some criteria are inapplicable for students, namely the ones included in 'Aims and Approaches' category of the teacher's questionnaire.Structural parameters are also more relevant for teachers, that is why they should not be included in learners' form.As to the aspects related to design and technical quality, in students' form they are limited to one criterion 'I liked the design of the coursebook and its illustrations'.In terms of the content of a coursebook, students' questionnaire touches upon only the most decisive factors influencing learning experience.The only area of control which is almost equally represented in both the questionnaires is related to the skills.
The topics of the units were interesting and actual.2.
The content was practical and up-to-date.

3.
The number and content of grammar activities met my expectations.4.
The number and content of vocabulary activities met my expectations.

5.
The functional language presented and practised in the 3 rd section of each unit was actual and useful. 6.
The number and content of listening activities met my expectations.7.
The video activities were interesting and useful.8.
The number and content of reading activities met my expectations.9.
The writing activities were relevant to my needs and effective.10.The speaking activities were engaging and effective.11.Doing homework in the interactive workbook (MyEnglishLab) was handy and useful.12.The number and content of the activities in (MyEnglishLab) met my expectations.13.I liked the design of the coursebook and its illustrations.14.I use the language which I learnt from the coursebook in the real-life communication.15.I'm happy with the progress I made with the help of this book.
The Likert-type scale has been used for evaluation.The factors in both the questionnaires are rated numerically, on a scale from 1 to 5, which are equal to, correspondingly, 'strongly disagree' and 'strongly agree'.This correlation with numbers makes the results of the survey more tangible.Besides, in each form, there are some open-ended questions for which respondents can provide more details concerning the criteria in question.In teachers' questionnaire, they follow each category, e. g. 'If you have any comments as to the structure of Speakout Intermediate coursebook, please share them here'.In students' form, there are two final open-ended questions: 'Please comment on what you particularly liked / disliked about Speakout Intermediate coursebook'.The rationale of open-ended questions is that they enable to acquire additional information which falls beyond the scope of the included criteria, make the evaluation more personalized.

Facilitating a coursebook evaluation with online forms
Nowadays getting feedback can be empowered by really efficient web tools, for instance, Google Forms.This free online application enables to streamline data mining and facilitates processing of the filled in forms.The online questionnaires can be created on Google Forms site featuring various types of questions and attractive formatting.Having clicked on 'Preview', you will be able to see the questionnaire in the form available for the survey participants.In the tab 'Responses' there are the answers of the survey respondents summarized as bar charts or presented individually.Pressing on a sign 'Create a spreadsheet' opens the answers in the format of a table.

Evaluating the effectiveness of Speakout Intermediate coursebook
The coursebook in question (Speakout Intermediate) was used for approximately a year as the main teaching material at several groups of adult learners.Following the completion of Intermediate course, the evaluation of the coursebook has been carried out in order to check to which extent it proved to be efficient and enjoyable for teachers and learners, facilitated evolvement of the key linguistic skills and enabled attaining communicative competence.The developed forms were supposed to structure users' feedback and reveal the strongest and the weakest areas of Speakout Intermediate coursebook.
The survey has been conducted at the micro level, which according to Ellis (1997), "can serve as a practical and legitimate way of conducting an empirical evaluation of teaching materials" (p.37).The effectiveness of Speakout Intermediate coursebook has been evaluated by 4 teachers and 21 students using the teachers' form and students' form.The results of the survey among the teachers show the high overall satisfaction level, which amounts to 94% for all 30 criteria.Percentage distribution among the 5 categories outlined in the teachers' form is as follows: aims and approaches -96%, structure -100%, technical quality and design -95%, content -89%, skills 89%.In the open-ended questions teachers have made the following positive comments:  MyEnglishLab saves teachers' time;  MyEnglishLab enables to visualise students' progress;  listening activities are authentic and challenging;  the skills are balanced.The aspects of Speakout Intermediate which got the lowest rating (70%) concerned the selection of the topics and effectiveness of the video.Teachers' comments revealed some more areas of potential improvement of Speakout Intermediate coursebook:  the ways of presenting grammar aren't varied;  not all the vocabulary topics are challenging for students (e.g.Internet vocabulary in Unit 8.2.);  sometimes it seems that the coursebook is aimed at promoting BBC programmes;  lexis in the vocabulary bank isn't practised enough.
As to the students, their overall level of satisfaction with Speakout Intermediate coursebook amounted to 85%.The most highly evaluated qualities of the concerned coursebook were related to its speaking activities and design (92% and 95% respectively).As to the answers to the last two open-ended questions, there were some positive comments:  tasks in MyEnglishLab go from easy to more complex and it's very good;  fast feedback from MyEnglishLab is positive;  I liked topics, design, video activities.Despite some positive comments, the criteria 'Doing homework in MyEnglishLab was handy and useful' had the lowest rating of 78% among students, followed by 'I use the language which I learnt from the coursebook in the real-life communication' with 79%.Students' negative remarks also partially concerned the use of the interactive coursebook:  sometimes the coursebook was not challenging enough;  MyEnglishLab is buggy;  sometimes MyEnglishLab didn't recognize alternative right answers.

Discussion
The survey has shown that both teachers' and students' evaluations of Speakout Intermediate coursebook report the high satisfaction level.It is also notable that no evaluations reported results below 70 %.This proves the effectiveness of the surveyed coursebook.As to the areas with the lowest rating, as well as negative comments, they should be assessed critically in terms of possible improvements.
Teachers' remarks related to the selection of the topics are quite predictable, considering the revenuedriven policy of publishing teaching materials, which is mostly pursued nowadays.The topics of the units in Speakout intermediate coursebook are universal, conventional and neutral, for instance: family, personality, movies, technology, life in the future, feelings, history, etc., which makes them acceptable for a very wide audience, but at the same time might seem uninteresting and irrelevant.In our opinion, in such cases, a teacher can modify the activities offered by a coursebook in order to make them appropriate for the interests of learners.
Regarding the video activities, it's undoubtedly positive that they are authentic, qualitative and thematically linked to the topic of each unit.However, the fact that they are short (don't exceed 4 minutes), sometimes contain little language (as it's the case with the video in unit 2.4.) and feature during-watching tasks which are not challenging enough, might prompt teachers to include alternative video materials into their lessons.
The mixed feedback on the use of the virtual workbook proves that the system might save users' time and provide a new level of user experience, but still needs to be intensively tested and improved.The areas which need most attention include elimination of technical glitches and expanding the sets of alternative correct answers.
The relatively low score of the criterion related to the students' use of the learnt language in the real-life situations might indicate a necessity for a teacher to find out more about their students' needs and expectations.It's sensible to conduct such surveys regularly and to use the obtained information in order to actualize the teaching materials or to include in the program some extra-curriculum topics.

Conclusion
The suggested online forms can be used to efficiently evaluate the effectiveness of a coursebook and get feedback on the quality of teachers' and learners' experiences.The combination of qualitative (the criteria scored numerically) and quantitative assessment (open-ended questions) enables to gather both formal and informal feedback.The ease of creation and application, the efficiency of data processing and opportunity to visualise the obtained information make them a useful assessment tool.
It should be noted that the presented survey on the effectiveness of Speakout Intermediate coursebook is limited to the 4 teachers and 21 students, which is why its results might be verified by similar surveys at the macro level.The suggested approach can be extended over other teaching contexts.Further studies might include the development of interactive evaluation forms, aimed at gathering feedback on students' satisfaction with their teacher, lessons, progress, etc.
Content10.The topics are of relevance and interest to the students.11.The presented language is authentic and real-life.12.The content of the coursebook is practical and up-to-date.13.The balance between listening, speaking, reading, and writing skills development is appropriate to our particular learners.14.There is a sufficient number of activities for both controlled and free practice.15.The ways of presenting and practicing target language are quite variegated.16.The tasks fit the learners' ability.17.The tasks for the activities are clearly formulated.18. Teaching notes in the teacher's resource book are helpful.19.Photocopiable activities in the teacher's resource book are interesting and useful.20.The coursebook contains an adequate number of revision activities and tests.21.The interactive workbook (MyEnglishLab) is a handy and effective tool to track students' progress.22.The scope and quality of activities in the interactive workbook (MyEnglishLab) is appropriate.