LONG-TERM RESEARCH ASSESSING IN THE HUMANITIES : A CASE STUDY

It is common practice in academic research projects to present final assessment by means of reporting and evaluating results. More often than not, these reports tend to focus on the quantity and quality of the knowledge acquired, and very little (if any) attention is given to the impact the research may have had on the researchers themselves. In an institutional setting where students participate in research, this may be a missing link in the educational chain. The case study described here tried to fill this gap by showing the changes effected by a project investigating literature and language from a multicultural perspective. More than stimulating research by students in different countries, it aimed at helping them become autonomous researchers using empirical methods. We argue that instead of lectures and seminars, the educational quality of the humanities will be greatly enhanced if we turn to our students doing research themselves. We counter the argument that students are not able to do so with reference to a ten-year project we were engaged in, together with our students, between our three universities. We describe the methods along which our students cooperated with each other in conceptualizing, carrying out and reporting on their research. Feedback from 53 participants indicates that not only did the project motivate them to pursue further academic degrees, but the skills they acquired considerably influenced their future in many different fields. Though the design of this study does not allow us to determine any causality in the effects of the program, we argue that the results are nevertheless highly suggestive of the program’s both motivational and intellectual success.


Introduction.
It is rather taken for granted that any research project is not complete without a final assessment where its contribution to the area is described and discussed.In the humanities, these considerations tend to focus on the quantity and the quality of the new knowledge produced.Rarely, if ever, do reports discuss the impact the project may have had on the researchers themselves.We hold that in an educational setting where students participate in research projects, long-term assessments of this kind are much needed.The concern should be given not only to what is produced in terms of knowledge, but also on how far the pedagogical strategy of learning by research affects students.In this sense, this article shows the reactions of students who participated in empirical research for some years.Their reactions have been made available to us ten years after they left the project.This case study holds evidence enough to enable us to argue for the need of long-term evaluations of research projects in an institutional setting.
According to Levitt et al. (2010, p. 1), "[t]he concept of research impact is regarded as controversial and causes strong reactions among many academics, some of whom prefer to consider the 'value' of arts and humanities research rather than its 'impact'".The authors hold that the research impact includes several changes: in access to research; in the extent to which research is considered, referred to or read; in knowledge and understanding; in attitudes and beliefs, and in behaviour (idem, p. 2).
The impact of research in the humanities is in many ways more complex, as it is a multifaceted and vast area which brings together many disciplines with different backgrounds and traditions.In broad terms, however, they share at least one aim: that of examining human culture, its production, its dissemination and its meaning.Basically, the humanities study the way human experience in the world is processed, produced and recorded, which includes languages, arts, religions, history, etc.
How to go about this investigation in universities has been the object of debate, since emphasis has been put on more rigorous methods (see van Peer, 2011).The tradition, however, is to follow more conventional research practices which differ from those of the natural or social sciences.Typical of this posture is the announcement on the Stanford University website: An Interpretive Approach to Research.A hallmark of humanistic study is that research is approached differently than in the natural and social sciences, where data and hard evidence are required to draw conclusions.Because the human experience cannot be adequately captured by facts and figures alone, humanities research employs methods that are historical, interpretive and analytical in nature.Professors who engage in humanities research are often posing questions about common assumptions, uncovering new meanings in artistic works, or finding new ways to understand cultural interactions.This type of inquiry can produce clearer pictures of the past, uncover the many insights that we can draw from our forbears, and in turn, help us better to prepare for the future.
This view of research methods has been recently challenged by empirical scholars.Although from a more radical stance and more directed to literary studies, Gottshall ( 2008) is correct when he holds that "[w]hile most other fields gradually accumulate new and durable understanding about the world, the great minds of literary studies have, over the past few decades, chiefly produced theories and speculation with little relevance to anyone but the scholars themselves".Joining forces with other empirical scholars (van Peer et al., 2012), he argues that methods in the humanities should come closer to those in the sciences.In his words, scholars should "apply science's research methods, its theories, its statistical tools, and its insistence on hypothesis and proof.Instead of philosophical despair about the possibility of knowledge, they should embrace science's spirit of intellectual optimism.If they do, literary studies can be transformed into a discipline in which real understanding of literature and the human experience builds up along with all of the words" (Gottshall, 2008, p. 2).
As a result of bringing scientific methods to research in the humanities, we will be legitimately contributing to scientific progress.We will be separating what is mere speculation from systematic explorations.We strongly hold that research in this area, as in others, has to be accountable and build up on accumulated knowledge.And this knowledge has to have been gained from scientific methods.Once this paradigmatic shift is transported to the educational setting, learning by research will be a much more effective pedagogical strategy than lecturing or relying on other traditional teaching methods.
Part of this new approach requires carrying out long-term evaluations.As argued above, research projects in the humanities tend to focus on immediate results and not on the way they affect the participants.Still less is the assessment geared towards long-term perspective.The rare exception is The University of Cambridge and the Arts and Humanities Research Council (AHRC) program (Levitt et al., 2010) that accounts for the impacts of its arts and humanities research activities.
A tradition of looking back to see whether the experience stays or not is virtually missing in the this area.Following a Freirean tradition in teaching (Freire, 1970; see also Apple, 2013, among others), we agree that "research in these fields [cultural and social studies] is in itself a social intervention" (Boman, 2013).As the researchers themselves are part of the experience and are also subjected to change, we believe that the research project only truly ends after its impact on the researchers themselves is taken account of.
By studying one specific case, we show here evidence that may serve as a reference to other researchers interested in preparing new generations of autonomous and critical thinkers who can invest in self-perception, work collaboratively, understand cultural phenomena from a more scientific perspective, and reflect on the changes their practice effected over themselves and others.Despite efforts in other more empirically-oriented disciplines, most humanities departments still work on a school principle: learners are required to do a good deal of reading and then subjectively analyze the information.A systematic introduction to methodological issues is rare, and the interpretative approach is dominant, resulting in narratives of some kind that remain, however, largely limited to the subjective view of the one who does the interpreting.This may contribute to understandings of phenomena by certain individuals and establish different perspectives.But they do not explain what really happens, neither can they offer predictions (see the difference between "understanding" and "explaining" in van Peer et al., 2012, p. 1-3).As a result, students develop little autonomy as critical thinkers.What they do develop is a dependency on critics and acknowledged theoreticians, and they eventually graduate without acquiring any methodological skill in research.Despite some efforts towards educating a new generation of researchers, the general setting described by van Peer et al (2011, p. 112) still persists: "[T]he state of the art in literary studies is particularly precarious: students are hardly confronted with research, staff are hardly aware of methodological issues, and graduation occurs without students ever having taken part in original investigations" (van Peer et al., 2011, p. 112).In this article, we demonstrate how learning by research has great potential in empowering learners as well as in shaping their future outlooks.
Case study.The research project in question is the REDES project (Research and Development in Empirical Studies) founded in 2002 by three researchers from different countries: W. van Peer (Ludwig Maximilian University, Munich, Germany), S. Zyngier (Federal University of Rio de Janeiro, Brazil) and F. Hakemulder (Utrecht University, the Netherlands).The seed was sown when van Peer and Hakemulder were invited to teach a course in Brazil and came into contact with an on-going project there which was being carried out following the tenets of critical pedagogy (Freire, 1970;van Peer et al., 2012;Apple, 2013).In 2003, two more regional groups joined REDES: the ones from Kiev, Ukraine, and Edmonton, Canada.By linking research groups of five countries from three continents, the project functioned for about a decade as an international research group aiming at the investigation of culture, literature, language and media from a multicultural perspective.More than stimulating research by (especially) students in different parts of the world (Viana et al., 2007;Mendes, 2008), a very important goal was to help students become autonomous researchers by way of using and disseminating the empirical investigation of literature and language (van Peer et al., 2012).
Rather than teaching, the growth of participants as researchers was seen as a process of continuous knowledge building, constant questioning and unending search, and the path to knowledge involved shared and collective activities.Contrary to more traditional forms of academic interaction, REDES offered a new interactive template, where the roles of teacher / student were replaced by those of senior and junior researchers, which in many ways put away the conventional academic hierarchy and instead shifted the focus onto the quality of the research produced.Junior researchers would bring up possibilities of research, and these would be jointly discussed and designed.For practical reasons, REDES functioned largely online by way of using a discussion forum (Viana et al., 2009).In order to guarantee that the new knowledge obtained was recorded, project participants prepared a number of publications, including Zyngier et al. (2007), which is unique as a collection of articles by junior researchers.Likewise, Zyngier et al. (2008) also contains a part written especially by REDES junior members.Other publications include Zyngier et al. (2002;2003); Fedorova et al. (2006);Watson & Zyngier, 2007.All these publications stand as evidence of the quality of the new knowledge produced.What they do not account for is the change that was being effected on the participants themselves.
Long-term assessment.This study used three methods to assess the long-term impact of the REDES project on its participants.These methods are complementary, each providing a different depth of evidence.The methods of assessment were: (1) a qualitative and quantitative profile of the participants; (2) a corpus analysis of their comments on the project itself to find out if patterns of language emerged, as we believe that these patterns reflect their collective thoughts about the project; (3) a qualitative analysis of these responses.In order to assess the extent to which the objectives of the project were achieved and how participants saw the experience one decade later, we asked for their feedback on how they thought REDES may have affected their lives and careers.To this end, three regional coordinators contacted 53 members from three regional groups (Brazil, Germany and Ukraine) who spent on average 3 years (standard deviation [henceforth SD] = 2.5) with the project.Only 7 participants were male, and the age distribution was the following: 13 were in the group 20 -29, 35 in 30 -39, and 5 in 40 -49.Participants answered a specifically designed questionnaire (see Appendix A) where they evaluated their experience on a scale from 1 to 6.They also assessed what they considered as the strengths and weaknesses of REDES.The quantitative data were analyzed with the help of the computer program Statistical Package for Social Sciences (SPSS), while the answers to open questions were given qualitative treatment.Their comments were also digitized into three files (one for each group) and submitted to corpus analysis using a concordancer software program AntConc (see Table 1).As the corpus size for the groups varied, we could only compare them in terms of ratios and the position where the most frequent words occurred.Type / Token ratio (henceforth TTR) indicates that the groups differed in terms of words they used to evaluate their experience, from the more individual and varied (German group: TTR 0.51), to the more consensual (Brazilian group: TTR 0.22).The Ukrainian group presented an intermediate position (TTR 0.35).We then singled out the parts in which they evaluated the strengths and the weaknesses of the project so as to check these aspects in particular.
Results.The statistical analysis indicates that participants acknowledged the project's contribution to their professional development (average 5.2, SD [standard deviation] = 1.1), to their personal development (average 5.1, SD = 1.0), as well as intercultural development (average 4.9, SD = 1.3).As the project increased intercultural awareness and flexibility of its members, not surprisingly, and unlike at the time of its functioning, when they lived in 5 different countries, at the time of collecting the data for this study the participants lived and worked in nine: in Brazil (22 persons), Ukraine (13), Germany (3), the UK (3), the USA (3), Japan (2), Canada (2), Switzerland (1) and the Netherlands (1).
Being given impetus for foreign language skills development (average 4.3, SD = 1.6), the former REDES members informed they now speak a larger array of foreign languages.Out of 53, 51 participants acknowledged their fluency in English (which was the working language of the Project), 21 in Portuguese, 16 in Russian, 15 in Ukrainian, 13 in German, 7 in French and Spanish, 1 in Japanese, and 2 in other languages.This indicates that an average participant of REDES now speaks more than one foreign language, and many members are competent in three languages and more.
The data also showed that most participants were motivated to pursue further degrees and that the skills they acquired by way of collaborative empirical research considerably influenced their careers in many different areas of action.Thirteen participants obtained their Ph.D. degree in various fields, and several have now even reached the Assistant or Associate Professor position.Fifteen more are currently working toward this degree, which means that as many as 53 % of the respondents have considerably advanced in their academic careers.Some participants also left academia for business, and 7 obtained MBA degrees while 4 more are currently working toward it (see Figure 1).
Despite the fact that the Project functioned in a unanimous collaborative way, the data also revealed a tendency towards significant differences between its Ukrainian and Brazilian participants.In their answers to the question 'How did the Project contribute to your personal development?'(see Figures 2 and 3), the Brazilian participants scored significantly higher (mean [henceforth M] = 5.5, p = .016)as compared to Ukrainian ones (M = 4.7).Additionally, the Brazilian group members predominantly remained involved in teaching (M = 5.1, p = .064),while many of the Ukrainian participants turned to business instead (M = 4.0).
As a second method employed in the evaluation was corpus analysis of respondents' comments on the project, the analysis cast light on participants' collective thoughts about REDES.Once submitted to corpus analysis, participants' comments resulted in the following table of most frequent words.Here we present only the top 10 words in the three corpora (see Table 2): The most frequent occurrences in the majority of corpora tend to be function words such as the, of, to, and, a, in (http://www.world-english.org/english500.htmor http://www.wordfrequency.info/free.asp?s=y).Quite interestingly, in the Brazilian corpus, the most frequent word is the personal pronoun I, which draws our attention to the fact that the members of the group must have felt the experience as something very personal.In the Ukrainian group it occurs in the 5th, and in the German, in the 3rd position, which tends to reinforce this interpretation.Another evidence is the high occurrence of the possessive adjective my, which holds the 10th position for the Ukrainian and the German groups, and the 8th for the Brazilian one.This fact indicates that learning by research does indeed make the experience something the students really see as their own.It stands as evidence for the goal of developing students' sense of appropriation (see Shor, 1980;Apple, 2011;Apple, 2013).
The notion of research is also very central to this experience.The word appears in the 8 th position in the Ukrainian and Brazilian research corpora and in the 15 th in the German one.It is quite interesting to note that the nouns teacher, professor, etc. occur only in the Ukrainian corpus and in the 52 nd position.Similarly, teaching occurs in the Ukrainian and Brazilian data in the 53 rd and 71 st positions respectively.
In terms of collocates, the possessive my occurs in the following vicinities (see Table 3): From this table we can see that participants from the three national chapters tended to differ in terms of outlook.The Ukrainian and German groups mentioned knowledge, whereas the Brazilians used ideas.Brazilian group members also emphasized life, personal, and experience.Additionally, the Ukrainian group concentrated on acquiring different kinds of skills, whereas the German and Brazilian ones targeted job and profession.
As to what participants evaluated as the strengths of the Project, the results are the following (see Table 4).

Table 4: Evaluation of the Project's strengths by the three groups
The strengths of REDES concentrate in terms of research for the Ukrainian and Brazilian (6 th , 5 th ) and work for all three groups (14 th , 5 th , and 18 th ).It is interesting to notice that the German and Brazilian groups pointed out how (8 th and 12 th ), which indicated that they saw the Project as promoting the means to arrive at something.Ukrainian and Brazilian participants also mentioned students (7 th and 9 th ), cooperation (16 th in the Ukrainian corpus), and group (11 th in the Brazilian corpus).The German group was the only one that emphasized results (13 th ), and allowed (16 th ).In trying to find out how participants responded to the item weaknesses, we ran into a problem of providing a table of occurrences that would indicate trends for two main reasons: (1) the wordings were too few; (2) when answering this item, many mentioned the strengths instead.Therefore, in order to understand the shortcomings of the Project, we had to check each answer and submit the data to qualitative treatment.
As a consequence, we found out that all three regional groups mentioned turnover of participants though this can be justified by the fact that the members would graduate from the university and thus leave the Project.In this connection, the Brazilian members displayed regret about "lack of extrinsic / intrinsic motivation to get new members on board".All three groups pointed out that they would appreciate more cultural diversity in REDES and more communication of its members.A bit more publicity, as suggested by the Ukrainian group, would help to solv the problem.
Though being explicitly different culturally, the Ukrainian and Brazilian groups shared many of their reactions.Both focused on practical matters: lack of funding and thus "the expense of going to conferences" (UKR), "difficulty to equate doing research, participating in conferences and finding means to live" (UKR), "difficulty to reconcile academic research with work" (BRA), and "tight deadlines for students who do not have any grants and still have to work on 2 or 3 different jobs" (BRA)."Lack of time" in general (UKR) and "lack of time to do all the reading and writing" (BRA) were also the shared categories.Additionally, both groups demonstrated certain dependence on senior researchers: the Brazilian group mentioned that there were "too many people involved in the Project, which reduced the time spent under supervision" while the Ukrainian one demonstrated their regret that just "few senior researchers [were] involved".
At the same time, regional groups pointed out disadvantages of REDES which were culturespecific.Hence, for the Ukrainian group, the weaknesses of the Project were largely connected with the conservatism of local academic life.As empirical studies have been a fairly new approach to Ukrainian universities, the participants acknowledged that they faced "difficulty of implementing empirical methods in the present context" and felt "need for more interactive practical seminars" to compensate for the lack of university training.At the same time, they worried that the Project results might not be "applicable to real life".However, some Ukrainian members mentioned that they did not see any weaknesses of the Project and felt really bad that it was too short and finally came to an end.For the German group, the weaknesses of REDES were mostly connected with the Project itself as they mentioned "lack of rigor and coherence", low effectiveness of the communication platform and "distance between participants" as they acknowledged that the "web forum often was not enough".
Although the Brazilian questionnaires, similarly to Ukrainian ones, had statements such as "I can't find any [weaknesses]" or '"REDES" was the best thing I found in University", some members of this group listed local bureaucracy and lack of institutional support, "the unfair competition from the market" as well as focused on the personal aspect, i.e., the problem of "relationships when working together".Similarly, to the German group, they also mentioned 'lack of structure' in the Project.
Quite contrary to the German group, who thought that REDES demonstrated "at times oversimplified approach to research questions", Brazilian members pointed out that the Project "may have been too demanding for undergraduate students" and "unfamiliarity with computer science" was an issue.

Conclusion.
Results in education are not immediate.Time is needed not only for the participants to mature concepts and attitudes, but also to allow them to progress in their careers and occupy positions in which they can implement what they learned.This is why assessing projects in the humanities also needs long-term evaluations.The present study contributed to the view that learning by research is valid and should be extended to many different disciplines, literature included.As this methodology is beginning to prove itself a pedagogically effective strategy and as the virtual world has enabled international exchanges among students, more energy must be invested in various aspects where the Project has proved to be faulty.There are still many different hurdles to be overcome in future projects so as to maximize the effects of this kind of academic activity.Among them, we would like to stress the need of incorporating it in the curriculum.Students should ideally also be granted some financial support so that they do not have to work outside the Project.This means universities should acknowledge the benefits this kind of teaching brings about and invest more heavily in promoting them.In the case study reported here, all efforts were voluntary from the part of both the senior and the junior researchers.The extra effort spent on this additional task tends to fade in time, due to more immediate material needs.In addition, although the coordinators designed the Project together and shared an array of common goals, strategies and theoretical assumptions and perspectives, this assessment shows that despite a strong emphasis on education, a sizeable number of participants pursued different careers.This might be interpreted as a result of a high degree of autonomy and empowerment promoted by the Project, although one cannot rule out the social pressures of each setting and the participants' perception that what was learned during their training could be applicable to other areas.Today, the emphasis is being given to exchange programs and sending undergraduates abroad.This policy would be highly beneficial to a project like REDES, as it would certainly enhance the bonds between participants from different settings.Having said that, REDES did have a multiplication factor.It has started a chain reaction, raising a generation of independent empirical scholars who have been acting and interacting internationally.This long-term evaluation demonstrates that this kind of conceptual framework, its methodology and evaluation techniques can be used to analyse similar projects in other field of humanities or beyond.How far can similar projects say the same?Perhaps they can if they are evaluated 10 years later.

Figure 2 .
Figure 2. Assessment of Project's contribution to professional development: Brazilian group.

Figure 3 .
Figure 3. Assessment of Project's contribution to professional development: Ukrainian group.

Figure
Figure 4. Assessment of Project's contribution to intercultural development: Ukrainian group.

Figure 5 .
Figure 5. Assessment of Project's contribution to intercultural development: Brazilian group.