COMPUTER ADAPTIVE LANGUAGE TESTING ACCORDING TO NATO STANAG 6001 REQUIREMENTS

Authors

  • Viktoriia Krykun National Defence University of Ukraine named after Ivan Cherniakhovskyi, Ukraine https://orcid.org/0000-0001-7409-3383
  • Nataliya Tarasenko National Defence University of Ukraine named after Ivan Cherniakhovskyi, Ukraine

DOI:

https://doi.org/10.20535/2410-8286.225018

Abstract

The article deals with the innovative, cutting age solution within the language testing realm, namely computer adaptive language testing (CALT) in accordance with the NATO Standardization Agreement 6001 (NATO STANAG 6001) requirements for further implementation in foreign language training of personnel of the Armed Forces of Ukraine (AF of Ukraine) in order to increase the quality of foreign language testing. The research provides the CALT method developed according to NATO STANAG 6001 requirements and the CALT algorithm that contains three blocks: “Starting point”, “Item selection algorithm”, “Scoring algorithm” and “Termination criterion”. The CALT algorithm has an adaptive ability, changing a complexity level, sequence and the number of items according to the answers of a test taker. The comparative analysis of the results of the CALT method piloting and the paper-and-pencil testing (PPT) in reading and listening according to the NATO STANAG 6001 requirements justifies the effectiveness of the three-level CALT method. It allows us to determine the following important benefits of CALT: test length reduction, control of measurement accuracy, objective assessment, improved test security, generation of a unique set of items, adaptive ability of the CALT algorithm, high motivation of the test takers, immediate score reporting and test results management. CALT is a qualitative and effective tool to determine test takers’ foreign language proficiency level in accordance with NATO STANAG 6001 requirements within the NATO Defence Educational Enhancement Programme. CALT acquires a special value and relevance in the context of the global COVID 19 pandemic.

Downloads

Download data is not yet available.

References

  1. Albano, A. D., Cai, L., Lease, E. M., & McConnell, S. R. (2019). Computerized adaptive testing in early education: Exploring the impact of item position effects on ability estimation. Journal of Educational Measurement, 56(2), 437-451. https://doi.org/10.1111/jedm.12215
    |
  2. ATrainP-5. NATO STANAG 6001: Language Proficiency Levels. Edition A Version 2. (2016). North Atlantic Treaty Organization, Standardization Office. Retrieved 12 June 2017. Retrieved from www.natobilc.org
  3. Babcock, B., & Weiss, D. J. (2012). Termination criteria in computerized adaptive tests: do variable-length CATs provide efficient and effective measurement? J. Comput. Adap. Test, 1, 1–18. https://doi.org/10.7333/1212-0101001
  4. Bachman, L. (2000). Modern language testing at the turn of the century: assuring that what we count counts. Language Testing, 17, 1–42. https://doi.org/10.1191/026553200675041464
    |
  5. Bachman, L. F. (1990). Fundamental Considerations in Language Testing. Oxford, Oxford University Press. Retrieved from https://www.academia.edu/28794667/Fundamental_Considerations_in_Language_Testing
  6. Bachman, L. F., Davidson, F. & Milanovich, M. (1996). The use of test methods in the content analysis and design of EFL proficiency tests. Language Testing, 13, 125–1
  7. Beckers, J.J., Schmidt, H.G. (2003) Computer experience and computer anxiety. Computers in Human Behavior, 19, 6, 785-797, https://doi.org/10.1016/S0747-5632(03)00005-0
  8. Blake, R. J. (2011). Current trends in online language learning. Annual Review of Applied Linguistics, 31, 19-35. https://doi.org/10.1017/S026719051100002X
  9. Canale, M., & Swain, M. (1980) Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1(1), 1–47. https://doi.org/10.1093/applin/I.1.1
  10. Chapelle, C., & Voss, E. (2017). Utilizing Technology in Language Assessment. Language Testing and Assessment. Encyclopedia of Language and Education, 3rd ed. (pp.149-161). https://doi.org/10.1007/978-3-319-02261-1_10
  11. Chen, C. -., Wang, W. -., Chiu, M. M., & Ro, S. (2020). Item selection and exposure control methods for computerized adaptive testing with multidimensional ranking items. Journal of Educational Measurement, 57(2), 343-369. https://doi.org/10.1111/jedm.12252
    |
  12. Eggen, TJHM. (2018). Multi-Segment Computerized Adaptive Testing for Educational Testing Purposes. Front. Educ, 3, 111. https://doi.org/10.3389/feduc.2018.00111
  13. Fedoruk, P. I. (2008). Adaptive tests: general provisions. Mathematical machines and systems. 1, 115−127. Retrieved from http://dspace.nbuv.gov.ua/handle/123456789/748
  14. Frick, T. W. (1992). Computerized adaptive mastery tests as expert systems. Journal of Educational Computing Research, 8(2), 187−213. https://doi.org/10.2190/J87V-6VWP-52G7-L4XX
  15. Fulcher, G. (1999). Computerizing an English language placement test. ELT Journal, 53, 289–299. https://doi.org/10.1093/elt/53.4.289
  16. Fulcher, G. (2017). Criteria for Evaluating Language Quality. Language Testing and Assessment. Encyclopedia of Language and Education, 3rd ed. (pp.179–192). https://doi.org/10.1007/978-3-319-02261-1_13
  17. Gibbons, R. D., & deGruy, F. V. (2019). Without wasting a word: Extreme improvements in efficiency and accuracy using computerized adaptive testing for mental health disorders (CAT-MH). Current Psychiatry Reports, 21(8) https://doi.org/10.1007/s11920-019-1053-9
    |
  18. Hambleton, R. K., & Zaal, J. N. (1991). Advances in educational and psychological testing: theory and applications. Springer Sciences & Business Media, LLC. https://doi.org/10.1007/978-94-009-2195-5
  19. Hambleton, R.K., Swaminathan, H. & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA, Sage Publications.
  20. Hockly, N. (2019). Automated writing evaluation. ELT Journal, 73(1), 82–88. https://doi.org/10.1093/elt/ccy044
  21. Holzknecht, F., McCray, G., Eberharter, K., Kremmel, B., Zehentner, M., Spiby, R., & Dunlea, J. (2021). The effect of response order on candidate viewing behaviour and item difficulty in a multiple-choice listening test. Language Testing, 38(1), 41-61. https://doi.org/10.1177/0265532220917316
    |
  22. Hrabar, Е. V. (2010). Basic types of tests in foreign languages in US pedagogy. Scientific notes of Ternopil National Pedagogical University named after Volodymyr Hnatyuk. Series: Pedagogy, 1, 194–201. Retrieved from http://nbuv.gov.ua/UJRN/NZTNPU_ped_2010_1_37
  23. ICAT (International Association for Computerized Adaptive Testing). Retrieved from http://www.iacat.org/
  24. Jamieson, J. (2005). Trends in computer-based second language assessment. Annual Review of Applied Linguistics. 25. 228-242. https://doi.org/10.1017/S0267190505000127.
  25. Kravchenko, O. M. & Plakasova, Zh.M. (2010). Model of intellectual controlling subsystem with multilevel adaptive testing. East European Journal of Advanced Technologies, 4/2 (46), 21–25.
  26. Larkin, K. C. & Weiss, D. J. (1975). An empirical comparison of two-stage and pyramidal adaptive ability testing (Research Report, 75-1). Minneapolis: Psychometrics Methods Program, Department of Psychology, University of Minnesota. Retrieved from https://eric.ed.gov/?id=ED106317
  27. Larson, J. (1999). Considerations for testing reading proficiency via computer-adaptive testing. In M. Chalhoub-Deville (Ed.), Studies in language testing, Vol. 10. Issues in computer-adaptive testing of reading proficiency (pp.71-90). Cambridge: University of Cambridge Press.
  28. Lin, G.-H., Huang, Y.-J., Chou, Y.-T., Chiang, H.-Y., & Hsieh, C.-L. (2019). Computerized adaptive testing system of functional assessment of stroke. Journal of Visualized Experiments, 2019(143). https://doi.org/10.3791/58137
    |
  29. Lord, F. M. (1971). The self-scoring flexilevel test. Journal of Educational Measurement, 8, 147–151. https://doi.org/10.1111/j.1745-3984.1971.tb00918.x
  30. Maia, M., Lilley, M. & Barker, T. (2003). Computer-Adaptive Testing in Higher Education: the way forward? XXXVIII Cladea - Latin American Council of Schools of Administration: Lima. https://doi.org/10.13140/2.1.2074.7520
  31. Mâsse, L. C., O’Connor, T. M., Lin, Y., Hughes, S. O., Tugault-Lafleur, C. N., Baranowski, T., & Beauchamp, M. R. (2020). Calibration of the food parenting practice (FPP) item bank: Tools for improving the measurement of food parenting practices of parents of 5–12-year-old children. International Journal of Behavioral Nutrition and Physical Activity, 17(1). https://doi.org/10.1186/s12966-020-01049-9
    |
  32. Meunier, L. (1994). Computer Adaptive Language Tests (CALT) Offer a Great Potential for Functional Testing. Yet, Why Don't They? CALICO Journal, 11(4), 23-39. Retrieved February 25, 2021, from http://www.jstor.org/stable/24152755
    |
  33. Mizumoto, A., Sasao, Y., & Webb, S. A. (2019). Developing and evaluating a computerized adaptive testing version of the word part levels test. Language Testing, 36(1), 101-123. https://doi.org/10.1177/0265532217725776
    |
  34. Monfils, L. F., & Manna, V. F. (2021). Time to achieving a designated criterion score level: A survival analysis study of test taker performance on the TOEFL iBT® test. Language Testing, 38(1), 154-176. doi:10.1177/0265532220940709
    |
  35. Newhouse, C. P., & Cooper, M. (2013). Computer-based oral exams in Italian language studies. ReCALL, 25(3), 321-339. https://doi.org/10.1017/S0958344013000141
  36. Oppl, S., Reisinger, F., Eckmaier, A Helm, C. (2017). A flexible online platform for computerized adaptive testing. Int J Educ Technol High Educ, 14(2). https://doi.org/10.1186/s41239-017-0039-0
  37. Padmavathi, M. (2016) A study of student-teachersʼ readiness to use computers in teaching: an empirical study. I-manager’s Journal on School Educational Technology, 11(3). Retrieved from https://pdfs.semanticscholar.org/ce38/ddfc90c28af40dfddd9a741fe703a632565f.pdf?_ga=2.127548237.1317622877.1612358315-1008902291.1612358315
  38. Sands, W.A., Waters, B.K. & McBride, J.R. (1997). Computerized adaptive testing. From inquiry to operation. Washington, American Psychological Association. Retrieved from https://pdfs.semanticscholar.org/4e9c/c706ea17628f970389a25b2d268b52320e13.pdf
  39. Serhiienko, V.P., Malezhyk, M.P., & Sitkar T.V. (2012). Computer technologies in testing: a textbook. Lutsk: Printing house “Volyn Polygraph”. Retrieved from https://www.coursehero.com/file/64399499/KTTpdf/
  40. Thompson, N. A. & Weiss, D. A. (2011). A Framework for the Development of Computerized Adaptive Tests. Practical Assessment, Research, and Evaluation, 16, Article 1. https://doi.org/10.7275/wqzt-9427
    |
  41. Thompson, N. A. (2007). A practitioner’s guide for variable-length computerized classification testing. Practical Assessment Research & Evaluation, 12 (1). https://doi.org/10.7275/fq3r-zz60
  42. Thompson, N.A. (2016). User’s manual for SIFT: Software for investigating test fraud. Minneapolis, Assessment Systems Corporation.
  43. Vispoel, W.P., Rocklin, T.R., & Wang, T. (1994). Individual differences and test administration procedures. A comparison of fixed-item, computerized adaptive, and self-adapted testing. Applied Measurement in Education, 7, 53–59.
    |
  44. Wang, C., Weiss, D. J., & Shang, Z. (2019). Variable-length stopping rules for multidimensional computerized adaptive testing. Psychometrika, 84(3), 749-771. https://doi.org/10.1007/s11336-018-9644-7
    |
  45. Weiss, D. J. & Kingsbury, G. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21, 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x
  46. Weiss, D. J. (1973). The stratified adaptive computerized ability test (Research Report 73-3). Minneapolis: University of Minnesota, Department of Psychology. Retrieved from http://iacat.org/sites/default/files/biblio/we73-3.pdf
  47. Weiss, D. J. (2004). Computerized adaptive testing for effective and efficient measurement in counseling and education. Measurement and Evaluation in Counseling and Development, 37 (2), 70–84. https://doi.org/10.1080/07481756.2004.11909751
  48. Wigglesworth, G. & Frost, K. (2017). Task and Performance-Based Assessment. Language Testing and Assessment. Encyclopedia of Language and Education, 3rd ed., 121-133. https://doi.org/10.1007/978-3-319-02261-1_8
  49. Yigit, H. D., Sorrel, M. A., & de la Torre, J. (2019). Computerized adaptive testing for cognitively based multiple-choice data. Applied Psychological Measurement, 43(5), 388-401. https://doi.org/10.1177/0146621618798665

Downloads

Published

2021-06-22

How to Cite

Gawliczek, P. ., Krykun, V., Tarasenko, N., Tyshchenko, M., & Shapran, O. (2021). COMPUTER ADAPTIVE LANGUAGE TESTING ACCORDING TO NATO STANAG 6001 REQUIREMENTS. Advanced Education, 8(17), 19–26. https://doi.org/10.20535/2410-8286.225018

Issue

2021: Special issue 17

Section

ARTICLES

License

Copyright (c) 2021 Piotr Gawliczek, Viktoriia Krykun, Nataliya Tarasenko, Maksym Tyshchenko, Oleksandr Shapran

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Authors who publish with this journal agree to the following terms:

    1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.

    1. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.

  1. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).