Volume 7, Number 1, July 2018
Assessment of Pen and Paper Examination Development in Open and Distance Education: Implications for Academic Counselling
Évaluation du développement de l'examen au stylo et au papier dans l'enseignement ouvert et à distance: implications pour l’orientation scolaire
Dr. Augustine C. Ukwueze1 and Professor Patrick E. Eya2
- Faculty of Education, National Open University of Nigeria
acukwueze@gmail.com - RETRIDOL, National Open University of Nigeria
Abstract
This study evaluated pen and paper examinations development with a view to improving examination validity in open and distance education. The study specifically examined the appropriateness, meaningfulness, and usefulness of pen on paper examinations in the National Open University of Nigeria (NOUN). The study employed the survey research design with a view to reaching a large number of respondents. In all, a total of 122 lecturers formed the respondents used for the study. A self-structured questionnaire was constructed and validated for data collection. Three research questions and two hypotheses were formulated to guide the study. The data were analyzed using the SPSS statistical tool. Descriptive statistics (mean and standard deviation) were used to answer the research questions while inferential statistics (t-test and ANOVA) were used to test the hypotheses at .05 level. The results show that valid procedures of item development and generation are always followed in setting pen and paper examinations in NOUN. Despite this, counsellors need to work on students and staff regularly on the rubrics of examinations to ensure both student and institutional success. The study, therefore, recommended appropriate procedures using Bloom’s taxonomy for test development.
Keywords: Evaluation, pen and paper, validity, distance education.
Résumé
Cette étude a évalué le développement des examens au stylo et au papier en vue d'améliorer la validité de l'examen dans l'enseignement ouvert et à distance. L'étude a spécifiquement examiné la pertinence, la signification et l'utilité des examens au stylo et au papier de l'Université nationale ouverte du Nigeria (NOUN). L'étude a utilisé la conception de la recherche de l'enquête en vue d'atteindre un grand nombre de répondants. Au total, 122 professeurs ont formé le nombre de répondants utilisés pour l'étude. Un questionnaire auto structuré a été construit et validé pour la collecte de données. Trois questions de recherche et deux hypothèses ont été formulées pour guider l'étude. Les données ont été analysées à l'aide de l'outil statistique SPSS. Des statistiques descriptives (écart moyen et standard) ont été utilisées pour répondre aux questions de recherche, tandis que des statistiques inférentielles (t-test et ANOVA) ont été utilisées pour tester les hypothèses au niveau de 0,05. Les résultats montrent que les procédures valides de développement et de génération d'objets sont toujours suivies dans la mise en place d'examens au stylo et au papier à NOUN. Malgré cela, les conseillers doivent travailler régulièrement sur les rubriques des examens sur les étudiants et le personnel afin d'assurer la réussite des étudiants et de l’institution. L'étude a donc recommandé des procédures appropriées utilisant la taxonomie de Bloom pour le développement des tests.
Mots-clés : Évaluation, stylo et papier, validité, enseignement à distance.
Introduction
In an academic community, evaluation is seen as a central process of assessing teachers and students as well as the overall quality of a study programme in order to improve standards (Shukla, 2005). According to Okonkwo (2013), teaching and learning process is incomplete without effective and meaningful assessment of students’ learning outcomes. This involves periodic assessment via tests or examinations to check the level of students’ knowledge acquisition and competence; and award of certificates at the end of programmes of study (Chima, 2006). Hence, examinations are used to assess instructional objectives, students’ progress and success, including certification. Learning outcomes are evaluated through organized procedures that determine the extent to which instructional strategies employed to achieve specific objectives are successful; these procedures also provide instructors with feedback regarding their actions (Ukwueze, 2012). According to Okoye (1986), the process of an examination requires an organized assessment technique that presents the students with a series of questions or tasks tailored towards ascertaining the candidates’ acquired skills, knowledge, content and ability to utilize these knowledge and skills effectively. Ajiboye, Ojo, Saidu and Okesina (2014) maintain that in Nigerian schools, examinations are administered to admit students into various courses, promote students to higher stages or levels of education, select students for special academic awards and most importantly, final year examinations determine the completion of the course of a study.
As a result of the huge importance of examinations, no institution toys with the quality of examination development and conduct as irregularities at any stage are often punished. In higher institutions for instance, punishment on the part of students ranges from cancellation of results of the affected courses to rustication for one or two semesters or outright expulsion (Kayode, 2006) while lecturers proved to be involved in any form of examination fraud are thrown into the labour market.
Examination is an age long exercise that could be done in several forms. Some of such forms as identified by Ezeh and Onah (2005) include oral, written and practical exercises. They categorized written examination into essay and objective test formats. In open and distance education where students are “unseen”, oral examination is not popular. According to Robbles and Broathen (2002), assessment methods in open and distance education include self-testing, assignments, electronic portfolios, online discussion, synchronous threaded discussion in groups, synchronous chatting and e-mailing questions. Whichever method is employed in any institution depends largely on its validity and reliability. Okonkwo (2013) maintains that for any assessment to be meaningful in higher education, especially in the open and distance education there must be a well designed and developed item bank.
According to Ukwueze (2012), the first pen and paper examination in the NOUN was conducted in 2005 when the continuous assessment score represented 40% and examination score represented 60% of the final score. Examinations in NOUN are conducted through continuous assessment and pen on paper examinations. The continuous assessment is a web-based assessment that carries 30% of students’ final grade. The pen on paper examinations involve essay mode of assessment which carries 70% of students’ overall grade. Emphasis in this study is on the pen and paper examination as a major determinant of students’ assessment and certification. However, the pen and paper is restricted to students in 300 level and above in all the schools except in the Faculty of Law where students in all the levels are examined using pen and paper examinations (Ukwueze, 2012). In all, pen and paper examinations are conducted in the first and second semesters of every academic session in the university (NOUN).
Few weeks to the commencement of examinations in each semester, lecturers are requested to submit their examination questions. Questions submitted by lecturers usually go through the process of both internal and external moderation for validation. This is to ensure that course contents are covered in line with the objectives of such courses. Thereafter, the questions are sent to the Directorate of Examinations and Assessment (DEA) to administer to students using a prepared examination timetable for the semester.
All the examinations, including pen and paper are conducted at the 78 study centres of the university located across the country to assess students’ performance in each semester. The 2005 examination scripts were collected from the study centres and marked at the headquarters while the 2007 and 2008 examination scripts were posted to various study centres outside their domicile areas for marking; the 2009 examination scripts were marked at the domicile study centres by facilitators. The 2012, 2013 and 2014 examination scripts were marked at the headquarters. The March/April 2015 examination scripts were marked at five designated study centres at Abuja, Benin, Enugu, Kaduna and Lagos to checkmate cases of missing scores hitherto reported during the previous years. Sadly, the reports of external examiners used to mark the examination scripts at the end of each examination period often indicate some lapses in the quality of examination questions used to assess students. As a result, faculty members are often accused of presenting poor quality questions in assessing their students.
The development of pen and paper examinations in open and distance education like in every other educational institution requires painstaking exercises and procedures to ensure validity. Opateye (2013) maintains that evaluation cannot take place without result oriented classroom testing, which is based on valid and reliable test items. Every test item meant for assessing students’ learning outcomes needs to be constructed with a laid down procedure if the validity of that test item is actually to measure its purpose. Thus, assessment design is taken by lecturers based on pre-determined criteria. Okonkwo (2013) maintains that good items represent an investment in learning; and to establish technical quality of test items requires that test development should incorporate cognitive complexity or various levels of learning that can be assessed, content quality in terms of skills, coverage and format, reality of the test items in relation to learners’ expectations (meaningfulness), and appropriateness of use of language in constructing test items. Other criteria include fairness in scoring schemes and reliability or consistency of questions in relation to what the students know.
Pen and paper examinations are teacher-made assessment procedures or forms of formative evaluation of students’ academic achievement, which many experts believe to be bedevilled by biases, thereby rendering them invalid, unreliable and imperfect (Metzener, 2013; Osuji, 2014). Researches indicate that students perform well in schools where teachers implement formative evaluation procedures professionally (Agu, 2014). The problem however, is that teachers do not follow item generation processes in conducting tests and examinations due to their inability to use construction procedures (Opateye, 2013). Ojerinde in Opateye (2013) states processes involved in item development such as specification of course content and objectives, clarification of instructional objectives, determination of type and number of test items, development of test blueprint and item specifications, screening and analysis of test instruction, assembling of test items and development of scoring key. For the validity of examination at final level, Ajiboye, Ojo, Saidu and Okesina (2014) opined that examinations should be subjected to moderation as well as the results to determine whether a student would graduate or not, and to determine the class of degree that students obtain.
Hence, the development of pen and paper examination questions has a laid down procedure. According to the Canadian Public Service Commission (2017), the development of pen and paper examination requires task listing, specification of response format, specification of response format, specification of the number of questions, the time limit and difficulty level. Other procedures include writing the questions and developing the the scoring guide; reviewing the questions and scoring guide to ensure predetermined scoring procedures and mark allocations. The Commission further states that if marks are to be deducted for incorrect grammar, spelling and punctuation, it must be stated in the instruction to the candidates.
Unfortunately, several examiners or teachers in several institutions of lower and higher learning hardly follow the aforementioned processes and procedures when generating examination items to assess students’ performance (Nenty, 2004; Falayajo, 2004; Schafer, Papapolydorou, Rahman & Paker, 2005). Schafer, Papapolydorou, Rahman and Paker (2005) further attested that test administration is very crucial and must involve detailed rules and specifications to ensure validity and reliable process of evaluation. Thus, this study evaluated the procedures of developing pen and paper examinations in order to improve validity in open and distance education in Nigeria.
The relevance of pen and paper examinations in educational institutions cannot be overemphasized. AL-Qdah and Abadneh (2017) are of the opinion that many students prefer pen and paper examinations to online examinations because the former are more readable and reliable, less tiring and readily available for expression of oneself. In addition, there is less technical failure or outright failure in pen and paper examinations if students put enough preparations and pay much attention to study time. As a result of this, preparation of pen and paper examination questions should be handled professionally in order to achieve desired objectives of using them for assessment. The need to validate the process of preparing pen and paper examinations in open and distance education is necessary bearing in mind that assessment activities of online distance education do not follow strictly, the principles suggested in the literature (Kim, Smith & Maeng, 2008). This is also necessary since some experts observed gender and seniority differences in teaching evaluations and assessment development procedures; female faculty members receive lower teaching evaluations than their male colleagues while junior faculty score lower than their experienced colleagues (Mengel, Sauermann, & Zolitz, 2017).
Statement of the Problem
For quite some time now, the researchers of this study have discovered that certain developmental processes in preparing pen and paper examination questions are fraught with inadequacies. It has been observed that not all examination questions are set using test blueprint. In a recent marking exercise of examination scripts of NOUN across the country using lecturers from other universities, several lapses were discovered. The wordings of several questions were inappropriate, which led to ambiguity and vague questions. It was also observed that there were no proper instructions on some question papers and several marking schemes provided were at variance with the questions they were meant to be used for.
In view of the above lapses, which threaten the quality and validity of pen and paper examinations, there is need to streamline issues pertaining to procedures in constructing valid essay questions especially in open and distance education. In addition, the literature reviewed above (Nenty, 2004; Opateye, 2013; Ajiboye, Ojo, Saidu & Okesina, 2014) showed a blanket approach without any one of them being particular about open and distance education. This suggests that more studies are needed to ascertain the validity of procedures in preparing and conducting pen and paper examinations in open and distance education institutions for quality education. It is in realization of these gaps that this study sought to evaluate the procedures of developing pen on paper examinations in the NOUN in order to improve validity in open and distance education in Nigeria. Similarly, in view of the dwindling student success in Open and distance education, there is need to explore the academic counselling implications of the study.
Purpose of the Study
The main purpose of this study was to assess the procedure for the development of pen and paper examinations in open and distance education with a view to ascertaining the academic counselling implications of the findings. Specifically, the study examined the appropriate procedures necessary for the development of pen and paper examinations in open and distance education. It also assessed the extent to which NOUN lecturers follow item generation procedures when setting pen and paper examination questions. Finally, the study was designed to assess how NOUN lecturers validate pen and paper examination questions before they are used for assessment.
Research Questions
The following research questions guided the study:
- What are the appropriate procedures necessary for developing pen and paper examination questions in open and distance education?
- To what extent do NOUN lecturers follow item generation procedures when setting pen and paper examination questions?
- How do NOUN lecturers validate pen and paper examination questions?
Hypotheses
The following hypotheses were tested at 0.05 level to further guide the study:
- There is no significant difference in the procedures of developing pen and paper examination questions between male and female lecturers in NOUN.
- Years of experience have no significant influence on lecturers’ procedures in preparing pen and paper examination questions in NOUN.
Methodology
The study employed the simple survey research design to obtain the opinions of lecturers on the development of valid examination questions for pen and paper examinations. This is to sample the views of a selected sample from a large population using a questionnaire in order to make a generalization (Asika, 1991). Thus, opinions of 122 lecturers (73 males and 49 females) who formed the research sample were sought on the appropriate procedures of developing pen and paper examination questions; the extent to which lecturers follow laid down principles of preparing pen and paper examination questions; and how lecturers in NOUN validate pen and paper examination questions before they are administered to students for assessment.
A self-structured questionnaire in a Likert format entitled Examination Development Procedures Questionnaire (EDPQ) was developed by the researchers for the purpose of data collection. The instrument was divided into four sections. Section A sought information on the bio-data of the respondents while Section B sought information from the respondents on the procedures for developing pen and paper examinations in NOUN using seventeen item-statements. Section C equally sought information from the respondents on the procedures for item generation in preparing pen and paper examinations using sixteen item-statements. Section D on the other hand, sought information on the validation procedures of pen and paper examinations using thirteen item-statements. In section B to D, respondents responded to Strongly Agree (SA), Agree (A), Disagree (D) and Strongly Disagree (SD) which were weighted 4, 3, 2, and 1 respectively. The face validity of the instrument was done by two experts in test development. Thus, the final version of the instrument was used after careful scrutiny of the items contained in the draft.
Data Collection
One hundred and thirty copies of the questionnaire were randomly distributed among the academic staff members of the National Open University of Nigeria (NOUN), who are domicile in the Lagos Liaison Office. In the end, only 122 of the questionnaires were returned, duly filled and used for the study.
Data Analysis
The data obtained from the questionnaires were subjected to data analysis using the SPSS tool. The research questions were answered using descriptive statistics (mean and standard deviation) while the hypotheses were tested at 0.05 level using inferential statistics (t-test and ANOVA). The use of standard deviation was to ascertain the disparity or variations in the opinions of the lecturers on the indices of the instrument used for data collection. A mean of 2.5 and above is an indication that an item was accepted while a mean score of less than 2.5 is an indication that an item was rejected by the respondents. This stems from the fact that the instrument was constructed using a four-scale response category.
Results
The results of this study are shown in Tables 1 to 5 below.
| Item-statements | Mean | Std. |
|---|---|---|
| Clearly stated instructions are required in setting questions | 3.91 | .294 |
| Pen and paper examinations should be timed | 3.86 | .351 |
| Open ended questions are required in setting questions | 2.82 | 1.006 |
| Students should be given options in essay examinations | 3.27 | .631 |
| All the questions should be compulsory in essay questions | 1.59 | .854* |
| Course objectives of courses should guide setting of questions | 3.32 | .780 |
| Questions should be set using test blueprint | 2.73 | .883 |
| Questions should be extracted from question bank only | 1.95 | .722* |
| There should be at least 70% content coverage of course materials | 3.32 | .716 |
| Questions can come from other textbooks outside students’ course materials | 2.73 | .827 |
| Questions should be set at the end of facilitation in a semester | 2.91 | .526 |
| Higher order questions are required in setting questions | 3.09 | .526 |
| Marking guide should be provided along with the questions set | 3.36 | .727 |
| Marking guide should be lifted from course materials only | 2.23 | .869* |
| Appropriate language devoid of ambiguity should be used in setting questions | 3.82 | .395 |
| Process of grading should be spelt out in the marking schemes | 3.59 | .503 |
| Questions for self-assessment can be used in setting questions | 2.95 | .486 |
Std. = Standard deviation; *Rejected item-statements
Table 1 indicates that all the procedures for pen and paper development are applied by lecturers of NOUN except three items whose mean ranges 1.59 to 2.23. In particular, the respondents strongly agreed that clearly stated instructions are required in setting examination questions (Mean = 3.91, Std = 0.294); pen and paper examinations should be timed (Mean = 3.86, Std = 0.351); appropriate language devoid of ambiguity should be used in setting questions (Mean = 3.82, Std = 0.395); and process of grading should be spelt out in the marking schemes (Mean = 3.59, Std = 0.503). The standard deviation shows a wide variation of the opinions of the respondents on the assertion that open ended questions are required in setting questions (SD = 1.006).
| Item-statements | Mean | Std. |
|---|---|---|
| The dimensions of Bloom’s Taxonomy are married in setting questions | 3.00 | 1.069 |
| Construction of test blueprint for each course every semester | 2.68 | .945 |
| Emphasis on course content more than course objectives in setting questions | 2.68 | .894 |
| Following proportions based on course contents and | 3.14 | .941 |
| Distribution of questions is based on various levels of course objectives | 3.05 | .999 |
| Items are written from course materials only | 2.95 | 1.046 |
| Items are written and submitted few days to exams to avoid leakages | 2.95 | .950 |
| Avoidance of fractions in allocation of marks | 2.95 | .999 |
| Questions are selected from question bank | 2.50 | .859 |
| Use of moderated questions in setting final examinations | 3.36 | 1.002 |
| Provision of instructions like duration and number of questions to answer | 3.64 | .902 |
| Use of simple language in setting questions | 3.59 | .908 |
| Asking students to explain some concepts based on named authorities | 2.41 | .959* |
| Provision of a two-paged detailed marking scheme for one question | 1.82 | .907* |
| Provision of points to serve as marking guide | 3.00 | 1.069 |
| Assigning more weights to compulsory questions | 3.14 | .990 |
Std. = Standard deviation; *Rejected item-statements
The data in Table 2 shows that all the procedures for item generation in preparing pen and paper examinations are used in NOUN with mean ratings of 2.50 to 3.64 except two items with mean ratings of less than 2.50. The respondents strongly agree that provision of instructions like duration and number of questions to answer (Mean = 3.64, Std = 0.902) as well as use of simple language in setting questions (Mean = 3.59, Std = 0.908) are very necessary in setting valid questions for pen and paper examinations. The standard deviation indicates variations on the opinions of the respondents on the fact that points should be provided to serve as marking guide (SD = 1.069); the dimensions of Bloom’s Taxonomy are married in setting questions (SD = 1.069); and that moderated questions should be used in setting final examinations (SD = 1.002)
| Item-statements | Mean | Std. |
|---|---|---|
| Moderation of questions is carried out externally | 3.41 | .590 |
| Questions are moderated alongside marking schemes | 3.36 | .953 |
| Deans of faculties vetted the questions before external moderation | 2.59 | .854 |
| All questions are subjected to external moderation | 2.73 | 1.120 |
| Only 400 level questions are moderated externally | 3.09 | 1.231 |
| Postgraduate questions are subjected to external moderation | 3.50 | .859 |
| External moderators are guided by test blueprint | 3.18 | 1.097 |
| Re-setting of questions if found wanting during moderation | 3.45 | .858 |
| Checking of the weights assigned to each question | 3.55 | .596 |
| Timing of marking exercise | 3.50 | .598 |
| External moderation of extermination scores | 3.23 | .813 |
| Internal moderation of examination scores | 2.64 | .581 |
| Release of examination results on time to provide feedback | 2.95 | .899 |
Std. = Standard deviation
The data in Table 3 show that all the items listed for validation of pen and paper examinations are employed in NOUN. However, the standard deviations of some items indicate divergent views by the respondents. The respondents’ opinions differ in the assertion that all questions must be subjected to external moderation (SD = 1.20); only 400 level questions would be moderated externally (SD = 1.231); and that external examiners should be guided by test blueprint (SD = 1.097).
| Gender | N | Mean | Sth | Standard Error | df | t | Sig. |
|---|---|---|---|---|---|---|---|
| Male | 73 | 50.50 | 2.844 | .821 | 20 | -1.223 | .180 |
| Female | 49 | 52.60 | 5.082 | 1.607 |
The data in Table 4 indicates that the t-test result, t(122) = -1.223, P = 0.180 is not significant at 0.05 level of significance. This suggests that the hypothesis of no significant effect of gender on the ratings of the respondents on the procedures for developing pen and paper examinations is hereby accepted. By implication, the respondents, irrespective of gender had similar opinion about what constitute appropriate procedures for developing pen and paper examination questions in NOUN and other open and distance learning institutions.
| Groups | Sum of Squares | df | Mean Square | F | Sig. |
|---|---|---|---|---|---|
| Between Groups | 216.658 | 2 | 108.329 | .866 | .436 |
| Within Groups | 375.933 | 119 | 125.049 | ||
| Total | 592.591 | 121 |
The one-way ANOVA result in Table 5 shows that the F-ratio of 0.866 is not significant at 0.05 level (P > .05). By implication, therefore, the hypothesis which states that years of experience have no significant influence on lecturers’ procedures in preparing pen and paper examination questions in NOUN is hereby accepted. This shows that years of experience among academic staff in NOUN have no significant influence on their procedures in preparing pen and paper examinations.
Discussion
Procedures for developing valid pen and paper examinations that can achieve desired results and global standard in line with the laid down rules is always a herculean task. This study observed that NOUN lecturers follow due process in developing pen and paper examinations. For instance, in setting questions, required instructions are usually clearly stated and timed with optional and open ended questions; course objectives using test blueprint form the benchmark for setting questions; and appropriate language is often used in setting questions and marking schemes. In addition, higher order questions are usually set in examinations at the end of facilitation in a semester with the process of grading clearly spelt out in the marking schemes. These observations are in contrast with the positions of Nenty (2004) and Opateye (2013) that teachers do not follow item generation processes in conducting tests and examinations. For illustration, all the questions in essay examinations in NOUN are not usually made compulsory, questions are not usually extracted from question bank only; and marking schemes are not lifted from course materials. This is to demonstrate that each lecturer working with NOUN often works hard to set fresh and valid questions for pen and paper examinations.
This study also observed that the procedures in item generation during the process of preparing pen and paper examinations in NOUN encompasses the dimensions of Bloom’s Taxonomy, construction of test blueprint for every course with emphasis on course objectives and course contents. Items are written from course materials given to students and externally moderated questions are used for final examinations. Besides, students are not usually asked to explain any concept according to named authorities in their course materials to avoid rote learning. In the marking schemes, more weights are usually given to compulsory questions and points are given using simple language to serve as a marking guide. These observations are in line with the position of Agu (2014) and Canadian Public Service Commission (2017) that the process of laid down rules in generating items for formative evaluation enables students to perform well in schools. However, Al-Qdah and Ababneh (2017) do not believe that assess type determines students’ performance.
Furthermore, it was revealed that validation procedures of pen and paper examinations in NOUN incorporates external moderation of examination questions and marking schemes, using test blueprint as a guide. Also included are re-setting of questions if found wanting, careful scrutiny of answer scripts/sheets during test administration, orderly submission of examination scripts, timing of marking exercise and moderation of examination scores before results are approved and released to students. These are in agreement with the assertion of Kim, Smith and Maeng (2008) that to validate the process of preparing pen and paper examinations in open and distance education is in order since online distance education does not strictly follow the principles suggested in the literature.
This study equally revealed no significant difference in the procedures of developing pen and paper examination questions between male and female lecturers in NOUN. This result is contrary to the findings of Mengel, Sauermann & Zolitz (2017) who observed gender and seniority differences in teacher evaluation and assessment development procedures. The result of this study is a clear testimony of gender equality that is being preached and practiced over the world today. In particular, this is in line with the emphasis on gender equality by the Common Wealth of Learning (COL) on issues relating to open and distance education. Similarly, it was observed that years of experience do not influence lecturers’ procedures in preparing pen and paper examination questions in open and distance education. This could be that the younger ones in the university system are being mentored by their experienced counterparts in the process of learning the rudiments of setting valid and standard examination questions, especially in open and distance education. It is not inappropriate therefore, to conclude that in open and distance education institutions, lecturers interact meaningfully in carrying out their academic activities.
Implications for Academic Counselling
Academic counselling is an aspect of counselling services that assists learners in making adequate academic adjustment in both conventional and distance learning institutions. Such assistance increases students’ academic persistence and achievement, school attendance, better self-concepts and improved attitudes towards school, work and peers (Border & Drury, 1992). The purpose of academic counselling is to equip students with the skills for achieving fulfilled academic life; assist them develop effective time management skills; provide information on effective study behaviours; guide students on course unit system; enable students avail themselves of available support services in schools and create examination awareness among students (FME, 2000). Thus, it is the professional responsibility of counsellors to assist students in coping with educational issues, especially examination anxiety and tutor them for higher level achievement. In line with this, the findings of this study have various implications for academic counselling in open and distance learning institutions where students mostly engage in virtual activities except during pen and paper examinations, which must be written in their designated study centres.
During the orientation programmes, usually organized for fresh students at the beginning of every academic session, counsellors owe it as a professional responsibility to get students informed on the patterns and shape that pen and paper examinations are taken in their institution. In actual sense, student counsellors in the Study Centres should always emphasize the rubrics of writing pen and paper examinations during orientation programmes for fresh students and other students during student forum, which should be organized at least twice in a semester.
Furthermore, the Learner Support Services are for both students and staff. Hence, there is equally a serious need to organize an orientation programme for the newly employed academic staff after each recruitment exercise by the Directorate of Learner Support Services. During such a time, a counsellor from the directorate would be mandated to talk on the appropriate procedures for preparing pen and paper examinations that can enhance validity, excellent academic performance and student success in open and distance learning institutions.
Finally, it behoves on student counsellors to educate their open and distance learning students that to perform well in examinations requires adequate preparations and amount of study put in before every examination. According to Al-Qdah and Ababneh (2017), the type of examination adopted for assessment does not determine students’ performance but their preparations and amount of study. This information is very important so that students will start on time to equip themselves with what it takes to do well in examinations. Hence, students should be encouraged to organise themselves into various study groups where they can jointly engage in group discussions, which should be monitored by counsellors before and during examinations in order to perform well and achieve their academic dreams. During their meetings, students can come across some past questions and study the pattern through which such questions were developed and use the knowledge to predict what they might face in their future examinations.
Conclusion and Recommendations
Assessment of pen and paper examinations development procedures often encompasses investigation into procedures for item development and generation as well as validation processes after setting questions. This is to show the extent to which the examinations could be valid and reliable. In NOUN, the procedures for developing pen and paper examinations include clearly stated instructions, setting of open ended questions, use of course objectives and course contents based on test blueprint to cover at least 70% of course materials in setting questions at the end of each semester after facilitation; high order questions in unambiguous language with clearly constructed marking schemes are usually set. Examinations are also timed. Investigations also revealed that NOUN validates her pen and paper examinations through both internal and external moderation of examination questions and scores. In developing pen and paper examination questions, therefore, the preliminary instructions on the question papers and a comprehensive marking guide cannot be ignored. Finally, it is unreasonable and highly against examination ethics to ask students to explain some concepts according to some named authors in the course materials in order not to encourage rote learning.
In line with the above revelations, it is recommended in this study that in open and distance education:
- Lecturers should endeavour to follow the laid down procedures in developing len and paper examination questions. The appropriate procedures recommended in this regard include listing the tasks required for assessment, specification of the response format, number of questions, time allocation and scoring guide.
- Appropriate procedures using Bloom’s Taxonomy should be applied in constructing pen and paper examinations to ensure content coverage and validity.
- Questions meant for pen and paper examinations should be subjected to both internal and external moderation to ensure quality.
- All the lecturers, irrespective of gender should be exposed to the tenets of the procedures for developing pen and paper examination questions through workshops and seminars, especially before the commencement of any semester examinations.
- The green horns in academics should be mentored by their senior and more experienced colleagues in the art of setting examination questions. These experienced lecturers should be used to vet and moderate the examination questions set by the junior faculties before external moderation for the purpose of sustained quality assurance and quality control.
References
Agu, N. (2004). The effect of formative evaluation in reducing variations in the rate at which students of different achievement levels master instruction in mathematics. The Nigerian Educational Psychologist, 3(1), 66-79.
Ajiboye, S. K., Ojo, O. J., Saidu, A. B. & Okesina, F. A. (2014). Sources of examination anxiety among final year students of University of Ilorin, Nigeria. Studies in Education, 14(2), 213-219.
Al-Qdah, M. & Ababneh, I. (2017). Comparing online and paper examinations: Performances and perceptions of Saudi students. International Journal of Information and Education Technology, 7(2), 106-109.
Asika, N. (1991). Research methodology in the behavioural sciences, Lagos: Longman Nigeria PLC.
Border, L. D. & Drury, S. M. (1992). Comprehensive school counselling programmes: A review for policy makers and practitioners. Journal of Counselling and Development, 7(4), 104-108.
Chima, I. M. (2006). Self-confidence training and effective study habits as veritable strategies for curbing examination malpractices among secondary school students. The Nigerian Educational Psychologist, 4(2), 314-327.
Ezeh, C. O. & Onah, P. C. (2005). Measurement and evaluation in education. Enugu: Computer Edge Publishers.
Falayajo, A. (2004). Methods in evaluation. In O. A. Afemikhe & J. G. Adewale (Eds.). Issues in educational measurement and evaluation in Nigeria. Ibadan: ICEE.
Federal Ministry of Education, FME (2000). Blueprint on guidance and counselling for educational institutions in Nigeria, Abuja: Art and Culture Education Unit.
Kayode, B. (2006). Practical approach towards eliminating examination malpractices in Nigerian higher institutions. The Nigerian Educational Psychologist, 4(2), 262-268.
Kim, N., Smith, am. J & Maeng, K. (2008). Assessment online. Retrieved from c:\Documentsandsetthings\user\Desktop\ASSESSMENT ONLINE.htm.
Mengel, F., Sauermann, J. & Zolitz, U. (2017). Gender bias in teaching evaluations. Retrieved from http://ftp.iza.org/dp.11000.pdf
Metzener, S. (2013). Teacher bias in public evaluation: A critical analysis. Journal of Teacher Evaluation, 22(7). 45-47.
Nenty, H. J. (2004). From classical test theory to item response theory: An introduction to a desirable transition. In O. A. Afemikhe & J. G. Adewale (Eds.). Issues in educational measurement and evaluation in Nigeria. Ibadan: ICEE.
Okonkwo, C. A. (2013). Building quality item bank for assessment of learning outcomes. In I. O. Salawu, A. I. Ikeotuonye & J. O. Inegbedion (Eds.). Perspectives on Nigerian education (51-77). Lagos: School of Education, NOUN.
Okoye, N. N. (1986). The challenges of Nigerian school certificate examinations. Ibadan: Evans Brothers.
Opateye, J. A. (2013). Test item generation and administration among secondary school teachers: Implications for achieving goals of education for all (EFA). In T. A. Bolarin & G. C. Unachukwu (Eds.). Education for all: Progress and challenges (334-343). Lagos: Nigerian Academy of Education.
Osuji, U. S. A. (2014). Group self-assessment and learning outcomes. NOUN Journal of Education, 1(1), 28-35.
Robbles, M. & Broathen, S. (2002). Online assessment techniques. Delta PI Epsilon Journal, 44(1), 39-49.
Schafer, W. D., Papapolydorou, M., Rahman, T. & Parker, L. (2005). Effects of test administrators’ characteristics on achievement test scores. Paper presented at the conference of the National Council on Measurement and Evaluation in Montreal, Canada.
Shukla, R. (2005). Dictionary of education. New Delhi: A. P. H. Publishing Corporation.
Ukwueze, A. C. (2012). Correlation between web-based continuous assessment and examination scores in open and distance education: Implications for academic counselling. Malaysian Journal of Distance Education, 14(2), 27-37.