On my working experience in the University of Tartu, Estonia
The Study Regulations of the University of Tartu (https://www.ut.ee/en/university/documents) enact grading of students according to their gained learning outcomes (§ 93, 94). I very much like this, in particular because the position of the modern education science also supports this approach (e.g. Biggs, J., Tang, C.: Teaching for Quality Learning at University). This opposes to rank-based grading which means that the teacher must follow a certain distribution of grades predefined in terms of percentages of students taking the course. Rank-based grading is still popular in the world, although criticized by scientists because it implies that the grades a student gets much depend on the skills of other students who took the same courses and cannot be associated to the learning outcomes achieved by that particular person. Grading according to the gained learning outcomes is stated also in the Estonian decree on grading standards in higher education.
The reality unfortunately differs from what one could expect by the documents. I worked many years in the Institute of Compute Science of the University of Tartu and I experienced that, at least in this institute, the officially required standard-based grading was followed on paper only. Prof. Vene, the head of the Chair of Programming Languages and Systems where I was working, explicitly imposed requirements to the teachers of courses in terms of passing rate. For many years, he also required the students' average feedback grade to every course to exceed a certain predefined level; if it was lower, he required changes to increase this indicator. While the feedback grade was above the norm, nobody among the authorities was ever interested what was actually done in the course. The students' average feedback grades enjoyed a status of a holy cow in the whole institute, although research in the world and in the University of Tartu in particular had shown that students' feedback grades poorly correlate with the actual teaching quality. (For just one example: A research done in the University of Tartu showed among other results that, on average, the highest average feedback grade is obtained by lecturers who only have a bachelor degree; lecturers with a master degree get lower feedback grades and those with PhD degree obtain the lowest feedback grades.)
Prof. Vene has even changed the leading lecturers of courses against their own will in order to achieve higher passing rate or student feedback grade. When collecting students' feedback in the form of grading was finally stopped by the university, Prof. Vene explicitly expressed himself being disappointed by this decision and promised from that moment on to assess the quality of lecturers' work by the grade distribution in their courses. This was despite it contradicts the aims of standard-based grading at their very grounds! The University of Tartu offers a course for the academic staff on learning and teaching in higher education; in this course, specifying the correspondence of grades and expected learning outcomes in terms of percentages of students who get the grade was explicitly presented as an example of a wrong way of doing it. Prof. Vene has taken this course, too. It seems that he thought that the wrong way is wrong only if written on paper but not when implemented in practice. (I don't know if Prof. Vene actually fulfilled his promise, but clearly the promise was not a joke.)
I must say that the authorities of the institute have been largely successful in carrying out the aforementioned improper grading principles because the academic staff of the institute seemed to be mostly satisfied with this state of the art. And apparently, the requirements about grade distribution have been unidirectional: There are bachelor level courses with hundreds of students where every year a majority of students pass with grade A or B and only a few ones fail, and I never noticed the authorities of the institute complaining about it.
A particular issue related to this topic is academic fraud. According to my experience, people working in the Institute of Computer Science are quite uninterested in dealing with cheating students. The fact that every year several cases of irregularity are discovered and reported is not an evidence of a good job, because there are many more cheating cases that stay unrevealed. If the number of mice near a sleeping cat is very large then some of them run into the cat's mouth just by statistics.
I must give some evidence about the amount of academic fraud among the bachelor level computer science students of the University of Tartu. When I taught a functional programming course for the first time, I was not aware of some tricks students use for cribbing, so I left these opportunities open unintentionally. In the middle of the term, I discovered it and closed the backdoors. When the course ended, I found that about 10% of students who took the course had had weekly test results near maximum before the change and got near zero results after the change. The percentage of cheaters was probably larger since not all of them might have got nearly all right answers with help of cribbing. And this is just the state of the art in the specialty courses. I have also taught math courses to computer science students and was faced with the phenomenon that many students who successfully solve an exercise in graded homework are not able to solve a similar exercise in the classroom. Apparently the percentage of students who use illegal help at least to some extent is even much larger in math courses.
Leaving backdoors for cheating open is typical in the courses of this institute according to my experience. Even worse, the lecturers and authorities are conscious of it but just don't care of fair grading. In March 2020, at a seminar of the Chair of Programming Languages and Systems, this issue was discussed and it turned out that if no more than 20% of exam points can be earned with illegal help then most staff members consider the grading still acceptable. Note that 20% of exam points in the standard grading system means two grades; so someone who deserves an E (i.e. the minimal level of passing) can get a C ("good") and the university staff considers it acceptable!
A typical backdoor for cheating is weekly homework containing easy training exercises. Giving such homework and offering exam points for doing it is very popular among the lecturers of this chair, but it is nearly impossible to reveal cheating in them (for several reasons, e.g., easy programming exercises have short solutions which are often identical just incidentally, and asking the students an oral explanation is problematic due to the large number of them). The lecturers tend to justify the backdoors with the need to motivate students to work more on the course material. If I need to go to the other end of the town quickly, does it justify stealing a car? Of course not. Why do people think that the need to motivate students justifies laundering their illegal activities? It is not a matter of motivation in the first place but a matter of pedagogical competence, moral and ethics.
The aforementioned seminar took place before the Covid-19 pandemic measures came into force or were even discussed, which means that the position of the staff reflected the normal situation. Under the pandemic measures, fair grading was given even a lower priority. Exams in large courses with hundreds of students were organized in the web. Performances were recorded into videos and the videos were sporadically checked but the chances to reliably reveal cheaters under such conditions are very small. At the same time, the Vice Rector for Academic Affairs, Prof. Valk, when answering to people's concerns about cheating, declared that there exist tools that can ensure honest performance of exams in the web. My experience does not support this claim. For instance, when I wanted to use Safe Exam Browser (https://safeexambrowser.org/), I was discouraged by IT support guys claiming that this software is suitable for simple cases such as Moodle tests only. I don't know about any such software tool having been used in the Institute of Computer Science or in other institutes for regular exams.
During years, I have many times complained about the situation to Prof. Vilo, the head of the Institute of Computer Science, but without any positive results. I must say that he is a contradictory figure. Without doubt, he has had a positive influence to the research done in the institute. Doctoral level studies in the institute reached the world top class largely due to Prof. Vilo leading the institute. But the quality of undergraduate studies has remained unchanged for the decade he has led the institute, if not even decreased. Prof. Vilo has sonorous rhetoric ("we have world-class computer science education", "we must drive all cheating students out of the university" and, most remarkably, "let's make the impossible possible") but it is not supported by any real actions. I also wrote an e-mail to Prof. Valk, describing my concerns about the gap between the teaching results on paper and in reality, but she even didn't essentially answer (she sent a purely formal reply).
So it turns out that institutes in the University of Tartu are mafia-like units that can establish their own rules and nobody is even willing to enforce the official rules, as long as everything looks correct on paper. Honesty is not valued. The latter expresses not only in grading. For instance, I noticed more than once that minutes of the meetings of the institute council reflected the circumstances of decisions in an untrue way. Knowing the mentality widespread in the Institute of Computer Science and more generally in the University of Tartu, I would not blame the particular person who wrote the minutes. For another instance, several colleagues who assisted me in teaching courses during many years had agreed before the courses with certain principles of organizing exams and grading but later turned out to have violated the principles without any notice or discussion with me.
Let me give one more example. One of my colleagues from the Chair of Programming Languages and Systems once mentioned me that there has been a case of Prof. Vene being included into the list of authors of a research paper with contribution he feels is not enough to be a coauthor. As the University of Tartu has a Code of Conduct for Research Integrity (https://www.ut.ee/sites/default/files/www_ut/code_of_conduct_for_research_integrity_eng.pdf) that regulates such issues, I recommended him in an e-mail to turn to the research integrity councellor. But he ignored that e-mail, and after insistence by me, replied that no-no, everything is correct, the contribution was sufficient. Of course the colleague could be mistaken in his initial judgement. But why can't I believe it? Mainly because of the reason why we started talking about that case: He offered me an opportunity to be included into the list of authors of a planned new research paper just for being the supervisor of a PhD student who had to be one of the main contributors. This suggests that cases of research papers having formal authors with no real contribution might be regular. (I declined the offer of course.)
The habits described in this post place the University of Tartu to the same level of immorality together with the students who practise academic fraud. These habits are similar to corruption by nature and should also be condemned alike, but this has not happened (or happens on paper only). May this post be a warning to anyone who will ever consider working in the University of Tartu.
You highlight some very valid criticisms but the analysis why things are how you describe them is lacking. I do agree that grade inflation is bad and cheating is a problem, they're a problem everywhere really, but the proposed or implied solutions are frankly bad or just naive.
ReplyDeleteI'd also address a few other points in your writing:
> A research done in the University of Tartu showed among other results that, on average, the highest average feedback grade is obtained by lecturers who only have a bachelor degree; lecturers with a master degree get lower feedback grades and those with PhD degree obtain the lowest feedback grades.
The work you cite has been taken without precaution of bias or a cause of second order. If you consider the facts that MsC and PhD students probably rate courses more objectively and that harder courses are probably taught by more competent people, it starts getting difficult to simply compare average opinion of the students. Making any conclusions based on that is only for the most bold but not necessarily wise.
> Explicitly imposed requirements to the teachers of courses in terms of passing rate. For many years, he also required the students' average feedback grade to every course to exceed a certain predefined level
Here you haven't asked yourself, "what does this rule discourage?" I think it would be pretty safe to assume that a single lecturer shouldn't cause a huge dropout rate and that the learner's wellbeing is important. (Keep in mind that a single student initially has maybe 0-6EAP they can fail.) If every second course is perceived shit then there would be little motivation for students to continue. There’s an immensely delicate balance here - being able to continue at all and demanding the best from students. Policy changes in that aspect have really long-lasting effects – lack of researchers due to dropout or low research output due to poor education.
So I would again ask you to really think why the rules are in place before labeling them as inherently bad, maybe they're just the best of the worst.
> I found that about 10% of students who took the course had had weekly test results near maximum before the change and got near zero results after the change.
I'm not entirely familiar with your approach, I presume you disabled feedback? If so, are you surprised that without feedback some students find a task useless for learning and won't do it? If so, why are you surprised?
> I have also taught math courses to computer science students and was faced with the phenomenon that many students who successfully solve an exercise in graded homework are not able to solve a similar exercise in the classroom.
There are of course those that pay for those tasks to be solved. But the larger factor here is smaller amount of pressure, more time to solve and find materials. If homework would be solved worse than exercises in the classroom, something is probably wrong with your methods. For example, the materials available to the students could be dogshite, which is not unusual at all.
> Note that 20% of exam points in the standard grading system means two grades;
But not a passing grade if you don't learn the material at all.
> For instance, when I wanted to use Safe Exam Browser, I was discouraged by IT support guys claiming that this software is suitable for simple cases such as Moodle tests only.
Yes, IT support is correct. SEB is cumbersome and doesn't really solve the problem. Cheating is really not a technical problem that could be solved that way. It takes a lot of careful effort, none of which are additional restrictions.
I think a lot of your problems really stem from the fact that you don't perceive education (including cheating and what require from the students) a complex human interaction based challenge with tradeoffs at every step.
Lack of familiarity with humans is a visible common characteristic throughout your entire text.
I was thinking for a while if I should reply or not to the post by the anonymous void loop. I decided to do it because it presents typical arguments heard tens of times in the University of Tartu. These arguments serve the purpose to divert attention away from the unethical or illegal behavior of the university staff.
DeleteAs I wrote, standard-based grading is required by official Study Regulations in the University of Tartu and also by a state decree. Grades the students get in courses are printed on an official appendix of the univerisity diploma to be used by employers, grades obtained in previous terms are used during studies for selecting the best students who get scholarships. Grades are more than just feedback to students on their learning progress, whence the university is really responsible for not allowing students to cheat. Surely, car robbers also can have plenty of excuses and reasons why they urgently needed the car. This does not make car robbing an acceptable choice. Or suppose a coach who recommends their athletes using illegal drugs to achieve better results replies to criticism: "Everybody does it. You do not understand humans!" Should the society indeed be satisfied with this?
Concerning my experience having 10% of registered students cheating in my course, the purpose why I presented it as an example was not to express my surprise. I was not surprised. The reason was to justify my expectation of stricter measures against cribbing. If there were no or almost no cheating students then there would also be no need for such measures, but my experience shows that this is not the case.
The void loop justifies accepting grading systems that allow up to 20% of exam points with illegal help with the fact that 20% of exam points does not enable a student to pass the course when having done no real learning progress. Really? Is pass/fail indeed the only important piece of information about the student's skills? If so then why are the grades there? There are many honest students studying in the University of Tartu (actually, I still hope that they are a majority), and for many of them, making progress in the studies requires hard work. They often do not get the highest marks. Cheaters who obtain 20% of exam points with illegal help can beat them while having done even lower real progress. In the future, they could be preferred by an employer as a candidate to a vacant job. Why should the honest students who get deceived this way be satisfied with your elegant answer: "He must have done at least some real progress on the course material because otherwise he would not have passed. See you later!" Note that bringing up complaints about the grading system because it allows getting points by cheating is difficult for students because they might get condemned by their comrades who like the features of the grading system. Thus there being no complaints does not imply that all students are satisfied.