Skip to 0 minutes and 10 seconds The first step in establishing an assessment program for rotation is to design the format of assessment. Now the assessment activities should flow naturally from the learning activities. Whatever activities are assignments students need to complete. We need to make sure that the assessment is focused on those activities. Because those are the activities that we’ve determined. It will enable the students to achieve the learning outcomes. We want to make sure that our assessment is clearly focused on the learning outcomes. We need to establish the relative priorities for each item that’s gonna be assessed. We may have several different activities or assignments that are being assessed as part of the rotation.
Skip to 0 minutes and 50 seconds They might not all have the same weight or the same importance, in the same relative importance, in terms of the rotation. For example, if students are spending a lot of time in patient monitoring, if they spend three or four hours a day monitoring patients on a critical rotation, infectious disease rotation, or oncology rotation whatever it may be. That should carry a very heavy weight in the assessment. And so when we’re determining how we’re going to evaluate student performance, we would need to make sure that patient monitoring carries a very heavy proportion of the assessment process.
Skip to 1 minute and 30 seconds Other assignments, such as giving a presentation or writing up a short report should certainly be assessed which should not carrying as much weight that something as important as patient monitoring. So we first have to consider all the aspects, all the significant assignments and activities. And make sure that we’ve assigned relative importance in terms of achieving the outcomes for the specific rotation. We also need to determine specific criteria for any activity or assignment. It needs to define exactly what it is that will be assessed. For example, if a student is going to give a verbal presentation, are we going to assess the slides that they prepare? The way they speak during the presentation?
Skip to 2 minutes and 16 seconds Or is it gonna focus more heavily on the content of the presentation? The conclusions that they form whatever the case may be. We need to define specific criteria that will be assessed. And these criteria not only guide the precentor so that every time a student delivers a verbal presentation, a receptor is using the same criteria to evaluate performance. But it also communicates to the student how they’re going to be graded. And help them to prepare effectively, so they have the best possible chance of achieving the learning outcome. We also need to define rating levels and the corresponding point values. Now I would strongly suggest to keep things as simple as possible, and as consistent as an objective as possible.
Skip to 3 minutes and 4 seconds It is to stick with a smaller number of ratings and rating scale. For example, if you go with three levels of ratings, you might use what one method that I have used in the past, is “meet standards”, “partially meet standards or expectation”, and “does not meet expectations”. A five-point scale could use a standard grading system. Such as “ A, B, C, D, F”. You could use a 5 rating scale with descriptions. Such as “excellent”, “very good”, “good”, “fair” and “poor”. The bottom line is for the preceptor, is the more levels of a rating scale, the more difficult it is to discern between one level and the next.
Skip to 3 minutes and 50 seconds It’s very easy to determine whether a student meet the expectations, partially meets the expectations or does not meet the expectation. As compared to determining should they receive a B or a C on a five-level rating scale. So the first recommendation is to keep the levels of ratings to a minimal, three or four. Once you get up to five or six levels of ratings, it becomes much more difficult to distinguish one from another. Another consideration about the point values is to make sure that their proportionate to the grading scale of the rotation or of the course is an example.
Skip to 4 minutes and 31 seconds If we use a three-level rating scale, we could give three points to the highest value, two points to the next highest value and one point to the lowest value. That may seem fair, simply based on a three-point scale. But what if the grading scale of the rotation indicates that the cutoff between passing and failing is seventy percent, which happens to be the case at my university. In that situation, the student who got the second highest level on three-point rating scale would receive a 2, which is sixty-seven percent of the highest rating 3. So if a student earns the middle rating scale, which should be acceptable, they’re receiving 67 percent, which in the course grading scale is failure.
Skip to 5 minutes and 20 seconds That’s a disproportionate point distribution. What I use for a three-point rating scale which turns out to be more comparable to the grading scale for a typical course is 543. So the student who receives the highest rating receives the full amount five points, a 100 percent. The student who receives the second highest rating receives a 4, which is 80 percent. And the student receives a 3, 60 percent, which is still failing. Lastly, we have to describe how the assessments fit together to determine the grade. And this is actually the first step in the process. I put it at the end because I wanna to cover the other four variables.
Skip to 6 minutes and 3 seconds But the first thing that should be done in defining an assessment process is to establish everything that needs to be evaluated. Determine how many points, the total sum of all those activities and assignments will be. And then divide those points up proportionately between the various elements that are being graded. For example, getting back to the the concept of grading a student patient monitoring. Their patient monitoring is the most important element and a very significant element of the rotation. And we determine that the rotation should be worth a thousand points. We would want to assign a large proportion, maybe as many as half, 500 points out of a thousand, to patient monitoring.
Skip to 6 minutes and 51 seconds Now a single assignment, a single verbal presentation, we would account for much less than that. We may assign that a hundred points. You certainly wouldn’t want to assign a point value for a verbal presentation is equivalent to the daily patient monitoring that the students going to be expected to accomplish. So overall the first step is to design an overview of the full rotation assessment process. How many points are going to be allocated to patients grade for the entire rotation? And how should those points be divided up between all the major activities and all the assignments are going to be assessed? Now how does the level of competence affect the nature of assessment?
Skip to 7 minutes and 41 seconds Now we can’t measure knowledge and performance in exactly the same way. Talk about that in session 1. Students have to know what it is we want them to learn. They have to know how to perform whatever it is that we want them to learn. They have to actually demonstrate it, show it and they have to be able to do it in a real-life setting. Now as we discussed session one, you can kind of break up the “know it” and “know how to do it” as to what takes place in the classroom. That’s rather easily assess with exams and quizzes, which is typically the case and didactic instruction and an APPE rotations.
Skip to 8 minutes and 18 seconds It’s much easier to have students show what they can do. They can turn in monitoring form that show what they’ve been doing to monitor their patients. That can easily be assessed by the preceptor on a regular basis. We can actually observe them performing patient monitoring when they’re on a patient care rotation. So that’s fairly easy to do. Now one other things that I suggest, that can also be used to evaluate how well the students are actually doing what it is we want them to do. In relation to patient monitoring is to offer a verbal final exam, in which the students are asked to SOAP or identify the problem list of a patient.
Skip to 9 minutes and 3 seconds When all the characteristics of the patient had been put on a piece of paper that’s given to them. They have the lab values and description of the patient, physical exam results, all the findings from medical history and surgical history and so on. And the student has to identify the problems and SOAP each problem. To identify the data that relates to the problem. The assessment of the problem and a plan for that problem. The preceptor can then use a checklist to check off when the students identified all the anticipated elements of each of those problem. And give the student two points for each element that they correctly identify.
Skip to 9 minutes and 44 seconds And then during follow-up questioning, the patient can still get perhaps half credits, if the preceptor can question the patient and identify some of the things that were missed during the initial presentation. This type of verbal examination is an excellent way to identify how well the students have developed their patient monitoring skills. How much knowledge they’ve developed in particular a disease states that they had exposure to during the rotation. And it helps to instill confidence in the students. To demonstrate to them how much they’ve actually learned and how much they’ve accomplished in terms of developing their critical skills.
Design the format of assessment
We can’t measure knowledge and performance the same way.
Thus, formats of assessment should be designed. A “rubric” is a common way to be used to design an assessment. In this video, Prof. Brown will explain how to design a rubric.