SOM Review of High Stakes by Drummer

Owned by Kirk Alexander
Last updated: Oct 26, 2006Version comment

Here is a listserv that John suggested might be a great place to post soem
of our general questions: http://omerad.msu.edu/dr-ed/index.html

From 2006 CRIME meeting:

6. Online High-Stakes Exams
<Group Discussion>
Stanford uses WebCT for some final exams and for some clerkship SHELF-like exams. Otherwise just for quizzing and self-tests.
UCLA distributes block exams with the Angel CMS. They use randomized questions from a question databank. Faculty appreciate that exams are graded automatically and easily. Internet explorer is controlled during exam by a plugin tool that doesn't allow them to leave anything within Angel site
Drew uses Angel LMS for online exams.
UC Davis uses a home-grown system from the 90's for quizzing. For an exam lab, they set up 50 portable computers in a lecture hall and students come in 2 waves.
UCSD is constructing a new building and they hope to set up an exam room where administrators can watch all students taking the exam, using video screens etc.
Colorado does testing in a home-grown system.
UCSF uses WebCT for 4 hour essay exams in surgery clerkship but they're moving to a SHELF exam. Students write 8 of 10 cases. Paper exams are scanned into a scantron machine and scores are presented to students in WebCT. For integrated exams they split questions up by topic. Version 4 of questionmark is supposed to work for high-stakes exams, but the software is generally only currently use for quizzes.
General thoughts:
? students logging in all at once causes server blowout
? do students log in for the online exam at a certain time or is it an open timeframe?
? Online exams- students don't know how much to write into the text box. The visual cue of paper allows you to limit the amount of space. It took students about twice as long. Added a wordcount button to count the words.
Benefits of electronic exams
? Using questions in other ways
? Feedback on questions
? Online exams allow you to present a rubric or best-answer for graders of essay questions
? Automated essay grading
Drawbacks of electronic exams
? In middle of exams, it's critical to have a support person immediately available. Servers go down.
? Difficult to find a facility for testing of all students at one time.
? Lack visual cues of paper for estimating how much to write for an essay (see above)

----Original Message----
From: Medical Education Research and Development DR-ED@LIST.MSU.EDU On Behalf Of Forehand, Cynthia J
Sent: Friday, November 18, 2005 9:31 AM
To: DR-ED@LIST.MSU.EDU
Subject: Re: DR-ED Computerized Testing

At the University of Vermont we do all preclinical examinations - and many clerkship examinations - on computers. Students use their own laptops. Formative assessments are delivered through Blackboard. Summative assessments are delivered through Blackboard and Secure Exam Browser, which essentially disables all functionality of the laptop except seeing the exam and selecting answers. All our students (106) can take a secure exam at the same time in a wired lecture hall; privacy screens prevent side to side viewing of screens and a proctor is in the room as there would be for a written exam. The questions for many exams are delivered in random order. One major disadvantage is the statistical package available through Blackboard. We have to pull the data and run a SAS program to do item analysis and regrade after flawed items are pulled. This adds 2-3 days before students see their scores, a source of disgruntlement... -Cindy Forehand

----Original Message----
From: Medical Education Research and Development DR-ED@LIST.MSU.EDU On Behalf Of Nehad El-Sawi
Sent: Friday, November 18, 2005 9:44 AM
To: DR-ED@LIST.MSU.EDU
Subject: DR-ED Computerized Testing

Dear Colleagues,

I am trying to find out information about the use of computerized testing in years I and II courses. My questions are:

1. Do you use computerized testing for formative assessment?
2. Do you have dedicated facility for that purpose?
3. What kind of soft ware is used?
4. How is proctoring done?

Thanks
Nehad

Nehad El-Sawi, Ph.D.
Senior Associate Dean for Academic Affairs/Curriculum Governance and Chair of Microbiology Kansas City University of Medicine and Biosciences 1750 Independence Avenue Kansas City, MO 64106 816-283-2210 - phone 816-460-0559 - fax nelsawi@kcumb.edu

Hi Nehad et al.
1. At UTMB we have used computerized testing for 2 years in several components of our Gastrointestinal/Nutrition course for Year II students. For formative assessment, the students take practice quizzes for the midterm and final exams and also over learning issues in the PBL cases. Their final image exam is also computer-based, covering normal histology and all pathology, both gross and histopathology. Our class is over 200, and we have two testing facilities on campus of about 100 computers, which allow us to give the exam to half at a time. I feel more comfortable, security-wise, using university computers than I would if students were allowed to use their own. 2. Our course directors are the dedicated faculty! We also have outstanding indispensable support from our Instructional Management Office, who arrange all of the details. 3. We use WebCT for the formative assessment and LXR for the summative evals. Advantages of LXR include the coding of questions by several key terms, and the ability to scramble not only the questions but also the answer choices, a deterrent to dishonesty. We also use SQL servers. 4. In addition to the methods above to discourage cheating during the image exam, we have screens over each monitor that make it difficult to read from an angle. We also have a number of different questions, all clinical scenarios, that use the same images. We have just a few proctors, who mainly help students get logged into the exam but who stay for the entire exam.

I look forward to the day when all of our exams are computerized. norma

Norma H. Rubin, Ph.D.
Professor, Department of Neuroscience & Cell Biology The University of Texas Medical Branch Galveston, Texas 77555-1069 Phone 409.772.6657 or 409.772.1294 Fax 409.762.9382

----Original Message----
From: Medical Education Research and Development DR-ED@LIST.MSU.EDU On Behalf Of Nehad El-Sawi
Sent: Friday, November 18, 2005 8:44 AM
To: DR-ED@LIST.MSU.EDU
Subject: DR-ED Computerized Testing

From: Medical Education Research and Development DR-ED@LIST.MSU.EDU On Behalf Of Dean Parmelee
Sent: Friday, November 18, 2005 7:43 AM
To: DR-ED@LIST.MSU.EDU
Subject: Re: DR-ED Computerized Testing
We have had all exams administered by computer for several
years, including the objective 'formative' ones. We have
dedicated space for 1/2 of a class, so we do back to backs -
new renovation of building will have sufficient space and terminals
for entire class. We use WebCT, although we are developing
our own program to have better stats analysis and banking
of questions. Proctoring done by computer facility personnel,
honor code, barriers between stations. We use Macs throughout and
are fortunate to have the latest ones which provide great images,
graphics, reliability. More expensive, but well worth it.

Dean Parmelee, M.D.
Associate Dean for Academic Affairs
Boonshoft School of Medicine
Wright State University
Dayton, Ohio

From: Medical Education Research and Development DR-ED@LIST.MSU.EDU On Behalf Of CAROLYN E PRUCHA
Sent: Friday, November 18, 2005 7:10 AM
To: DR-ED@LIST.MSU.EDU
Subject: Re: DR-ED Computerized Testing

Good Morning Nehad,
I initiated computerized testing for the Univ. of WI Medical School many years ago. We continue to use this format for some of our course exams. We have never had enough computers to do the entire class in one sitting so have done it the testing in several groups. Proctoring is very important since the exams can be copied, e-mailed. etc. unless you have a server set-up that prohits this activity. We recently moved to a new building and have much more advanced capabilities in this area. The talk now is to get more computers so we can test the entire class in one sitting. This would be extremely helpful. We have some simple/homemade dividers to put up inbetween computer stations for testing. Our computer lab is open to everyone other times and the dividers are stored away.

We started with Web-CT and do not recommend Learn@UW.

If you need more information just e-mail.
Carolyn