Summary of Evaluation

Most, if not all, of the issues we see with Samigo Tests & Quizzes can be assigned to one of the following categories.

• Authoring: the process of creating assessments within the Samigo tool
• Delivery: the process of administering an assessment to a group of students or other users
• Reports: collecting statistics on the assessments
• Importing/Exporting: the effort to draw existing assessments into Sakai, and to output them in some format that other systems will be able to use

Authoring

Workflow

Creating a new assessment

When the instructor opens the Samigo tool, she is presented with this screen:

To create a new assessment, she will need to follow these steps:

1. Enter a title in the appropriate textbox
2. Select the desired assessment type (for example, 'Survey', 'Test', or 'Quiz'), and
3. Click the button labelled Create.

If the instructor clicks Create, she will then be presented with a screen prompting her to enter her first question (see immediately below). It's also possible to click on the button labelled Import, which will then allow questions to be imported from a QTI compliant file.

Adding a new question to the assessment

The navigation buttons available on this screen are:

• Add Part: 'Part' is the Samigo term for Section, and allows the instructor to break up a given assessment into discrete chunks that can then be ordered.
• Settings: this will take the user to the complete settings screen, which contains all the fields available for a given assessment type.
• Preview Assessment: this will bring up a screen that actually allows the instructor to walk through the assessment from beginning to end seeing it approximately as a student or survey-taker would.

But under normal conditions, the instructor will most likely first choose a question type from the drop down box containing the text 'select a question type'. Available question types include: Multiple Choice, Survey, Short Answer/Essay, Fill in the Blank, Matching, True/False, File Upload, and Copy from Question Pool.

If she selects Multiple Choice, for example, then the screen will immediately refresh.

Composing a multiple choice question

In addition to providing instructors with the ability to build assessments of the six built-in types: Formative Assessment, Problem Set, Quiz, Survey, Test, and Timed Test, the authoring tool also allows for these types themselves to be customized, and for new assessment types to be created. Of course, there is still plenty of functionality that could be added – and most of the effort out there in the community seems to be either fixing bugs with or adding new functionality in this space.

Currently, the instructor can control, for example, how many times an assessment can be submitted, whether late submissions will be accepted, whether to provide the user with feedback after an answer is submitted, and even whether the program should accept submissions only from specific IP addresses. It's also possible to break up an assessment into discrete sections and to control how questions are presented to the user – i.e. one per page or in batches.

Unfortunately, the user interface itself is neither exactly intuitive or streamlined. For example, in order to 'publish' an assessment – a necessary step if you want anyone to be able to use it – it's necessary to click on 'Settings' before you can find or select the 'Publish' button.

Also, the process of creating an assessment requires each question to be entered one-at-a-time and on its own screen. This means that there's no easy way for an instructor to simply copy and paste a list of questions that she's written up in a word processor.

Tasks/Feature Requests

• There's some interest in sharing assessment types so they could be modified on a departmental basis, for example SAK-5199 and SAK-3526
• Request for more flexibility to accommodate students with special needs - allow instructor to adjust time available on a student-by-student basis SAK-3427

Limitations

• Big thing is adding new question types – adding calculated/algorithmic questions
• Currently, to share question pools between users requires import/export - can't do it staying within the authoring tool itself
• Copying question pools doesn't actually copy questions, it creates links SAK-3526
• During authoring the list of question-types available is not context-sensitive SAK-1379 – this could be a feature or a limitation depending on POV.

Delivery

• Performance is the most fundamental issue here, both in terms of speed and reliability
  • Most likely we will need some kind of load testing procedure in place to ensure that we don't repeat the problems of MyUCDavis, currently it doesn't seem like anybody is developing this
• The ability to administer high-stakes quizzes through Samigo is not yet supported in a realistic way (see Limitations below)

Limitations

• Timing of feedback may need to become more flexible SAK-4005
• Help pages are not yet context sensitive SAK-4103
• Need a high-stakes question type SAK-1749
  • Some clarification is needed about the distinction between a linear assessment and a high-stakes quiz SAK-3211
  • There are various security issues that may be prerequisites to high-stakes quizzes SAK-1905
• Generally, it doesn't seem like the test-taking is bulletproof enough yet, clicking too fast on buttons can cause problems, etc. SAK-6553 and SAK-6692

Reports

Minimal functionality at the moment.

• For example: School of Medicine wants to have "how long did student spend on each question"?

Importing/Exporting

Obviously, being able to bring existing quizzes and tests into SmartSite is crucial for all the current users of MyUCDavis Quizbuilder.

Currently, you can only import and export assessments to and from Samigo and MyUCDavis materials – even though it's QTI compliant, it's specific to Samigo. You can import assessments now from MyUCDavis to Samigo. You can also export/import assessments between sites within SmartSite.

One way: Samigo -> UCDavis.

Limitations:

1. Can't export question pools yet, except with workaround – create a quiz that pulls a random selection from a pool, when export it exports all questions. Bug fix may make that no longer work.
2. Calculated questions can't be imported from MyUCDavis
3. Also, some things lost - documented in this gap analysis
   • e.g. Matching questions in other systems have distractors may need a link/description
   • MyUCDavis allows you

• There seems to be an upper limit on the number of questions you can have in a question pool SAK-61 – this may be an authoring limitation as well

Some things that are ahead in the import/export space:

On the horizon:
First thing: import question pools, which will take exported quizzes from Samigo. Key thing is that you can export a quiz and hand it to somebody else (share with other faculty), import to pool. Main thing, initially.
Second thing: export question pools and collections from MyUCDavis (new feature for MyUCDavis).

Futuristic:
In design phase: importing Respondus quizzes or question pools – the advantage of R. is that it's a nice authoring environment - desktop app, less than $100 a copy.

QTI: Interoperability standard. Predecessor to Common Cartridge. Distinct from Zach's effort to implement CC. Part of Sakai's committment to IMS standards for compatibility.

Consider expanding QTI import/export to look at IMS Content Packaging – includes capability to import/export media with a quiz, possibly importing/exporting subpools. Still gathering use cases from the community for this.

Common Cartridge: Still unclear when this will be incorporated into Sakai Foundation core. Prob. sign. work to draw into our world. But, it could make it possible to bring in various outside data...
Outside tool from Sakai Foundation, does migration via SiteInfo, long-term effort to bring in textbook publisher data, etc.