Project Summary

The purpose of this project is to create a tool to allow faculty to migrate their existing question pool content from the MyUCDavis course management system (cms) to the Question Pool feature of Tests & Quizzes (Samigo) in the UC Davis implementation of Sakai.

Question pool migration entails several different functions which may be addressed as separate projects:

These functions are being approached separately for several reasons:

Samigo import functionality as of 8/11/06 is limited to single xml file using QTI 1.2 standards
Samigo does not yet support importing media via QTI content packaging

Success of this project will be measured by the following criteria:

proven ability to export a faculty's entire question pool as a single, Samigo-compliant assessment xml file
written notification of export exceptions, i.e., calculated questions will not be exported
failure to export a question pool successfully will elicit an clear and concise error message
export of media in a zip file and documentation associating media with specific questions
proven ability to import the MyUCDavis-generated xml file into Samigo as a question pool
failure to import successfully will elicit an error message
proven ability to import media files into Resources

Background

IET intends to move faculty and staff using the MyUCDavis cms to using a UC Davis Sakai implementation by Fall 2007. MyUCDavis is a custom-coded, locally-supported ColdFusion application, while Sakai is an open source, collaborative Java application.

To encourage faculty to adopt the new cms, it is imperative that we enable course content migration for both faculty and support staff. Migration must be a smooth and opaque process for faculty. It is also desirable that the migration be consistent with existing functionality in Sakai and be well documented for ease of use by faculty and staff.

Out of Project Scope

The following issues are out of scope of this project:

migration of media files within a question pool (media files are migrated separately)
insertion of media files into questions after migration
export of question pools in Samigo
migration of MyUCDavis content outside of Quiz Builder
migration of MyUCDavis quizzes (completed February 2006)

Analysis

Analysis of question pool data was completed to determine the scope and character of question pool content. Previous analysis for migrating MyUCDavis quiz content indicates that some questions/question types will not be compatible with the current Sakai environment.

MyUCDavis Question Pool Usage Analysis

As of 8/11/06 there are 10,423 questions stored in the MyUCDavis equivalent of question pools. Some of the content will not be migrated because the original owners are no longer valid Kerberos/UCD personnel; other questions are not valid, that is, were incompletely formed. A portion of these questions are calculated question type which has no equivalent in Samigo.

The question pool numbers break down as follows:

Total number of questions in the database: 10,423
Total number of valid questions with valid owners: 9514
Number of calculated question types: 238
Total number of valid questions, not calculated type, with valid owners: 9297
Total number of owners: 335
Total number of valid owners: 281
Question count/valid owners breakdown:
- 1 - largest question count: Harold Davis, VetMed - 1181 questions
- 1 - second largest question count: Beth Post, Psych - 638 questions
- 4 - instructors in 300 - 400 count range
- 19 - instructors in 100 - 299 count range
- 21 - instructors in 50 - 99 count range
- 34 - instructors in 25 - 49 count range
- 27 - instructors in 11 - 24 count range
- 174 - instructors in 1 - 10 count range

Question pool owners tend to be clustered in particular departments where one or more faculty are active cms users. Some of the valid owners are IET personnel who use the questions for testing. A detailed breakdown of question pools by question count, department or last name is documented in Excel.

A percentage of the questions in each users pool may be garbage, since questions cannot be deleted from the pool once they are created. This means that users who create 'test' questions when learning to use MyUCDavis cannot get rid of those questions.

MyUCDavis Collection Usage Analysis

Some instructors have organized their questions into collections, an optional Quiz Builder tool for grouping and sequencing questions. Collections or individual questions may be added to quizzes.

Breakdown of collection data as of 8/14/06:

Number of valid owners: 60
Number of collections owned by valid users: 206
Number of valid collections containing questions: 197
Number of distinct questions contained in valid collections: 1777
Breakdown of question count in the 197 valid collections:
- 3 largest collections contain a respective question count of: 72, 84, 165
- the three largest collections are owned by professors D. Holoman and Anna Song
- 10 collections contain between 21-43 questions
- 31 collections contain between 11-20 questions
- 63 collections contain between 6-10 questions
- 90 collections contain between 1-5 questions

Quiz Builder / Samigo Question Pool Gap Analysis

In MyUCDavis Quiz Builder (QB), questions are stored in a single question pool owned by a single individual. Questions from the pool can optionally be stored in a collection and pulled into a quiz either individually or as a collection. Questions are stored in a single 'pool' by default and in collections by choice.
Once quizzes are published, the status of a quiz question changes - the question can no longer be modified/edited, although a copy of the question can be generated. Questions can never be deleted. All questions within this structure are contained in a single table and their editable status is based on whether they exist within a 'published' quiz.

In Samigo, questions are stored in relationship to a quiz, a question pool, or both. A question which exists solely in a quiz or a question pool may be 'removed' and is deleted from the database. When a question created in a quiz is stored in a question pool, the original question is shared, so editing it in either the quiz or the pool will reflect the changes in both. If a question is created in a quesion pool, a duplicate is created when it is copied to a quiz, but the original question is used if the quiz generates a random draw.
Once a quiz is published, a duplicate of the question is stored in another table. Duplicates are generated if the quiz does a random drawing from a pool as well. There is some evidence that a random draw quiz with unlimited tries may pull the same questions for an individual as were pulled during his/her firest draw. Published questions can not be edited and modifying the original question has no effect on the duplicate.

Please note that this gap analysis was done against Sakai version 2.2. Some of these features may have changed since earlier versions of Samigo.

Question Pool Function	MyUCDavis	Samigo	Funtion Loss / Gain
create a question	Questions are created in the pool and may then be added to a quiz or a collection	Questions may be created either in a pool or in a quiz. A question is shared from a quiz to a pool, but it is duplicated from a pool to a quiz.	Samigo questions are primarily associated by their relationship to a quiz or a pool. Creation is more flexible and can be shared from either creation point.
edit a question	Modifications to a question in the pool are displayed in the quiz the question appears in. The exception is that a question cannot be modified if it is on a quiz that has been taken.	A quiz question shared to a pool shares modifications; a pool question copied to a quiz does not share modifications. Published quizzes use copies of questions, not the original.	User should be aware of the question origination point depending on whether they want a copy or the original.
delete a question	Questions can be removed from a quiz, but cannot be deleted from the pool	Questions can be deleted from a quiz or a pool; once shared, the question must be deleted in both locations if complete removal is desired.	Deletion is a feature gained. Users should be aware of deletion specifics based on question origin; the program requires confirmation on every deletion request.
question pool hierarchy	QB consists of a single pool; collections offer a single subpool level of organization.	Samigo allows for multiple pools/subpools in unlimited number of levels.	Unlimited subpool design may change due to issues with performance and after usage studies. Even if subpool hierarchy is limited in the future, this is a gain.
pool/collection features	Questions may be sequentially stored in collections.	Multiple and multiple-level subpools are allowed.	Even if subpool hierarchy is limited in the future, this is a gain.
question/pool ownership	Questions, pools and collections are owned by a single individual.	Questions, pools and subpools are owned by a single individual.	No gain or loss.
sharing questions, pools	Question pools/collections cannot be shared through the UI, but individual questions or quizzes can be shared by 'publishing'. This function allows the user to select to publish to all users, to all courses within a subject, to a single course, or to a specific person. You can not edit a question you did not create, but you can make your own copy. Shared quizzes are always copies, not the original.	Quizzes may be shared with anyone who has 'maintain' access to your course, i.e., the ability to author quizzes. Questions pools and sharing quizzes with users who do not have authoring access is only available via import/export functions.	While the current 'sharing' model is limited in Samigo, we are currently interviewing within the community to design various access levels for sharing among faculty and other support personnel.Developing question pool import and export functions will also allow question pool sharing.
question types	QB has the following types: multiple choice, fill in the blank (ordered and unordered), matching, essay, and calculated	Samigo has the following importable types: multiple choice, survey, fill in the blank, matching, true/false, and essay.	All question types except for calculated questions are importable from QB to Samigo.

Export/Import Media Files

There is a separate project to export quiz media files from MyUCDavis into a single zip file. Development was completed in July and testing will be performed concurrent with the export question pool function.

Media files are exported as standalone files contained in a single zip file. In the export quiz function, the question each file is associated with is documented on the export screen. In MyUCDavis, media are associated with a question, while in Samigo, media are embedded. Because of this difference and because there is no successful import media within a process, adding media to a quiz (or question pool) in Samigo will require a separate manual process.

The suggested process is to store the media files in a Resources folder and to add the url link to each question individually. This process may change once QTI standards for content packaging import/export are applied to Samigo.

Sakai Import Pool Function

Currently, in Sakai version 2.1.x and 2.2, there is no import/export feature for question pools. However, import a question pool is in development and scheduled for release to production by mid-September. The ability to export a question pool will follow during Fall 2006.

The Import Pool function in Samigo will follow the design of Import Quiz. The Import feature will be available from the main Question Pool screen. In order to import successfully, the question pool must be an xml file formatted to Samigo-specific QTI v1.2 standards.

Initial import testing (August 9-11) of Samigo-compliant xml files has been successful for question pools with 60 and 240 multiple choice questions in Sakai v2.2. Additional testing for all question types and for various size pools is required prior to release to the community.

Authoring performance issues when the user has multiple question pools and subpools were reported for local Davis Sakai version 2.1.x. The issues was identified with generating the question pool tree for display. This suggests that importing a number of large pools may cause difficulty in authoring. In addition, not much is known about the affect of importing large (+300 question) pools or the performance of pool management, such as copying, moving or adding subpools. It is unkown at this time whether the authoring issue will affect importing or if the issue persists in version 2.2.

Details and graphics for import question pool functionality are found under Samigo-Davis Projects as Import question pools

Analysis Conclusions

Analysis of current MyUCDavis data indicates that there are 281 potential faculty who would desire the ability to export their question pools. At least 60 of these faculty also have some questions organized in collections for ease of management.

Important numbers to consider:

281 question pool owners of 9650 total questions
98% of owners manage question pools containing 1-299 questions
60 owners of 197 collections
98% of collections contain 1-43 questions
2% of questions are non-importable calculated questions

The six instructors with question pools of more than 300 questions may require assistance to successfully import and manage their question pool migration to Samigo. Those instructors with collections may find migration easier if there is an export function for collections, since they will be able to organise their content into subpools immediately after import.

The gap analysis concludes that most features will be neutral or show a gain as instructors move from using Quiz Builder to using Samigo.
The positives of question pool migration include:

the ability to delete a question allowing for better question pool management
more flexible, multi-level question pool design
future potential for sharing question pools via an export/import or access management process
all question types, except for the 2% calculated types, will migrate
media files will migrate to Samigo - although a manual process is required to add media to questions

Design

The question pool and collection export processes will be made available for each individual with access to MyUCDavis cms and content in his/her question pool. The export function will check that questions or collections are available, respectively, before it is available on the screen.

The migration path will exist of a function in Quiz Builder to download/export a single xml and/or a zip file containing question media. The process will mirror the existing Export Quiz function by opening up a new window with export information and a description of how to export from Quiz Builder and import into Samigo. Faculty and/or staff can then import the xml file in Samigo's pending Import Pool function or copy the zip file media to their Sakai course website via WebDav.

Desired features of a migration tool are:

Faculty, staff and TAs can export and import questions by question pools or collections
The process closely models existing processes, providing for maximum ease of use.

Proposed Question Pool/Collection Export/Import Process

The user chooses the Quiz Builder link on their MyUCDavis cms navigation.
Clicks on either the Edit/Browse Questions or Edit/Browse Collections from the Quiz Builder sub-menu.
The link Export All Questions or Export Collection appears on the respective screen if questions and collections exist.
After selecting either link, a new window will open with the following information:
- the title 'Export All Questions as Question Pool' or the name of the collection at the top of the screen
- export/import instructions for migration from MyUCDavis to Samigo
- single button labeled 'Download' for downloading the question pool/collection as an xml file
- question pool/collection export notes detailing the differences between question types in the two systems
- question pool/collection export notes listing media files available for download via a 'Download Zip File' button
After selecting the 'Download' or 'Download Zip File' button, the user is prompted to choose a download location for the file.
The user saves the xml or zip file to their local machine.
The user opens Sakai, and then Tests & Quizzes, if importing a question pool or collection xml file, and then:
- user selects Import on Question Pools screen after selecting Question Pools
- user selects Browse to select the xml file from their local system
- user selects Import to import the question pool
The user opens Sakai, and then Resources, if importing a media zip file, and then:
- user selects Upload-Download Multiple Resources
- user follows instructions for using WebDav
- user unzips his/her media file to export the files into Resources
- user may then manually add each image to each question as desired

UC Davis Sakai Collaborative

MyUCDavis Quiz Builder Question Pool Migration to Sakai