...
TEST PLAN OVERVIEW
Scope of testing
This document is designed to guide Quality Assurance in the functionality testing of the provider mechanisms within SmartSite. The functionality of the provider mechanisms, as detailed by the given test cases in this plan, are driven by the functional requirements at :http://confluence.ucdavis.edu:8080/confluence/display/UCDSAKAI/Sakai+Providers.
...
| Tip |
|---|
| title | Purpose for this testing |
|---|
|
The purpose of this test plan is to regression test the provider functionality, which in SmartSite includes: UserDirectoryProvider - Support for WebDav authentication (via Kerberos)
- Provide user account information (e.g. first,last,email,type) from outside data source (e.g. feed from Mothra)
GroupProvider - Assign roles to users who log in to SmartSite that correspond to Sakai internal roles (e.g. Student, Instructor) when applicable. These roles need to be queried from an external data source (e.g. Banner mViews), so that authorization can be determined in real-time.
- Define the sites a user belongs to when they log into SmartSite.
CourseManagementProvider - Retrieve dynamic rosters for courses
- Display current instructor(s) course sites and display them as members in the course with appropriate role
- Display Banner course details in the SmartSite UI
- Update roster dynamically for a course given a , including updates within specific time interval. Roster updates from Banner need to be displayed in the SmartSite application within a pre-determined allowable acceptable time period as defined by the user acceptance testing group.
|
KNOWN ISSUES / PREVIOUS BUGS FIXED
...
- Display current instructor(s) course sites and display them as members in the course with appropriate role
- Display site membership changes in a course site, including instructor adds/removals/updates
|
KNOWN ISSUES / PREVIOUS BUGS FIXED
Previously determined bugs have been fixed for this testing, they were identified as the following. They are also summarized in the table of Jiras referenced in this test plan.
- Realms not refreshing frequent enough
- Rosters not accurate with Banner
- Drops not showing in SmartSite UI
- Known (not fixed) issue #1 for Resources tool (e.g. using WebDav), is that when a file is uploaded to the system, metadata is written to the database before the file is stored on the file system. This is a feature of Sakai, which was detected during some resource discussion re:AFS file storage. For this reason, a resource may "seem" to be updated correctly, and listed in the resouce tool however the content cannot be seen. An http 404 error is thrown (No page can be found). This error is NOT fixed for this testing, just a known issue for Case 1a.
- Known (not fixed) issue #2 is that whatever user that creates the sites when logged in is placed as a member in the site. Therefore, if ssbatchadmin created sites then the roster for such sites may appear off by 1.
...
| title | JIRAs that explain features and past bugs fixed, to be tested in this plan |
|---|
...
- Known (not fixed) issue #3 is that the instructor presented in the member list is not presented first. This is a default sorting behavior within SmartSite and will not be fixed.
- Known (not fixed) issue #4 http://jira.ucdavis.edu:8080/jira/browse/SAK-202 is that the instructor presented in the member list may be still listed even if they have been "replaced"
| Info |
|---|
| title | JIRAs that explain features and past bugs fixed, to be tested in this plan |
|---|
|
Summary of the Jiras related to this test plan, which have been fixed:
BATCH TESTING BUGS FIXED:http://jira.ucdavis.edu:8080/jira/browse/SAK-184 (1000 row limit on associations (realms) for sakai internal query)
BATCH TESTING BUGS FIXED:http://jira.ucdavis.edu:8080/jira/browse/SAK-185 (term abbreviation and term description do not match in site information tool)
PREVIOUS PROVIDER TESTING BUGS FIXED:http://jira.ucdavis.edu:8080/jira/browse/SAK-124 (Banner rosters not in sync with SmartSite UI)197 Support and Training web link should be present, going to gateway site on sakai2
PREVIOUS PROVIDER TESTING BUGS FIXED:http://jira.ucdavis.edu:8080/jira/browse/SAK-201 authz group refresh interfering with batching of courses, noticed in provider integration testing
PREVIOUS PROVIDER TESTING BUGS FIXED:http://jira.ucdavis.edu:8080/jira/browse/SAK-125124 (drop student from Banner and rosters not reflected in sync with SmartSite UI)
PREVIOUS PROVIDER TESTING BUGS FIXED:http://jira.ucdavis.edu:8080/jira/browse/SAK-132 (realms not 125 (drop student from Banner and not reflected in SmartSite UI)
PREVIOUS PROVIDER TESTING BUGS FIXED:http://jira.ucdavis.edu:8080/jira/browse/SAK-132 (realms not refreshing fast enough in the providers) |
TEST ENVIRONMENT
INTEGRATION TESTING | SETUP |
Server Type | Location |
|---|
database | bricker.ucdavis.edu with banner-test (external db), and sakai |
...
dev (internal db) |
application |
...
hanley.ucdavis.edu, sakai- |
...
...
...
| hanley.ucdavis.edu:8443/portal |
population of data referenced | 2006 Fall Quarter Banner Test data (external), and clean sakai internal database with exception of necessary login accounts and manual sites created) |
- Note: The application does not need to be stopped for these tests to occur
...
QA TESTING | SETUP |
Server Type | Location |
|---|
database | bricker.ucdavis.edu with banner-test (external db), and sakai test (internal db) |
application | isaac.ucdavis.edu, sakai-test |
web server | url, http://sakai-test.ucdavis.edu |
population of data referenced | 2006 Fall Quarter Banner Test data (external), and clean sakai internal database with exception of necessary login accounts and manual sites created) |
- Note: The application does not need to be stopped for these tests to occur
- Additional Setup Needed
**QA admin account (e.g. QAadmin)
**All login accounts that use kerberos names can be "provided" by the system (e.g. UserDirectoryProvider implemented).
**SU tool and Quartz tools are available in admin workspace/realm
**Testers have access to testable sites (via SU tool, admin workspace, as user QAAdmin, etc.)
**Run CourseManagement tests on existing batched sites perhaps? or run CourseManagementTests on new Fall Quarter Courses batched? - mView refresh times must be recorded
- Additional External-Banner Test Data Used for Provider Testing
A copy of test data that may be used for these test cases is attached: http://confluence.ucdavis.edu:8080/confluence/download/attachments/17096/200603TestData.xls
DEPENDENCIES
- All UCDavis Providers (UserDirectoryProvider, GroupProvider, and CourseManagementProvider) will be enabled to allow user accounts to be An example of a white list file to be used for course creation is: http://confluence.ucdavis.edu:8080/confluence/download/attachments/17226/white-list.txt
DEPENDENCIES
- All UCDavis Providers (UserDirectoryProvider, GroupProvider, and CourseManagementProvider) will be enabled to allow user accounts to be looked up automatically via kerberos name
- Distributed authentication using AFS should be enabled (SAK-143)
- Sakai-test (Sakai-dev for integration testing) database will be a either a clean copy or a copy of existing production data, so sakai-test will have to be backed up prior to testing,
We Pradhu or Geeta will be referencing contacted to schedule this. - We will be referencing sakai-test (banner test external, mViews..) for populating course data
...
| Info |
|---|
*NOTE: The order of the test cases will be as follows. Setup Preliminary Case (0a): Accessing server
Setup Preliminary Case (0b): Login access via DistAuth
Setup Preliminary Case (0c): Login outside of DistAuth (e.g. internal account)
Setup Preliminary Case (0d): Accessing admin functionality Case 1: User Directory Provider
1a) Web Dav authentication
1b) User directory lookup functionality Case 2: Group Provider
2a) Roles in courses verified by tabs
(1) Roles in courses verified by system log output (system admins)
2b) Realm refresh via browser refresh button (1) Realm refresh seen in system log output (system admins) Case 3: Course Management Provider
3a) Adds/drops/updates from Banner seen in SmartSite UI
3b) Rosters from Banner viewed by instructor and administrator role in site
3c) Course Site information viewed by instructor seen in site summary details
3d3b) Adds/drops of students from Banner seen in SmartSite UI
3c) Roster details accurate (e.g. kname, role, etc)
3d) Instructor changes for a course seen in participants list for a site |
TEST CASES
| Info |
|---|
| title | Key for Test Case Scenarios |
|---|
|
Description = description of the test performed
Category of Testing = the described testing is at what level?
Prerequisite Steps = actions needed before performing any of the given tests |
...
| Note |
|---|
| title | Setup Test Case Scenario 0d, Logging into test server via SmartSite's internal admin account, seeing necessary tools |
|---|
|
Description = Test the capability of the user to log into the test server using a valid SmartSite admin account and see SU Tool, and other admin functionality Category of Testing = functionality,regression Prerequisite Steps
User has valid admin account name, password known Test Scenario 0d - go to http://sakai-test.ucdavis.edu/portal/xlogin in your browser
- user will enter username and password given
- User will use admin account name
- View site information for a given site (e.g. see Sites link in Admin workspace)
- User will logout of admin account
- User will use QAAdmin QAadmin account name
- Become a user using SU Tool
- Logout of using SU Tool
Test Cleanup
None |
...
Test | User | User Acting As | Result | Expected Result | Pass/Fail |
|---|
Login to SmartSite, SmartSite admin account | Pat Kava | admin | logged in, user in "Users Present" | logged in, name in "Users Present" | P |
Login to SmartSite, SmartSite admin account | Pat Kava | admin | see admin workspace | see admin workspace | P |
Login to SmartSite, SmartSite admin account | Pat Kava | admin | see Sites link | Sites link accessable | P |
Logout of SmartSite, using logout link | Pat Kava | admin | logout happens | logout happens | P |
Login to SmartSite, using QAAdmin QAadmin account | Pat Kava | QAAdminQAadmin | logged in, user in "Users Present" | logged in, name in "Users Present" | P |
See SU page link | Pat Kava | QAAdminQAadmin | SU page/tool available | SU page/tool accessable | P |
SU as some instructor using SU Tool | Pat Kava | QAAdminQAadmin | SU by "View User Info" first, then "Become User" | SU happens correctly, become user | P |
Logout of SmartSite, using logout link | Pat Kava | SU'd person | logout happens | logout happens | P |
...
| Note |
|---|
| title | WebDav Testing, Case 1a |
|---|
|
Description = Test the capability of the UserDirectoryProvider to provide Kerberos authentication for WebDav Category of Testing = functionality,regression Prerequisite Steps - One file must be uploaded via the Resources tool prior to using WebDav
Test Scenario 1a, WebDav - Log in as any user having access to any site
- Follow WebDav instructions based on instructions in Resource area
- * Use "Manage Multiple Resources" link in Resources tool to access WebDav
- Data to Be Used:
QA determines
Test Cleanup
None |
Test Results, Case 1a WebDav
Would yield an example Summary Test Result of:
...
results: Recorded in Jira http://jira.ucdavis.edu:8080/jira/browse/SAK-154
| Note |
|---|
| title | WebDav User Lookup Testing, Case 1b |
|---|
|
Description = Test the capability of the UserDirectoryProvider to provide information about an externally provided user via the SU Tool Category of Testing = functionality,regression Prerequisite Steps - SU tool available to tester
Test Scenario 1b, User lookup functionality - Log in as any user having administrative access (access to admin realm)
- Use a given kerberos name available via test data (e.g. instructor)
- Enter kerberos name, and click "View User Info"
- Verify user information from whois.ucdavis.edu and/or email.ucdavis.edu
- Data to Be Used:
QA determines
Test Cleanup
None |
...
| Note |
|---|
| title | Case 2a, Group Provider Tabs test |
|---|
|
Description = Test the capability of the GroupProvider to provide tabs for a user based on their enrollment/membership in sites Category of Testing = functionality,regression Prerequisite Steps - Site user is a member of must be already created prior to test (via Batch? via manually?)
- User must be a member of the sites in order to see the tab
- If user is an externally provided one, then they must be in either the external database (Banner) or already in the internal SmartSite database
Test Scenario 2a, Tabs - Log in as in Case 0d, but become the user and don't log out
- Log in instructor for a given site site (see test data attachmentin CourseManagementProvider test section)
- View tabs presented for sites available, and only those that you are a member of and have been created will appear
- Stay logged in for case 2b
-Additional step for programmers (Case 2ai), verify that data presented in tabs matches server log data- - Data to Be Used:
QA determines
Test Cleanup
None |
...
| Note |
|---|
| title | Case 2b, Realms refresh test |
|---|
|
Description = Test the capability of the AuthxGroups to be updated more frequently than default OOTB Category of Testing = functionality,regression Prerequisite Steps - Site user is a member of must be already created prior to test (via Batch? via manually?)
- User must be a member of the sites in order to see the tab
- If user is an externally provided one, then they must be in either the external database (Banner) or already in the internal SmartSite database
Test Scenario 2b, Realms refreshing - Staying as user from Case 2a, record the site tab that is highlighted
- Brian Donnelly or Scott Amerson will add a user to this given site, through the external db (by adding a user this way, the 15 minute refresh doesn't have to occur to see the end results)
- Scott will notify Pat Kava of what user has been added to the course externally
- Pat will verify that by hitting the browser refresh after 5 minutes elapsed since update, that the user added appears as a member in the site
-Additional step for programmers (Case 2bi), verify that authz refresh is happening via the log output, and at what intervals- - Data to Be Used:
QA determines
Test Cleanup
None |
...
Possible Test Data
| Note |
|---|
| Info |
|---|
| title | Case 3a(1), Roster add |
|---|
|
Test Scenario 3a(1), Roster Adds| title | Case 3a, Roster Update Tests (add(1), update(2), delete(3)) |
|---|
| Description = Test the capability of the CourseMangementProvider to accurately reflect within the SmartSite UI, students added to a Banner roster alotted time period
Category of Testing = functionality,regression
Prerequisite Steps
| Possible data set to use for CourseManagementProvider Tests, carried over from Batch Testing |
| CRN Type | Term code - crn | enrollment | instructor kname | Course info | Term |
|---|
CRN with no enrollment | 2006-06-60245 | 0 | fzduts (Dennis Dutschke) | ITA 198 S01 Dir Group Study | Summer Special Session 2006 | CRN with large enrollment number | 2006-01-60001 | 84 (all valid knames) | ewlarsen (Eric Larsen) | LDA, 190 , 001, ProSeminar | Winter Quarter 2006 | CRN with enrollment | 2006-01-40018 | 14 (all valid knames) | calymos (Calvin Lymos) | AAS, 154, 001, University Gospel Choir | Winter Quarter 2006 | CRN with enrollment | 2006-03-82882 | 10 | ltcmwc (Mark Connelly) | MSC 141 002 Army Mngmt Systems | Spring Quarter 2006 | CRN with enrollment | 2006-03-83634 | 10 (all valid knames) | eschang (Ernest Chang) | NPB 141P 001 Res In Physio Adapt | Spring Quarter 2006 |
|
| Note |
|---|
| title | Additional data set to use for CourseManagementProvider Tests, using Fall Quarter 2006 data |
|---|
|
CRN Type | Term code - crn | enrollment | instructor kname | Course info | Term |
|---|
CRN with enrollment | 2006-10-13185 | 7 | jordi, (Donald Palmer) | ARE,200A,001,Microeconomic Theory | Fall Quarter 2006 | CRN with enrollment, no inst | 2006-10-18933 | 29 | staff | EAP,350,001,France | Fall Quarter 2006 | CRN with enrollment, no inst | 2006-10-18954 | 45 | staff | EAP,862,001,UK/Ireland/Wales | Fall Quarter 2006 | CRN with enrollment | 2006-10-26423 | 5 | osleger | GEL,025,001,Geology National Parks | Fall Quarter 2006 | CRN with enrollment | 2006-10-28142 | 8 | bossler | HIS,204,003,Historiography | Fall Quarter 2006 | CRN with enrollment, no inst | 2006-10-30220 | 6 | staff | MAT,012,002,Precalculus | Fall Quarter 2006 | CRN with enrollment | 2006-10-32457 | 5 | palmer, (Joaquim Silvestre) | MGP,223,001,Power and Influence | Fall Quarter 2006 | CRN with enrollment | 2006-10-32454 | 2 | hemantb, (Hemant Bhargava) | MGP,207,001,Mgmt Info Systems | Fall Quarter 2006 | CRN with enrollment, no inst, no enrollment | 2006-10-21225 | 0 | staff | ECS,298,017,Group Study | Fall Quarter 2006 | CRN with enrollment, no inst, no enrollment | 2006-10-40185 | 0 | staff | SAS,097T,000,Tutoring Sci & Society | Fall Quarter 2006 |
|
Case 3a, Roster View Tests
| Note |
|---|
| title | Case 3a, Roster View Test |
|---|
|
Description = Test the capability of the CourseMangementProvider to accurately reflect within the SmartSite UI, the roster from a site administrator perspective Category of Testing = functionality,regression Prerequisite Steps
Current roster given to QA for respective courses to be tested | Info |
|---|
| title | Case 3a, Roster view, admin or instructor |
|---|
| Test Scenario, Case 3a - SU as instructor as given by Case 0d, and for this case use kerberos name palmer.
Data Used CRN Type | Term code - crn | enrollment | instructor kname | Course info | Term |
|---|
CRN with enrollment | 2006-10-32457 | 5 | palmer, (Joaquim Silvestre) | MGP,223,001,Power and Influence | Fall Quarter 2006 |
- Verify roster is same as given
- Do same as above for the admin user. Note, if the admin user is not a member of the site, then you will not see a tab. You will need to go to "WorkSite Setup" and click "Revise" to see roster information.
|
|
Case 3b, Roster Update Tests
| Note |
|---|
| title | Case 3b, Roster Update Tests (student add(1), student drop(2)) |
|---|
|
Description = Test the capability of the CourseMangementProvider to accurately reflect within the SmartSite UI, students added to a Banner roster within an alotted time period Category of Testing = functionality,regression Prerequisite Steps
*Current roster given or available to QA for respective courses to be tested
*Courses with test CRN's are created either by batch list or full batch mode | Info |
|---|
| title | Case 3b(1), Roster add |
|---|
| Test Scenario 3b(1), Roster Add - Coordinator will talk to Libby about adding:
student pidm (2204701), kerberos name kdas to CRN (2006-10-)28142
student pidm (163004),kerberos name larivers to CRN (2006-10-)28142
student pidm (163004),kerberos name larivers to CRN (2006-10-)26423 - Record times when updates start, and when they are finished
- Wait until external db refresh happens, based on recorded time(s)
- Coordinator will advise testers when update has occured
- Log in as instructor for one of these courses, and click Site Info to view updated roster adds above
|
Description = Test the capability of the CourseMangementProvider to accurately reflect within the SmartSite UI, students dropped from a Banner roster within an alotted time period Category of Testing = functionality,regression Prerequisite Steps
Current roster given to QA for respective courses to be tested | Info |
|---|
| title | Case 3b(2), Roster drop |
|---|
| Test Scenario 3b(2), Roster Delete - Coordinator will talk to Libby about adding:
student pidm (1107630), kerberos name dclu dropped from CRN (2006-10-)13185
student pidm (2218507), kerberos name djf27 dropped from CRN (2006-10-)32457 - Record times when updates start, and when they are finished
- Wait until external db refresh happens, based on recorded time(s)
- Coordinator will advise testers when update has occured
- Log in as instructor for one of these courses, and click Site Info to view updated roster deletes above
- Log in as kerberos names above, see if any 2006 Fall tabs show up
|
- Data to Be Used:
QA determines if there are additional updates from list of available sites above.
Test Cleanup
None |
Test Results Case 3b(1),3b(2), Roster update functionality
Would yield an example Summary Test Result of:
Test | User | User Acting As | Result | Expected Result | Pass/Fail |
|---|
Roster add check | Pat Kava | x | roster added with Student xxxxx | Roster added with Student kerb name | P |
Roster drop check | Pat Kava | x | roster changed with Student xxxxx removed | Roster changed accordingly with kerb name xxxx removed | P |
results: Recorded in Jira http://jira.ucdavis.edu:8080/jira/browse/SAK-124
Case 3c, Roster Accuracy Tests
| Note |
|---|
| title | Case 3c, Roster Accuracy Tests |
|---|
|
Description = Test the capability of the CourseMangementProvider to accurately reflect within the SmartSite UI, members in a Roster (site member) list Category of Testing = functionality,regression Prerequisite Steps
*Current roster given or available to QA for respective courses to be tested
*Courses with test CRN's are created either by batch list or full batch mode | Info |
|---|
| title | Case 3c, Roster accuracy check |
|---|
| Test Scenario 3c, Roster accuracy - Find Fall Quarter CRN available from the list above, one with roster
- Log in as instructor for one of these courses, see roster appears
- Verify roster with that of MyUCDavis website
|
- Data to Be Used:
QA determines if there are additional updates from list of available sites above.
Test Cleanup
None |
Test Results Case 3c Roster check
Would yield an example Summary Test Result of:
Test | User | User Acting As | Result | Expected Result | Expected Result | Expected Result | Expected Result | Pass/Fail |
|---|
Roster check | Pat Kava | instructor x | site has accurate number of members | site has accurate instructor kerb name, user name | site has accurate students kerb name, user name | Format correct for member list | Member list formatted correctly | P |
results: Recorded in Jira http://jira.ucdavis.edu:8080/jira/browse/SAK-124
Case 3d, Instructor Change Tests
| Note |
|---|
| title | Case 3d, Instructor Change Tests (instructor add(1), instructor remove(2), instructor change from staff(3)) |
|---|
|
Description = Test the capability of the CourseMangementProvider to accurately reflect within the SmartSite UI, instructors added to a Banner course within an alotted time period Category of Testing = functionality,regression Prerequisite Steps
*Current roster given or available to QA for respective courses to be tested
*Courses with test CRN's are created either by batch list or full batch mode | Info |
|---|
| title | Case 3d(1), Instructor add |
|---|
| Test Scenario 3d(1), Instructor Add - Coordinator will talk to Libby about adding:
instructor uid (612026), kerberos name jordi to CRN (2006-10-)10013 (mfharris is previous instructor)
instructor uid (610902),kerberos name palmer to CRN (2006-10-)10033 (oluwa is previous instructor) - Record times when updates start, and when they are finished
- Wait until external db refresh happens, based on recorded time(s)
- Coordinator will advise testers when update has occured
- Log in as the respective instructor for one of these courses, and click Site Info to view updated member list (instructor) above
- Record other instructor represented in course, given by [] above
|
Description = Test the capability of the CourseMangementProvider to accurately reflect within the SmartSite UI, instructors removed from a Banner course within an alotted time period Category of Testing = functionality,regression Prerequisite Steps
Current roster given to QA for respective courses to be tested | Info |
|---|
| title | Case 3d(2), Instructor remove |
|---|
| Test Scenario 3d(2), Instructor Delete - Coordinator will talk to Libby about removing:
kerberos name mfharris (148041) from CRN 10025 (mfharris was only instructor, now should be Staff)
kerberos name fzturner (610916) from CRN 10024 (fzturner was only instructor, now should be Staff)
- Record times when updates start, and when they are finished
- Wait until external db refresh happens, based on recorded time(s)
- Coordinator will advise testers when update has occured
- Log in as instructor for one of these courses, see if site tabs for CRNs appear
- Log in as admin, see if instructors appear for courses
|
Description = Test the capability of the CourseMangementProvider to accurately reflect within the SmartSite UI, staff removed from a Banner course, and replaced with another instructor within an alotted time period Category of Testing = functionality,regression Prerequisite Steps
Current instructors given to QA for respective courses to be tested | Info |
|---|
| title | Case 3d(3), Instructor added in place of staff |
|---|
| Test Scenario 3d(3), Instructor added, Staff Removed - Coordinator will talk to Libby about adding:
kerberos name mfharris (148041) to CRN 10037 (mfharris should be only instructor, replacing Staff)
- Record times when updates start, and when they are finished
- Wait until external db refresh happens, based on recorded time(s)
- Coordinator will advise testers when update has occured
- Log in as instructor mfharris for one of these courses, see if site tabs for CRNs appear
- Log in as admin, see if instructor appears for course
|
- Data to Be Used:
QA determines if there are additional updates from list of available sites above.
Test Cleanup
None |
Test Results Case 3a3d(1),3d(2), Roster Add 3d(3) Instructor update functionality
Would yield an example Summary Test Result of:
...
Test
...
User
...
User Acting As
...
Result
...
Expected Result
...
Pass/Fail
...
Test Result of:
Test | User | User Acting As | Result | Expected Result | Pass/Fail |
|---|
Instructor add check | Pat Kava | x | CRN xxxx added with instructor xxxxx | Instructor added to list | P |
Instructor remove check | Pat Kava | x | CRN changed with instructor xxxxx removed | Member list changed with kerb name xxxx removed | P |
Instructor change staff to instructor check | Pat Kava | x | roster added with Student xxxxx | Roster added with Student kerb name changed staff for course 2006-10-10037 to instructor kerb name mfharris | Member list changed with kerb name xxxx removed | P |
results: Recorded in Jira http://jira.ucdavis.edu:8080/jira/browse/SAK-124202
Recording Results
The test results will be recorded in Jira by QA testers. They can be of table form or text form.
...