TEST PLAN DATE
Testing date: Scheduled 9/14/06, early morning
TEST PLAN OVERVIEW
Scope of testing
This testing is designed to guide Quality Assurance in the functionality testing of the batch creation mechanism of course sites in SmartSite. The functionality of the batching mechanism, which can be run at any time by an application administrator, is detailed by the given test cases in this plan. Additionally, the level of involvement by QA, system administrators, and programmers will be detailed for each test case.
...
- Related to the Home Page web content displayed for course sites. Currently, the Course Description information displays course title, which will be changed to description when the external data is mapped to a description field.
- There were two bugs found during unit/integration testing that are presumed fixed.
- The term information on the site tool didn't match the tab abbreviation for the term
- When a user, especially an instructor, logged in (e.g. SU Tool) there is a limit of associations the instructor can have to courses. This limit is 1000, and since has been filtered down to pull back only terms that are <= year 2005.
(These JIRAs (SAK-184, SAK-185) are listed below , in the summary of JIRAs covered in this test plan.)
| Info | |||||
|---|---|---|---|---|---|
| |||||
TESTING BUGS FIXED:http://jira.ucdavis.edu:8080/jira/browse/SAK-141 (Batching is done under a 'batch admin' user185 (1000 row limit on associations (realms) for sakai internal query) other non-batch related JIRAs covered:
|
TEST ENVIRONMENT
...
Server Type
...
Location
...
database
...
...
application
...
|
...
web server
...
url,
...
- Note: The application does not need to be stopped for these tests to occur
...
|
...
...
...
...
- Additional Setup Needed
**Batch admin account (e.g. ssbatchadmin)
**QA admin account (e.g. QAadmin)
**Testers have access to testable sites (via SU tool, admin workspace)
**SU tool and Quartz tools are available in admin workspace/realm
**All login accounts that reference kerberos names can be "provided"
- External-Banner Test Data to be Used As Representative Course Scenarios (see each test case)
- A few test course sites need to be created in advance so that the batch job can modify these sites with a respective provider Id. These sites are documented in the Provider Id Test Section.
DEPENDENCIES
- At least UserDirectoryProvider enabled to allow user accounts to be looked up automatically via kerberos name (Note:all Providers will be enabled)
- Distributed authentication using AFS (SAK-143)
- Sakai-test database will be a copy of existing production data, so sakai-test will have to be backed up prior to testing
- We will be referencing sakai-test external for populating course data
ROLES AND RESPONSIBILITIES
- Testers (QA,students etc)*
- Admins and SmartSite Programmers
- Any of the following persons that may be available:
***Application admins (Programmers available, including Thomas, Lisa W., James, Brian, Jon G., Scott)
***Database/system admins (Geeta, Chris Callahan)
- Any of the following persons that may be available:
*All of the above users will have access to the admin realm in Sakai, in addition to:
***ssbatchadmin
***QAadmin
COURSE SITE BATCHING DOCUMENTATION
- Requirements
The requirements for batching are documented at https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/Course+Batching+Strategy
UNIT AND INTEGRATION TEST RESULTS
Unit Test Results
...
|
| Warning | ||
|---|---|---|
| ||
No Scenario 2b covered here, no Provider id consulting Black list functionality to be tested |
TEST ENVIRONMENT
- Hardware Setup
Server Type
Location
database
bricker.ucdavis.edu with banner-test (external db), and sakai test (internal db)
application
isaac.ucdavis.edu, sakai-test
web server
url,
http://sakai-test.ucdavis.edupopulation of data referenced
2006 Banner Test data (external), and clean sakai internal database with exception of necessary login accounts and manual sites created)
- Note: The application does not need to be stopped for these tests to occur
- Software Setup
run test under https://mware.ucdavis.edu/svn/ucdsakai/branches/sakai-core-2.1.x branch, with mods from https://mware.ucdavis.edu/svn/ucdakai/sakai-core-2.1.x-scottsdev branch incorporated
- Additional Setup Needed
**Batch admin account (e.g. ssbatchadmin)
**QA admin account (e.g. QAadmin)
**All login accounts that use kerberos names can be "provided" by the system (e.g. UserDirectoryProvider implemented).
**SU tool and Quartz tools are available in admin workspace/realm
**Pat Kava will be given access to admin realm
**Testers have access to testable sites (via SU tool, admin workspace)
- External-Banner Test Data to be Used As Representative Course Scenarios (see each test case)
A copy of test data that may be used for these test cases is attached: http://confluence.ucdavis.edu:
...
...
...
Integration Test Results
TEST PLAN PROCESS
This Course Batching Unit/Integration Test Plan will be coordinated by Scott Amerson, and will document the results of unit testing and integration testing for the batching process before handing off to QA. All tests must pass before handing off to QA.
Steps involved in the test process will be varied depending on the given test performed (as described below in the Test Cases section).
| Info |
|---|
*NOTE: The order of the test cases will be as follows.
|
TEST CASES
| Info | ||
|---|---|---|
| ||
Prerequisite Steps = actions needed before performing any of the given tests |
White List
...
| title | White List Testing |
|---|
Description = Test the capability of the system to add "on-demand" course sites that are listed in a file. Courses are not required to have any enrollment in order to be created.
Category of Testing = unit, integration
Prerequisite Steps
- Scott will add a list of CRNs (valid or not) as the white-list.txt file in /var/sakai/ucd-data folder on the server, in the form of TERMYR-TERMCD-CRN for each entry. Use this file file to use here
- Scott will que the batch job to run, only for the above list (5 min elapsed time needed)
- Scott will give maintain permissions on the applicable sites, to the appropriate users
- Brian = look up CRN 1995-01-76288
- Sandra = look up CRN 2006-06-60005
- Scott will assist users in accessing the site(s) if needed
- Thomas = view logs, look for CRN 20061-06-111, 2006-03-60024
- Jon G. = look up CRN 2006-sfs-sfsfdfsdfsfsfs
- Scott will advise programmers that the batch has been run and sites have been created
- Thomas and Scott will monitor server activity while testers are using sakai-dev.
Test Scenario
...
- A few test course sites need to be created in advance so that the batch job can modify these sites with a respective provider Id. These sites are documented in the Provider Id Test Section.
DEPENDENCIES
- At least UserDirectoryProvider enabled to allow user accounts to be looked up automatically via kerberos name (Note:all Providers will be enabled)
- Distributed authentication using AFS (SAK-143)
- Sakai-test database will be a copy of existing production data, so sakai-test will have to be backed up prior to testing
- We will be referencing sakai-test external for populating course data
ROLES AND RESPONSIBILITIES
- Testers (QA,students etc)*
- Admins and SmartSite Programmers
- Any of the following persons that may be available:
***Application admins (Programmers available, including Thomas, Lisa W., James, Brian, Jon G., Scott)
***Database/system admins (Geeta, Chris Callahan)
- Any of the following persons that may be available:
*All of the above users will have access to the admin realm in Sakai, in addition to:
***ssbatchadmin
***QAadmin
COURSE SITE BATCHING DOCUMENTATION
- Requirements
White List/Black List/Provider Ids setting documented requirements are in Jira
Default Tool Set/Pages and Batching Strategy https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/Course+Batching+Strategy
Site Tab abbreviations documented at: https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/Limitations+of+adding+courses+manually+to+SmartSite
General Provider and Batching interaction documented at https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/Requirements+for+Data+Provisioning+using+Providers+and+Batch+Processes
UNIT AND INTEGRATION TEST RESULTS
Unit Test Results
The JUnit tests results are located on the server at:
isaac/var/sakai/ucd-data/course-batch-unit-tests-082006
or..
https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/Course+Batch+Creation+Unit+Test+Results
= All unit tests passed!!
Integration Test Results
https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/Course+
...
...
...
...
...
...
- Data to Be Used:
Scenario
Course
Roster Count
Instructor Kerb Name
Course Details (Catalog Root, Suffix, Section, Description)
Test Case Used In
File Associated
Invalid CRNs
Invalid CRN (Number)
20061-06-111
NA
NA
NA
NA
White List
white-list.txt
Invalid CRN (Number)
2006-03-60024
NA
NA
NA
NA
White List
white-list.txt
Invalid CRN (Bogus Text)
2006-sfs-sfsfdfsdfsfsfs
NA
NA
NA
NA
White List
white-list.txt
*additional,not used for this testing* Invalid CRN (Number) available
1995-03-76288
NA
NA
NA
NA
White List
white-list.txt
Valid CRNs
CRN with no enrollment
1995-01-76288
0
fzthorn, (Robert Thornton)
PLB, 092, 012, Internship
Winter Quarter 1995
White List
white-list.txt
CRN with enrollment
2006-06-60005
11 (all valid knames)
acham (Christine Acham)
AAS, 198, S01, Directed Gp Study
Summer Special Session 2006
White List
white-list.txt
CRN with enrollment
2006-03-93307
10 (all valid knames)
fzvasili (S Spyridakis)
HIS 111C 001 Ancient History
Spring Quarter 2006
White List
white-list.txt,black-list.txt
Test Cleanup
- Scott will remove all sites created in this step when all test cases are completed.
White List Test Results
Would yield an example Summary Test Result of:
Test | User Role | User Acting As | Result | Expected Result | Pass/Fail |
|---|---|---|---|---|---|
Site xxxxx (created): Functionality Results | samerson | fzolupon | All tests pass | batching of course succeeded | P |
Site xxxxx (created): Course Management Test Results | samerson | QA Admin | All tests pass | course content correct | P |
Site xxxxx (created): Provider Id Test Results | samerson | QA Admin | All tests pass | provider id correct, roster number correct | P |
or..
Test | User | User Acting As | Result | Expected Result | Pass/Fail |
|---|---|---|---|---|---|
Site xxxxy (not created):Test Results | samerson | QA Admin | All tests pass, not provisioned | course not created, correct | P |
Black List
...
| title | Black List Testing |
|---|
Description = Test the capability of the system to not add course sites that are listed in a file. Courses in this list will never be batch created.
Category of Testing = unit, integration
Prerequisite Steps
- Scott will add a list of CRNs (valid or not) to the white-list.txt file in /var/sakai/ucd-data folder on the server, in the form of TERMYR-TERMCD-CRN for each entry. This list will be different than that used for the White List Test Scenario. Use this white list
- Scott will add a list of CRNs (valid or not) to the black-list.txt file in /var/sakai/ucd-data folder on the server, in the form of TERMYR-TERMCD-CRN for each entry. Use this black list file file to use here
- Scott will que the batch job to run (White List), (5 min time elapse needed)
- Scott will advise programmers that the batch has been run
- Thomas and Scott will monitor server activity while testers are using sakai-dev.
Prerequisite Steps needed for Scenario #2 only (in addition to Scenario #1 steps)
- Scott will update existing hand created sites manually (TBD site ids) with an associated CRN as the provider id
Test Scenarios
*Scenario #1 = Black list consulted when white list batch run is executed*
- James. = will look for CRN 2004-03-60097 as user admin. This CRN will be listed in the White List files only
- Brian = will look for CRN 2006-01-60001 that is listed both in the white-list.txt and black-list.txt files
- Sandra = will look for both CRN 2006-01-40018 that are listed both in the white-list.txt and black-list.txt files
- Thomas = view logs, internal sakai db, will look for both CRN 2006-03-82882, 2006-03-83634 that are listed in the black-list.txt files only
- Scott = help as needed
- Data to Be Used:
Scenario
Course
Roster Count
Instructor Kerb Name
Course Details (Catalog Root, Suffix, Section, Description)
Test Case Used In
File Associated
Valid CRNs
CRN with no enrollment
2004-03-60097
0
dlmiller (Douglas Miller)
ECN, 298 , 026, Group Study
Spring Quarter 2004
Black List
white-list.txt
CRN with large enrollment number
2006-01-60001
84 (all valid knames)
ewlarsen (Eric Larsen)
LDA, 190 , 001, ProSeminar
Winter Quarter 2006
Black List
white-list.txt,black-list.txt
CRN with enrollment
2006-01-40018
14 (all valid knames)
calymos (Calvin Lymos)
AAS, 154, 001, University Gospel Choir
Winter Quarter 2006
Black List
white-list.txt,black-list.txt
CRN with enrollment
2006-03-82882
10
ltcmwc (Mark Connelly)
MSC 141 002 Army Mngmt Systems
Spring Quarter 2006
Black List
black-list.txt
CRN with enrollment
2006-03-83634
10 (all valid knames)
eschang (Ernest Chang)
NPB 141P 001 Res In Physio Adapt
Spring Quarter 2006
Black List
black-list.txt
*Scenario #2 = Black list consulted when existing site already has provider id that matches black list entry*
...
TEST PLAN PROCESS
This Course QA Test Plan will be coordinated by Pat Kava, with assistance from the programming group (e.g. Scott Amerson), and will document the results of QA testing. It is assumed that all bugs have been documented by the programming group as issues before handing off to QA.
Steps involved in the test process will be varied depending on the given test performed (as described below in the Test Cases section).
| Info |
|---|
*NOTE: The order of the test cases will be as follows.
|
TEST CASES
| Info | ||
|---|---|---|
| ||
Prerequisite Steps = actions needed before performing any of the given tests |
Case 1 #White List
| Note | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Description = Test the capability of the system to add "on-demand" course sites that are listed in a file. Courses are not required to have any enrollment in order to be created. Category of Testing = unit, integration,functionality,regression Prerequisite Steps
Test Scenario
Test Cleanup
|
results: Recorded in Jira https://jira.ucdavis.edu:8444/jira/browse/SAK-119
White List Test Results
Would yield an example Summary Test Result of:
Test | User Role | User Acting As | Result | Expected Result | Pass/Fail |
|---|---|---|---|---|---|
Site xxxxx (created): Functionality Results | samerson | fzolupon | All tests pass | batching of course succeeded | P |
Site xxxxx (created): Course Management Test Results | samerson | fzolupon | All tests pass | course content correct | P |
Site xxxxx (created): Provider Id Test Results | samerson | fzolupon | All tests pass | provider id correct, roster number correct | P |
or..
Test | User | User Acting As | Result | Expected Result | Pass/Fail |
|---|---|---|---|---|---|
Site xxxxy (not created):Test Results | samerson | fzolupon | All tests pass, not provisioned | course not created, correct | P |
Case 2 #Black List
| Note | |||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||
Description = Test the capability of the system to not add course sites that are listed in a file. Courses in this list will never be batch created. Category of Testing = unit, integration, functionality,regression Prerequisite Steps
-Prerequisite Steps needed for Scenario #2b only (in addition to Scenario #2a steps)- Test Scenarios
Test Cleanup
|
Black List Test Results
Would yield an example Summary Test Result of:
Test | User | User Acting As | Result | Expected Result | Pass/Fail |
|---|---|---|---|---|---|
Site xxxxy (not created):Test Results | samerson | QA Admin | All tests pass, not provisioned | course not created, correct | P |
...
|
Case 3 #Update Existing Site with Provider Id
| Note | ||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||||||||||||||||||
Description = Test the capability of the system to update existing course sites with associated provider ids. This functionality can be done either in the UI or batch, however we want to Category of Testing = unit, integration, functionality,regression Prerequisite Steps
*Data To Be Used:
Test Scenario
Manual Process Check (update provider ids for sites manually):
Test Cleanup
|
Provider Id Update Test Results
Would yield an example Summary Test Result of (both batch and manual updates):
Test | User | User Acting As | Result | Expected Result | Pass/Fail |
|---|---|---|---|---|---|
Site xxxxx: Provider Id Update Results | samerson | admin | provider id is ....., roster is same as CRN for provider id | provider id updated and seen in site tool under "provider id", roster reflects provider id roster | P |
Site Functionality
| Note | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||
Description = Test the capability of the system to add sites with the correct pages, tools, tabs, and look and feel. Category of Testing = unit, integration, functionality Prerequisite Steps
Test Cleanup
|
Course Management Data Presented in Admin Tools
| Note | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
Description = Test the course management content displayed in admin tools. Category of Testing = unit, integration, functionality Prerequisite StepsResults Recording Steps summarized in: **perform course management test
Scenario 3a (Batch updates existing sites provider id)
Scenario 3b (Manual Process Check (update provider ids for sites manually) QA will update site First Test Course To Have Manual Provider Id with:
Test Cleanup
|
results: Recorded in Jira https://jira.ucdavis.edu:8444/jira/browse/SAK-146, and is a part of White List Test https://jira.ucdavis.edu:8444/jira/browse/SAK-119
Provider Id Update Test Results
Would yield an example Summary Test Result of (both batch and manual updates):
Test | User | User Acting As | Result | Expected Result | Pass/Fail |
|---|---|---|---|---|---|
Site xxxxx: Provider Id Update Results | samerson | admin | provider id is ....., roster is same as CRN for provider id | provider id updated and seen in site tool under "provider id", roster reflects provider id roster | P |
results: Recorded in Jira https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/
...
...
...
...
...
...
| Panel | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
|
Test Cleanup
- Scott will remove all sites created in this step when all test cases are completed.
results: 5%2C+2006
Case 4 #Site Functionality
| Note | ||
|---|---|---|
| ||
Description = Test the capability of the system to add sites with the correct pages, tools, tabs, and look and feel. Category of Testing = unit, integration, functionality, regression Prerequisite Steps |
...
SmartSite+Course+Site+Tests%2C+Site+Functionality+Integration+Test+ |
...
Full Batch Run on Term
| Note | |||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||
Description = Test the batch method for a whole term Category of Testing = unit, integration, performance Prerequisite Steps
Test Cleanup
|
...
Results+Matrix for each site
Test Cleanup
|
results: Recorded as part of White List Scenario, and Jira https://jira.ucdavis.edu:8444/jira/browse/SAK-119
Jiras covered in this test case:
https://jira.ucdavis.edu:8444/jira/browse/SAK-121 = Tools and Pages
https://jira.ucdavis.edu:8444/jira/browse/SAK-23 = Tools configurable
Case 5 #Course Management Data Presented in Admin Tools
| Note | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||
Description = Test the course management content displayed in admin tools. Category of Testing = unit, integration, functionality, regression Prerequisite Steps
Test Cleanup
|
results: Recorded as part of White List Scenario, and Jira https://jira.ucdavis.edu:8444/jira/browse/SAK-119
Jiras covered in this test case:
https://jira.ucdavis.edu:8444/jira/browse/SAK-136 = Course Management Data (Description) details
Recording Results
The test results will be recorded in the above summary tables by integration testers, and a summary of this data will be ready for QA and the Sakai Development team to reviewJira by QA testers. They can be of table form or text form.
Feedback Mechanism
The above results will be analyzed by the QA team, SmartSite programming team, and SmartSite management to determine the steps needed and whether or not to hand off to QAquality of code. Results of this test should be discussed as near to the Monday meetings as possibleand/or Code Migration meetings when available.