Versions Compared

Old Version 88

changes.mady.by.user Scott Amerson

Saved on

compared with

New Version 89

changes.mady.by.user Scott Amerson

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

TEST PLAN DATE

preliminary schedule of testing: 9/1/06, 3:30pm.

TEST PLAN OVERVIEW

Scope of testing
This testing is designed to document the unit and integration test results of the programming team. For certain test cases, as described in the Test Cases section, the level of testing previously performed to handing off to QA will be documented. The level of involvement by system admins and programmers will be detailed for each test case.

Note
titlePurpose for this testing

The automatic course creation mechanism for SmartSite. Course sites, when done automatically via chron jobs, can be created and/or updated in several ways

  1. On-Demand List of courses (White list)
    • Will create only a pre-defined list of course sites
    • Not specific to a Term
  2. Excluded List of courses (Black list)
    • Will not be created by any mechanism
  3. Auto-update of existing course sites with group update (e.g. provider id)
    • Sets existing sites with the assigned provider id value, and will not provision a duplicate site with same provider id from full other batch run
  4. Full run of course creation for a designated term (will not batch white or black lists, as well as not create sites with Provider Id to be set)
Info
titleAdditional Items slated for testing
  1. Manually update provider Id for a site
    • Need to be done as a an admin user, or one that has access to admin realm
  2. See the results of the realms being refreshed (Jon's patch)

KNOWN ISSUES / PREVIOUS BUGS FIXED

  1. There is one known issue for this testing, related to the Home Page web content displayed for course sites. Currently, the Course Description information displays course title, which will be changed to description when the external data is mapped to a description field.
Info
titleJIRAs that explain features to be tested in this plan

http://jira.ucdavis.edu:8080/jira/browse/SAK-23 (Add ability to customize pages and tools added to pages (both default and provisional) when provisioning course sites)
http://jira.ucdavis.edu:8080/jira/browse/SAK-25 (Remove admin user from sites, during course creation script)
http://jira.ucdavis.edu:8080/jira/browse/SAK-119 (Course White List)
http://jira.ucdavis.edu:8080/jira/browse/SAK-121 (Batch provisioning should make a toolset based on OOTB and other (provisional and contrib) tools)
http://jira.ucdavis.edu:8080/jira/browse/SAK-136 (Add Course title (from Banner) is the field presently mapped for course description information when provisioning sites. A course description (short and long) should be mapped from external data so that it can be referenced by certain course tools)
http://jira.ucdavis.edu:8080/jira/browse/SAK-140 (Course Black List)
http://jira.ucdavis.edu:8080/jira/browse/SAK-141 (Batching is done under a 'batch admin' user)
http://jira.ucdavis.edu:8080/jira/browse/SAK-146 (Set Provider Id For Manually Created Courses)

other non-batch related JIRAs covered:
http://jira.ucdavis.edu:8080/jira/browse/SAK-143 (enable distributed authentication using AFS file system)
TODO http://jira.ucdavis.edu:8080/jira/browse/SAK-133 (AuthZ refresh test)

Note
titleJIRAs that reference test scenarios described in this test plan

TODO

TEST ENVIRONMENT

  • Hardware Setup

    Server Type

    Location

    database

    bricker.ucdavis.edu with banner-test (external db), and sakai dev (internal db)

    application

    isaac.ucdavis.edu, sakai-dev

    web server

    url,

    http://sakai-dev.ucdavis.edu
  1. Note: The application does not need to be stopped for these tests to occur
  • Additional Setup Needed
    **Batch admin account (e.g. ssbatchadmin)
    **Programmers have access to testable sites (via SU tool, admin workspace)
    **SU tool and Quartz tools are available in admin workspace/realm
    **All login accounts that reference kerberos names can be "provided"
  • External-Banner Test Data to be Used As Representative Course Scenarios

    Scenario

    Course

    Roster Count

    Instructor Kerb Name

    Course Details (Catalog Root, Suffix, Section, Description)

    Term Identifier

    Test Case Used In

    Invalid CRNs

    Invalid CRN (Number)

    20061-06-111

    NA

    NA

    NA

    NA

    White List, Provider Id

    Invalid CRN (Number)

    2006-06-12345

    NA

    NA

    NA

    NA

    White List

    Invalid CRN (Number)

    2006-03-60024

    NA

    NA

    NA

    NA

    White List

    *additional* Invalid CRN (Number) available

    1995-03-76288

    NA

    NA

    NA

    NA

    White List

    Invalid CRN (Bogus Text)

    2006-sfs-sfsfdfsdfsfsfs

    NA

    NA

    NA

    NA

    White List

    No CRN (empty file)

    null

    NA

    NA

    NA

    NA

    White List,Black List, Provider Id

    No CRN (null CRN value)

    null

    NA

    NA

    NA

    NA

    Provider Id

    Valid CRNs

    CRN with no enrollment

    1995-01-76288

    0

    fzthorn, (Robert Thornton)

    PLB, 092, 012, Internship

    Winter Quarter 1995

    White List

    CRN with no enrollment

    2004-03-60097

    0

    dlmiller (Douglas Miller)

    ECN, 298 , 026, Group Study

    Spring Quarter 2004

    Provider Id

    CRN with enrollment

    2006-06-60005

    11 (all valid knames)

    acham (Christine Acham)

    AAS, 198, S01, Directed Gp Study

    Summer Special Session 2006

    White List,Black List,Provider Id

    CRN with enrollment

    2006-01-40018

    14 (all valid knames)

    calymos (Calvin Lymos)

    AAS, 154, 001, University Gospel Choir

    Winter Quarter 2006

    Black List

    CRN with large enrollment number

    2006-01-60001

    84 (all valid knames)

    ewlarsen (Eric Larsen)

    LDA, 190 , 001, ProSeminar

    Winter Quarter 2006

    White List

    TODO|CRN with large enrollment number|2006-03-xxxxx|NA|NA|NA|x|Provider Id|
    TODO|CRN with large enrollment number, some student kerbs don't exist|2006-03-xxxxx|NA|NA|NA|x|Provider Id|

    Valid Term

    Term with large number CRNs

    2004-01

    NA

    NA

    32573 sites

    Winter Quarter 2004

    Full Batch Run

  1. A few test course sites need to be created in advance so that the batch job can modify these sites with a respective provider Id. These sites are documented in the Provider Id Test Section.

DEPENDENCIES

  • At least UserDirectoryProvider enabled to allow user accounts to be looked up automatically via kerberos name (Note:all Providers will be enabled)
  • Distributed authentication using AFS (SAK-143)
  • Sakai-dev will be used in its current state
  • We will be referencing sakai-test external for populating course data

ROLES AND RESPONSIBILITIES

  • Unit/Integration Testers (Programmers, Managers, etc)*
    which would include any of the following persons that may be available:
    • Application admins (Programmers available, including Thomas, Lisa W., James, Brian, Jon G., Scott)
    • Database mViews admin (Brian Donnelly)

COURSE SITE BATCHING DOCUMENTATION

JUNIT TEST RESULTS

The JUnit tests results are located on the server at:
/var/sakai/ucd-data/course-batch-unit-tests-082006

Also, the test results can be found here https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/Course+Batch+Creation+Unit+Test+Results

All unit tests passed!!

TEST PLAN PROCESS

This Course Batching Unit/Integration Test Plan will be coordinated by Scott Amerson, and will document the results of unit testing and integration testing for the batching process before handing off to QA. All tests must pass before handing off to QA.

Steps involved in the test process will be varied depending on the given test performed (as described below in the Test Cases section).

Info

*NOTE: The order of the test cases will be as follows. White List Testing, Black List Testing, Provider Id Testing *

  1. Site Functionality Testing, Course Management Testing, and Provider Id testing are each part of the White List testing scenario.
  2. In order to test the Black List functionality, we will need to have a superset of CRNs that are trying to be added at the same time. To accomplish this, we will use the White List scenario with a same CRN that is included in the Black List file. See Black List Test Scenario for more information.

TEST CASES

Info
titleKey for Test Case Scenarios

Prerequisite Steps = actions needed before performing any of the given tests
Category of Testing = the described testing is at what level?
Description = description of the test performed

White List

Note
titleWhite List Testing

Description = Test the capability of the system to add "on-demand" course sites that are listed in a file. Courses are not required to have any enrollment in order to be created.

Category of Testing = unit, integration

Prerequisite Steps

  1. Scott will add a list of CRNs (valid or not) as the white-list.txt file in /var/sakai/ucd-data folder on the server, in the form of TERMYR-TERMCD-CRN for each entry. Use this file file to use here
  2. Scott will que the batch job to run, only for the above list (5 min elapsed time needed)
  3. Scott will give maintain permissions on the applicable sites, to the appropriate users
  • Brian = look up CRN xxxxx
  • Sandra = look up CRN yyyyy
  • Scott will assist users in accessing the site(s) if needed
  • Thomas = view logs, look up CRN zzzzz
  • Jon G. = look up CRN wwwww
  1. Scott will advise programmers that the batch has been run and sites have been created
  2. Thomas and Scott will monitor server activity while testers are using sakai-dev.

Test Scenario

  1. Above users will perform functionality testing for each CRN, including :
    **perform functionality test https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/Course+Site+Functionality+Tests%2C+SmartSite+Batch+Integration+Testing+Matrix for each site
    **perform course management test https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/SmartSite+Course+Site%2C+Course+Management+Integration+Test+Matrix for each site
    **perform provider id test https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/SmartSite+Provider+Id+Update+of+Existing+Sites%2C+Integration+Test+Matrix for each site

Test Cleanup

  1. Scott will remove all sites created in this step when all test cases are completed.

White List Test Results

Would yield an example Summary Test Result of:

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site xxxxx (created): Functionality Results

samerson

fzolupon

All tests pass

batching of course succeeded

P

Site xxxxx (created): Course Management Test Results

samerson

QA Admin

All tests pass

course content correct

P

Site xxxxx (created): Provider Id Test Results

samerson

QA Admin

All tests pass

provider id correct, roster number correct

P

or..

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site xxxxy (not created):Test Results

samerson

QA Admin

All tests pass, not provisioned

course not created, correct

P

Info
borderColor#ccc
bgColorefefef
titleBGColorffffcc
titleTest Results, White List
borderStyledashed

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site xxxxx (created): Functionality Results

bdonnell

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site yyyyy (created): Functionality Results

smavocet

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site wwwww (created): Functionality Results

tpamsler

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site zzzzz (created): Functionality Results

ccjon

Black List

Note
titleBlack List Testing

Description = Test the capability of the system to not add course sites that are listed in a file. Courses in this list will never be batch created.

Category of Testing = unit, integration

Prerequisite Steps

  1. Scott will add a list of CRNs (valid or not) to the white-list.txt file in /var/sakai/ucd-data folder on the server, in the form of TERMYR-TERMCD-CRN for each entry. This list will be different than that used for the White List Test Scenario. Use this white list
  2. Scott will add a list of CRNs (valid or not) to the black-list.txt file in /var/sakai/ucd-data folder on the server, in the form of TERMYR-TERMCD-CRN for each entry. Use this black list file file to use here
  3. Scott will que the batch jobs to run, first Black List, then White List, (5 min time elapse needed)
  4. Scott will advise programmers that the batches have been run
  5. Thomas and Scott will monitor server activity while testers are using sakai-dev.

Test Scenario

  • Jon G. = will look for CRN xxxxx, yyyyy, zzzzz as user ....... These CRNs will be listed in the Black List
  • Brian = will look for CRN aaaaa that is listed both in the white-list.txt and black-list.txt files
  • Thomas = view logs, internal sakai db
  • Scott = help Jon

Test Cleanup

  1. Scott will remove all sites created in this step when all test cases are completed.

Black List Test Results
Would yield an example Summary Test Result of:

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site xxxxy (not created):Test Results

samerson

QA Admin

All tests pass, not provisioned

course not created, correct

P

Info
borderColor#ccc
bgColorefefef
titleBGColorffffcc
titleTest Results, Black List
borderStyledashed

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site xxxxx (not created):Test Results

ccjon

x

x

site not created in UI or db

x

Site yyyyy (not created):Test Results

ccjon

x

x

site not created in UI or db

x

Site zzzzz (not created):Test Results

ccjon

x

x

site not created in UI or db

x

Site aaaaa (not created):Test Results

bdonnell

x

x

site not created in UI or db

x

*Note, pass/fail depends on db check also for this test! Jon and Brian will consult with Thomas and Scott about test case result.

Update Existing Site with Provider Id

Note
titleUpdate existing sites with appropriate provider id, enabling dynamic rosters (**When CourseManagementProvider enabled**)

Description = Test the capability of the system to update existing course sites with associated provider ids. This functionality can be done either in the UI or batch, however we want to
test the ability to run this at any time.

Category of Testing = unit, integration

Prerequisite Steps
Results Recording Steps summarized in:
**perform provider id test https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/SmartSite+Provider+Id+Update+of+Existing+Sites%2C+Integration+Test+Matrix for each site

  1. Scott will add a list of sites and provider ids (valid or not) to the provider-list.txt file in /var/sakai/ucd-data folder on the server, in the form of site id, TERMYR-TERMCD-CRN for each entry. Use this file use file here
  2. Scott will que the batch jobs to run, just the provider id que batch run, (5 min time elapse needed)
  3. Scott will advise programmers that the batch has been run
  4. Thomas and Scott will monitor server activity while testers are using sakai-dev.

Test Scenario

  • Jon G. = will look for site xxxxx's providerid and roster, logging in as admin
  • Sandra = will look for site yyyyy's providerid and roster, logging in as ssbatchadmin
  • Sandra = will look for site zzzzz's providerid and roster, logging in as ssbatchadmin
  • Brian = will validate provider ids and rosters
  • Thomas = view logs, internal sakai db
  • Scott = help Sandra, Jon, and log in as himself for manual updates

Manual Process Check (update provider ids for sites manually):

  1. After site xxxxx,yyyyy, and zzzzz's provider ids are validated,
    Scott will update site xxxxx with:
    providerId =
    and.. site yyyyy with:
    providerId =
  2. Thomas will check site xxxxx and site yyyyy's provider id and rosters

Test Cleanup

  1. Scott will remove all sites created in this step when all test cases are completed.

*Provider Id Update Test Results
Would yield an example Summary Test Result of (both batch and manual updates):

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site xxxxx: Provider Id Update Results

samerson

admin

provider id is ....., roster is same as CRN for provider id

provider id updated and seen in site tool under "provider id", roster reflects provider id roster

P

Info
borderColor#ccc
bgColorefefef
titleBGColorffffcc
titleTest Results, Provider Id Update, Batch and Manual
borderStyledashed

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site xxxxx: Provider Id Update Results (batch)

ccjon

admin

x

provider id updated and seen in site tool under "provider id", roster reflects provider id roster

P

Site yyyyy: Provider Id Update Results (batch)

smavocet

ssbatchadmin

x

provider id updated and seen in site tool under "provider id", roster reflects provider id roster

P

Site zzzzz: Provider Id Update Results (batch)

smavocet

ssbatchadmin

x

no roster appears because provider id invalid

P

Test

User Role

User Acting As

Result

Expected Result

Pass/Fail

Site xxxxx: Provider Id Update Results (manual)

samerson

samerson

x

provider id updated via realm edit with accurate data, that from provider id of site yyyyy (provider id=)

P

Site yyyyy: Provider Id Update Results (manual)

samerson

samerson

x

provider id updated via realm edit with accurate data, that from provider id of site yyyyy (provider id=)

P

Site Functionality

Note
titleSite functionality

Description = Test the capability of the system to add sites with the correct pages, tools, tabs, and look and feel.

Category of Testing = unit, integration, functionality

Prerequisite Steps
Results Recording Steps summarized in:
**perform functionality test https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/Course+Site+Functionality+Tests%2C+SmartSite+Batch+Integration+Testing+Matrix for each site
looking for expected results
Test Scenario

Panel
borderColor#ccc
bgColorF7D6C1
titleBGColor#efefef
titleIndividual Functionality Tests
borderStyledashed
  • -Look and Feel- *
  • Pages *
  • Tools on each page *
  • Click-through on tools *
  • Order *
  • Tabs *

Test Cleanup

  1. Scott will remove all sites created in this step when all test cases are completed.
Info
borderColor#ccc
bgColorwhite
titleBGColorefefef
titleSite Functionality Test Results
borderStyledashed

Test

User Role

User Acting As

Page Results

Tool Results

Click Through Results

Order of Pages Results

Tab Results

Pass/Fail

Site xxxxx: Functionality Test Results, white list testcase

samerson

admin

x

x

x

x

x

P

Test

User Role

User Acting As

Page Results

Tool Results

Click Through Results

Order of Pages Results

Tab Results

Pass/Fail

Site xxxxx: Functionality Test Results, white list testcase

smavocet

admin

x

x

x

x

x

P

Test

User Role

User Acting As

Page Results

Tool Results

Click Through Results

Order of Pages Results

Tab Results

Pass/Fail

Site xxxxx: Functionality Test Results, white list testcase

ccjon

admin

x

x

x

x

x

P

Test

User Role

User Acting As

Page Results

Tool Results

Click Through Results

Order of Pages Results

Tab Results

Pass/Fail

Site xxxxx: Functionality Test Results, white list testcase

tpa

admin

x

x

x

x

x

P

Test

User Role

User Acting As

Page Results

Tool Results

Click Through Results

Order of Pages Results

Tab Results

Pass/Fail

Site xxxxx: Functionality Test Results, white list testcase

bdonnell

admin

x

x

x

x

x

P

Course Management Data Presented in Admin Tools

Note
titleCourse Management data present

Description = Test the course management content displayed in admin tools.

Category of Testing = unit, integration, functionality

Prerequisite Steps
Results Recording Steps summarized in:
**perform course management test https://confluence.ucdavis.edu:8443/confluence/display/UCDSAKAI/SmartSite+Course+Site%2C+Course+Management+Integration+Test+Matrix for each site
looking for expected results

Panel
borderColor#ccc
bgColorwhite
titleBGColorefefef
titleTest Scenario
borderStyledashed
  • Site navigation *
  • WorkSite Setup navigation *
  • Site Info tool *

Test Cleanup

  1. Scott will remove all sites created in this step when all test cases are completed.
Info
borderColor#ccc
bgColorwhite
titleBGColorefefef
titleCourse Management Test Results
borderStyledashed

Test

User Role

User Acting As

Site Navigation

WorkSite Setup Navigation

Site Info Tool

Pass/Fail

Site xxxxx: Course Management Test Results, white list testcase

samerson

admin

x

x

x

P

Test

User Role

User Acting As

Site Navigation

WorkSite Setup Navigation

Site Info Tool

Pass/Fail

Site xxxxx: Course Management Test Results, white list testcase

smavocet

admin

x

x

x

P

Test

User Role

User Acting As

Site Navigation

WorkSite Setup Navigation

Site Info Tool

Pass/Fail

Site xxxxx: Course Management Test Results, white list testcase

ccjon

admin

x

x

x

P

Test

User Role

User Acting As

Site Navigation

WorkSite Setup Navigation

Site Info Tool

Pass/Fail

Site xxxxx: Course Management Test Results, white list testcase

bdonnell

admin

x

x

x

P

Test

User Role

User Acting As

Site Navigation

WorkSite Setup Navigation

Site Info Tool

Pass/Fail

Site xxxxx: Course Management Test Results, white list testcase

tpa

admin

x

x

x

P

Recording Results

The test results will be recorded in the above summary tables by integration testers, and a summary of this data will be ready for QA and the Sakai Development team to review.

Feedback Mechanism

The above results will be analyzed by the SmartSite programming team, and management to determine the steps needed and whether or not to hand off to QA. Results of this test should be discussed as near to the Monday meetings as possible.