Implementation Plan for, My Campus Messages, 2nd Try

Implementation Plan for My Campus Messages, October 28, 2008, Sak-2260 (story)

October 28, 2008

--Confirm maintenance mode

--We need to allow access from dedicated IPs for possible pre-upgrade work

--Adjust iptables on production nodes to allow JMX access to port 9000 for specified IPs

PORT 8443:

# 169.237.11.246 (thomas)

# 69.12.224.247 (thomas)

# 169.237.97.91 (zoidberg: vpn, terminal server)

# 169.237.97.133 (kirk)

# 169.237.11.220 (james)

# 169.237.11.130 (sandra)

# 169.237.11.6 (jon)

# 169.237.11.242 (mike)

# 169.237.11.37 (Pete Peterson)

# 169.237.205.242 (Nancy Wilkison)

# OTHERS

PORT 9000:

# 169.237.11.246

# OTHERS

(04:00 Sandra) Set a marker in the database for restore comparison. Send email to ucd-sakai-dev, and ucd-sakai-qaqc--This is the marker and a date.

(04:20 programmer) Put Smartsite prod into maintenance mode so no further updates will be made against the database.
(4:30 Joncarlo) Begin installation of 64 bit jdk 1.5.0_16, sak-2326.
- ) stop the application on each node
- ) rm the java5 link and re-create it pointing to the new JVM

(04:30 Prabhu) Verify that the regularly scheduled database back up is running as scheduled.
(05:05 Prabhu) Verify that the database back up completed successfully.
(05:15 Prabhu) Stop the log transfer / log apply process on primary database server
(05:15 Prabhu) Copy the tables Sakai_site_tool_property and sakai_site_tool in the database for fast restore if needed.
(05:20 Prabhu) Run the queries to verify the state of the database. Find the script here: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2260_info.sql
(05:22 Prabhu) Display the output of the information runs in the IRC script for others to see.
- Team members: Save the output for comparison after the update scripts.
- Agree that we're ready to move forward
(05:25 Prabhu) Run the database script to set the correct height and add the name My Campus Messages. Find the script here: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2260_name+size.sql
(05:35 Prabhu) Run Count information scripts after the name and height change. Find the script here: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2260_info.sql
(05:36 Team) Determine that the information scripts are correct:
- Total number of sites after the update script equals total number of site before the update script.
- Total number of home sites with tool with title of "My Campus Messages" equals total number of sites minus corrupt or unpublished ones.
- Total number of home sites with Workspace Tool equals one less than total sites.
- Total number of Workspace Tool sites with "My Campus Messages equals total number of sites with heights of 200 px..
- Total number of Workspace tools with heights of 200 px equals total number of Workspace tools with heights records.
(05:46 Prabhu) Commit the sql to the database for the update scripts
(05:50 programmer) Change the sakai.properties to add a url to the Workspace Tool. The url is for MyUCDavis's My Campus Messages. The url is:

myworkspace.info.url=https://my.ucdavis.edu/message_center/smartsite/messages.cfm

(05:55 programmer) Programmers deploy tag Prod007 (Start-up head node ).

- Check catalina.out for any errors.
- Once Sakai has started, login and click on a few sites/tools to make sure that no errors are reported in catalina.out.

(05:55 Kirk, Sandra, QA) use sakaiprod1 to log in and check sites. Verify they have the My Campus Messages.(sakaiprod1 https://sakaiprod1.ucdavis.edu:8443/portal

(06:00 programmer) Startup remaining nodes, one after another and monitor each catalina.out for potential problems.
(06:05 Kirk, Sandra, QA, others) Release SmartSite for internal testing to determine "successful upgrade".
- Verify that we can access the system using: https://smartsite.ucdavis.edu:8443/portal
(06:15 Kirk, Sandra, QA, Programmers, ...)
- Agree on go/no-go

(06:15 - after go vote, Programmers) Adjust iptables on all nodes production nodes (sakaiprod1, ..., sakaiprod6) to allow public access.
(06:15 Prabhu) Resume log transfer / log apply process.
(06:15 Sandra) Notify Operations, ITExpress that system is ready. Ask Operations to send note (we compose) to TechNotices. Use Release Mgmt mailing list (use the motd that Roger wrote for us).
(06:25 programmer) Roll out sak-1411, Mailtool has typo: "Open Campuss"
(07:00 Kirk/Sandra) Update MOTD indicating upgrade successful and leaving request for input if problems are found by users.

QA: Updated Test Plan - Post Production
----------------------------------------------------------------
Basic Function
- Does the My Campus Messages frame appear?
- Is there a Certificate error?
- Is it in the correct position on the page?
- Is the data the same as the data on smartsite-ng (production data)
WARNING: DO NOT DELETE OR MODIFY MESSAGES ON OTHER USERS ACCOUNTS, THIS IS PRODUCTION DATA

Staff, Student, Faculty Accounts
- New Accounts
- Open Campus Accounts
- Do all user accounts have the My Campus Messages frame?
Fall-back Procedure (If needed)
--------------------------------------------
In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks

(06:00 Programmers) Shutdown all application servers
(06:05 Prabhu) Restore two affected tables in Sakai internal production database,
(06:15 Programmers) Start nodes and test
(06:20 Kirk, Sandra, QA,...) Start testing, verify restore is correct
(06:25 programmer) Adjust iptables
(06:25 Prabhu) Resume log transfer/log apply process
(07:00) System ready for public access