Implementation Plan for, My Campus Messages, 2nd Try
Implementation Plan for My Campus Messages, October 28, 2008, Sak-2260 (story)
October 28, 2008
--Confirm maintenance mode
--We need to allow access from dedicated IPs for possible pre-upgrade work
--Adjust iptables on production nodes to allow JMX access to port 9000 for specified IPs
PORT 8443:
# 169.237.11.246 (thomas)
# 69.12.224.247 (thomas)
# 169.237.97.91 (zoidberg: vpn, terminal server)
# 169.237.97.133 (kirk)
# 169.237.11.220 (james)
# 169.237.11.130 (sandra)
# 169.237.11.6 (jon)
# 169.237.11.242 (mike)
# 169.237.11.37 (Pete Peterson)
# 169.237.205.242 (Nancy Wilkison)
# OTHERS
PORT 9000:
# 169.237.11.246
# OTHERS
- (04:00 Sandra) Set a marker in the database for restore comparison. Send email to ucd-sakai-dev, and ucd-sakai-qaqc--This is the marker and a date.
- (04:20 programmer) Put Smartsite prod into maintenance mode so no further updates will be made against the database.
- (4:30 Joncarlo) Begin installation of 64 bit jdk 1.5.0_16, sak-2326.Â
- ) stop the application on each node
- ) rm the java5 link and re-create it pointing to the new JVM
- (04:30 Prabhu) Verify that the regularly scheduled database back up is running as scheduled.
- (05:05 Prabhu) Verify that the database back up completed successfully.
- (05:15 Prabhu) Stop the log transfer / log apply process on primary database server
- (05:15 Prabhu) Copy the tables Sakai_site_tool_property and sakai_site_tool in the database for fast restore if needed.
- (05:20 Prabhu) Run the queries to verify the state of the database. Find the script here: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2260_info.sql
- (05:22 Prabhu) Display the output of the information runs in the IRC script for others to see.
- Team members: Save the output for comparison after the update scripts.
- Agree that we're ready to move forward
- (05:25 Prabhu) Run the database script to set the correct height and add the name My Campus Messages. Find the script here: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2260_name+size.sql
- (05:35 Prabhu) Run Count information scripts after the name and height change. Find the script here: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2260_info.sql
- (05:36 Team) Determine that the information scripts are correct:
- Total number of sites after the update script equals total number of site before the update script.
- Total number of home sites with tool with title of "My Campus Messages" equals total number of sites minus corrupt or unpublished ones.
- Total number of home sites with Workspace Tool equals one less than total sites.
- Total number of Workspace Tool sites with "My Campus Messages equals total number of sites with heights of 200 px..
- Total number of Workspace tools with heights of 200 px equals total number of Workspace tools with heights records.
- (05:46 Prabhu) Commit the sql to the database for the update scripts
- (05:50 programmer) Change the sakai.properties to add a url to the Workspace Tool. The url is for MyUCDavis's My Campus Messages. The url is:
myworkspace.info.url=https://my.ucdavis.edu/message_center/smartsite/messages.cfm
- (05:55 programmer) Programmers deploy tag Prod007 (Start-up head node ).
-
- Check catalina.out for any errors.
- Once Sakai has started, login and click on a few sites/tools to make sure that no errors are reported in catalina.out.
(05:55 Kirk, Sandra, QA) use sakaiprod1 to log in and check sites. Verify they have the My Campus Messages.(sakaiprod1 https://sakaiprod1.ucdavis.edu:8443/portal
- (06:00 programmer) Startup remaining nodes, one after another and monitor each catalina.out for potential problems.
- (06:05 Kirk, Sandra, QA, others) Release SmartSite for internal testing to determine "successful upgrade".
- Verify that we can access the system using: https://smartsite.ucdavis.edu:8443/portal
- (06:15 Kirk, Sandra, QA, Programmers, ...)
- Agree on go/no-go
- (06:15 - after go vote, Programmers) Adjust iptables on all nodes production nodes (sakaiprod1, ..., sakaiprod6) to allow public access.
- (06:15 Prabhu) Resume log transfer / log apply process.
- (06:15 Sandra) Notify Operations, ITExpress that system is ready. Ask Operations to send note (we compose) to TechNotices. Use Release Mgmt mailing list (use the motd that Roger wrote for us).
- (06:25 programmer) Roll out sak-1411, Mailtool has typo: "Open Campuss"
- (07:00 Kirk/Sandra) Update MOTD indicating upgrade successful and leaving request for input if problems are found by users.
Â
QA: Updated Test Plan - Post Production
----------------------------------------------------------------
Basic Function
- Does the My Campus Messages frame appear?
- Is there a Certificate error?
- Is it in the correct position on the page?
- Is the data the same as the data on smartsite-ng (production data)
WARNING: DO NOT DELETE OR MODIFY MESSAGES ON OTHER USERS ACCOUNTS, THIS IS PRODUCTION DATA
Staff, Student, Faculty Accounts
- New Accounts
- Open Campus Accounts
- Do all user accounts have the My Campus Messages frame?
Fall-back Procedure (If needed)
--------------------------------------------
In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
- (06:00 Programmers) Shutdown all application servers
- (06:05 Prabhu) Restore two affected tables in Sakai internal production database,
- (06:15 Programmers) Start nodes and test
- (06:20 Kirk, Sandra, QA,...) Start testing, verify restore is correct
- (06:25 programmer) Adjust iptables
- (06:25 Prabhu) Resume log transfer/log apply process
- (07:00) System ready for public access