Implementation Plan for, My Campus Messages, 2nd Try

Owned by Former user (Deleted)
Last updated: Oct 27, 2008Version comment
3 min read

Implementation Plan for My Campus Messages, October 28, 2008, Sak-2260 (story)

October 28, 2008

--Confirm maintenance mode

--We need to allow access from dedicated IPs for possible pre-upgrade work

--Adjust iptables on production nodes to allow JMX access to port 9000 for specified IPs

PORT 8443:

# 169.237.11.246 (thomas)

# 69.12.224.247 (thomas)

# 169.237.97.91 (zoidberg: vpn, terminal server)

# 169.237.97.133 (kirk)

# 169.237.11.220 (james)

# 169.237.11.130 (sandra)

# 169.237.11.6 (jon)

# 169.237.11.242 (mike)

# 169.237.11.37 (Pete Peterson)

# 169.237.205.242 (Nancy Wilkison)

# OTHERS

PORT 9000:

# 169.237.11.246

# OTHERS

  • (04:00 Sandra) Set a marker in the database for restore comparison.  Send email to ucd-sakai-dev, and ucd-sakai-qaqc--This is the marker and a date.
  • (04:20 programmer) Put Smartsite prod into maintenance mode so no further updates will be made against the database.
  • (4:30 Joncarlo) Begin installation of 64 bit jdk 1.5.0_16, sak-2326. 
    • ) stop the application on each node
    • ) rm the java5 link and re-create it pointing to the new JVM
  • (04:30 Prabhu) Verify that the regularly scheduled database back up is running as scheduled.
  • (05:05 Prabhu) Verify that the database back up completed successfully.
  • (05:15 Prabhu) Stop the log transfer / log apply process on primary database server
  • (05:15 Prabhu) Copy the tables Sakai_site_tool_property and sakai_site_tool in the database for fast restore if needed.
  • (05:20 Prabhu) Run the queries to verify the state of the database.  Find the script here: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2260_info.sql
  • (05:22 Prabhu) Display the output of the information runs in the IRC script for others to see.
    • Team members: Save the output for comparison after the update scripts.
    • Agree that we're ready to move forward
  • (05:25 Prabhu) Run the database script to set the correct height and add the name My Campus Messages.  Find the script here: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2260_name+size.sql
  • (05:35 Prabhu) Run Count information scripts after the name and height change.  Find the script here: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2260_info.sql
  • (05:36 Team) Determine that the information scripts are correct:
    • Total number of sites after the update script equals total number of site before the update script.
    • Total number of home sites with tool with title of "My Campus Messages" equals total number of sites minus corrupt or unpublished ones.
    • Total number of home sites with Workspace Tool equals one less than total sites.
    • Total number of Workspace Tool sites with "My Campus Messages equals total number of sites with heights of 200 px..
    • Total number of Workspace tools with heights of 200 px equals total number of Workspace tools with heights records.
  • (05:46 Prabhu) Commit the sql to the database for the update scripts
  • (05:50 programmer) Change the sakai.properties to add a url to the Workspace Tool. The url is for MyUCDavis's My Campus Messages. The url is:

myworkspace.info.url=https://my.ucdavis.edu/message_center/smartsite/messages.cfm

  • (05:55 programmer) Programmers deploy tag Prod007 (Start-up head node ).
    • Check catalina.out for any errors.
    • Once Sakai has started, login and click on a few sites/tools to make sure that no errors are reported in catalina.out.

(05:55 Kirk, Sandra, QA) use sakaiprod1 to log in and check sites.  Verify they have the My Campus Messages.(sakaiprod1 https://sakaiprod1.ucdavis.edu:8443/portal

  • (06:00 programmer) Startup remaining nodes, one after another and monitor each catalina.out for potential problems.
  • (06:05 Kirk, Sandra, QA, others) Release SmartSite for internal testing to determine "successful upgrade".
  • (06:15 Kirk, Sandra, QA, Programmers, ...)
    • Agree on go/no-go
  • (06:15 - after go vote, Programmers) Adjust iptables on all nodes production nodes (sakaiprod1, ..., sakaiprod6) to allow public access.
  • (06:15 Prabhu) Resume log transfer / log apply process.
  • (06:15 Sandra) Notify Operations, ITExpress that system is ready. Ask Operations to send note (we compose) to TechNotices. Use Release Mgmt mailing list (use the motd that Roger wrote for us).
  • (06:25 programmer) Roll out sak-1411, Mailtool has typo: "Open Campuss"
  • (07:00 Kirk/Sandra) Update MOTD indicating upgrade successful and leaving request for input if problems are found by users.

 
QA: Updated Test Plan - Post Production
----------------------------------------------------------------
Basic Function
- Does the My Campus Messages frame appear?
- Is there a Certificate error?
- Is it in the correct position on the page?
- Is the data the same as the data on smartsite-ng (production data)
WARNING: DO NOT DELETE OR MODIFY MESSAGES ON OTHER USERS ACCOUNTS, THIS IS PRODUCTION DATA

Staff, Student, Faculty Accounts
- New Accounts
- Open Campus Accounts
- Do all user accounts have the My Campus Messages frame?
Fall-back Procedure (If needed)
--------------------------------------------
In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks

  • (06:00 Programmers) Shutdown all application servers
  • (06:05 Prabhu) Restore two affected tables in Sakai internal production database,
  • (06:15 Programmers) Start nodes and test
  • (06:20 Kirk, Sandra, QA,...) Start testing, verify restore is correct
  • (06:25 programmer) Adjust iptables
  • (06:25 Prabhu) Resume log transfer/log apply process
  • (07:00) System ready for public access