Sakai Upgrade Process v2-4-x-HEAD to v2-5-x

Owned by Thomas Amsler
Last updated: Jul 18, 2008 by James RenfroVersion comment

IRC Channel for Communication

Maintenance Window

  • Date: July 19, 2008
  • Time: 08:00 to Sunday July 20, 12:00 (noon) (question)

Upgrade Process

  • (08:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
  • (08:45 Programmers) Run Content Conversion
    • On sakaiprod1, build ~/src/sakai_2-5-x-prod-XYZ
    • Make sure we export toolOrder.xml as well as sakai.properties into config-staging
    • change "db/db-util/conversion/runconversion.sh" replacing SNAPSHOT with M2
      • :%s/SNAPSHOT/M2/g
    • cd ~/src/sakai_2-5-x-prod-XYZ/content
    • change upgradeschema-oracle.config 
      • # UpgradeSchema Control File (Default)
# Conversion needs the database connection details
dbDriver=oracle.jdbc.driver.OracleDriver
dbURL=jdbc:oracle:thin:@ldap://oraldap1.ucdavis.edu:389/sakaitest,cn=OracleContext,dc=ucdavis,dc=edu
dbUser=sakai
dbPass=<?>
    • Run conversion 
      • nohup ./content-runconversion.sh -j "$CATALINA_HOME/common/lib/ojdbc14-10-2-0-3.jar"  -p "$HOME/sakai-home/local.properties" upgradeschema-oracle.config &> ~/content-runconversion.log &
    • The L1 and L2 conversion take about 50 minutes
  • (09:35 Programmers)
    • On each production node (sakaiprod1, ..., sakaiprod2) adjust the ucd-help location so that it points to the new ucd-help-2-5-x-prod branch 
      • cd /var/www/html/smartsite
rm ucd-help
ln -s /afs/.ucdavis.edu/app/sakai/misc/help/smartsite/sakai_2-5-x-prod ucd-help
    • Start-up head node (sakaiprod1)
      • During the first startup, the Calendar conversion will take place : FND SAK-11204
        • Check catalina.out for any errors
    • Once Sakai has started, stop and start Sakai again on sakaiprod1 to make sure that the Calendar conversion runs only once and thus indicates that it successfully converted its content
    • Once Sakai has started for the second time. Try to login and click on a few sites/tools to make sure that no errors are reported in catalina.out
      • (warning) Click on the "Help" tool and make sure that it loads correctly. Also monitor catalina.out for any help related errors while it initializes help.
  • (09:50 Programmers)
    • Startup remaining nodes, one after another and monitor each catalina.out for potential problems
  • (10:00 Kirk, Sandra, QA, ...) Release SmartSite for initial testing to determine "successful upgrade"
  • (11:45 Kirk, Sandra, QA, Programmers, ...)
    • Agree on go/no-go
  • (12:00 - after go vote, Programmers) Adjust iptables on all nodes production nodes (sakaiprod1, ..., sakaiprod6) to allow public access
  • (12:01 Programmers) Notify Operations, ITExpress that system is ready.  Ask Operations to send note (we compose) to TechNotices.  Use Release Mgmt mailing list.
  • (12:00  Prabhu) Resume log transfer / log apply process
  • (12:02) System ready for public access
  • (12:03 Kirk/Sandra) Update MOTD indicating upgrade successful and leaving request for input if problems are found by users.

(minus) Fall-back Procedure (If needed)

  • In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
  • (12:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
  • (12:01 Programmers) Shutdown all application servers
  • (12:10 Prabhu) Restore Sakai internal production database
  • (12:35 Prabhu) Resume log transfer/log apply process
  • (12:40 Programmers) Install sakai-2-4-x-HEAD-prod-latest on all application nodes (sakaiprod1, ..., sakaiprod6)
    • Restore sakai-2-4-x-HEAD-prod-latest configuration files
  • (13:30 Programmers) Start nodes and test
  • (13:40 Kirk, Sandra, QA,...) Start testing
  • (14:00 Joncarlo) Adjust iptables
  • (14:01) System ready for public access

Post Upgrade TODOs:

  • ...