/
Sakai Upgrade Process v2-4-x-HEAD to v2-5-x

Sakai Upgrade Process v2-4-x-HEAD to v2-5-x

Jul 18, 2008

IRC Channel for Communication

Maintenance Window

  • Date: July 19, 2008

  • Time: 08:00 to Sunday July 20, 12:00 (noon)

Upgrade Process

  • (08:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page

  • (08:45 Programmers) Run Content Conversion

    • On sakaiprod1, build ~/src/sakai_2-5-x-prod-XYZ

    • Make sure we export toolOrder.xml as well as sakai.properties into config-staging

    • change "db/db-util/conversion/runconversion.sh" replacing SNAPSHOT with M2

      • :%s/SNAPSHOT/M2/g

    • cd ~/src/sakai_2-5-x-prod-XYZ/content

    • change upgradeschema-oracle.config

      • # UpgradeSchema Control File (Default) # Conversion needs the database connection details dbDriver=oracle.jdbc.driver.OracleDriver dbURL=jdbc:oracle:thin:@ldap://oraldap1.ucdavis.edu:389/sakaitest,cn=OracleContext,dc=ucdavis,dc=edu dbUser=sakai dbPass=<?>

    • Run conversion

      • nohup ./content-runconversion.sh -j "$CATALINA_HOME/common/lib/ojdbc14-10-2-0-3.jar" -p "$HOME/sakai-home/local.properties" upgradeschema-oracle.config &> ~/content-runconversion.log &

    • The L1 and L2 conversion take about 50 minutes

  • (09:35 Programmers)

    • On each production node (sakaiprod1, ..., sakaiprod2) adjust the ucd-help location so that it points to the new ucd-help-2-5-x-prod branch

      • cd /var/www/html/smartsite rm ucd-help ln -s /afs/.ucdavis.edu/app/sakai/misc/help/smartsite/sakai_2-5-x-prod ucd-help

    • Start-up head node (sakaiprod1)

      • During the first startup, the Calendar conversion will take place : FND SAK-11204

        • Check catalina.out for any errors

    • Once Sakai has started, stop and start Sakai again on sakaiprod1 to make sure that the Calendar conversion runs only once and thus indicates that it successfully converted its content

    • Once Sakai has started for the second time. Try to login and click on a few sites/tools to make sure that no errors are reported in catalina.out

      • Click on the "Help" tool and make sure that it loads correctly. Also monitor catalina.out for any help related errors while it initializes help.

  • (09:50 Programmers)

    • Startup remaining nodes, one after another and monitor each catalina.out for potential problems

  • (10:00 Kirk, Sandra, QA, ...) Release SmartSite for initial testing to determine "successful upgrade"

  • (11:45 Kirk, Sandra, QA, Programmers, ...)

    • Agree on go/no-go

  • (12:00 - after go vote, Programmers) Adjust iptables on all nodes production nodes (sakaiprod1, ..., sakaiprod6) to allow public access

  • (12:01 Programmers) Notify Operations, ITExpress that system is ready.  Ask Operations to send note (we compose) to TechNotices.  Use Release Mgmt mailing list.

  • (12:00  Prabhu) Resume log transfer / log apply process

  • (12:02) System ready for public access

  • (12:03 Kirk/Sandra) Update MOTD indicating upgrade successful and leaving request for input if problems are found by users.

Fall-back Procedure (If needed)

  • In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks

  • (12:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page

  • (12:01 Programmers) Shutdown all application servers

  • (12:10 Prabhu) Restore Sakai internal production database

  • (12:35 Prabhu) Resume log transfer/log apply process

  • (12:40 Programmers) Install sakai-2-4-x-HEAD-prod-latest on all application nodes (sakaiprod1, ..., sakaiprod6)

    • Restore sakai-2-4-x-HEAD-prod-latest configuration files

  • (13:30 Programmers) Start nodes and test

  • (13:40 Kirk, Sandra, QA,...) Start testing

  • (14:00 Joncarlo) Adjust iptables

  • (14:01) System ready for public access

Post Upgrade TODOs:

  • ...