Sakai Upgrade Process v2.3.x to v2.4.x

IRC Channel for Communication

• server: irc.freenode.net
• channel: #ucdsakai
• Windows IRC client:
  • http://www.mirc.com/ (Information on the program)
  • Direct link to MIRC Download
• If you have any questions or would like help getting configured, talk to Emily (emily@ucdavis.edu)

Phase 1 : External Database Preparation

Phase 2 : Application and Internal Database Upgrade

Maintenance Window

• Date: June 23, 2007
• Time: 00:00 to 07:00

Upgrade Process

• (00:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
  • We need to allow access from dedicated IPs for possible pre-upgrade work, such as exporting Melete modules, etc.
  • Adjust iptables on head node (stubing) to allow JMX access to port 9000 for specified IPs
    • PORT 8443:
      • 169.237.11.246 (thomas)
      • 69.12.224.247 (thomas)
      • 169.237.97.91 (zoidberg: vpn, terminal server)
      • 169.237.97.133 (kirk)
      • 169.237.11.220 (james)
      • 169.237.11.23 (sandra)
      • 169.237.11.211 (emily)
      • 169.237.11.6 (jon)
      • 169.237.11.242 (mike)
      • 169.237.11.30 (lisa)
      • 169.237.97.223 (pat)
      • ...
    • PORT 9000:
      • IPs : 169.237.11.246
• (00:10 Prabhu) Backing up (CLONE) production database so that it can be fully restored.
• (01:00 Thomas) Run sakai_2_3_1-2_4_0_oracle_conversion.sql
  • sakai_2_3_1-2_4_0_oracle_conversion.sql
• (01:15 Thomas) Run SmartSite Extra Updates
  • smartsite_extra_update_2-4-x.sql
• (01:30 Thomas) Run SAKAI_USER_ID_MAP conversion script
  • eidChange.groovy
• (01:40 Thomas) Run SAKAI_REALM and SAKAI_REALM_PROVIDER conversion script
  • ProviderIdConversion.groovy
• (01:50 Thomas) Run SmartSite Extra Updates Part II
  • smartsite_extra_update_2-4-x.sql

• (01:55 Thomas/Emily) Backup all configuration files in production staging area
  • /afs/.ucdavis.edu/app/sakai/staging/config/*

    • mkdir /afs/.ucdavis.edu/app/sakai/staging/config/config-2-3-x.bak
      cp /afs/.ucdavis.edu/app/sakai/staging/config/* /afs/.ucdavis.edu/app/sakai/staging/config/config-2-3-x.bak/
      

      
      
• (02:10 Thomas/Emily) Start process of preparing Sakai v2.4.x source on the head node (stubing).
  • Getting the source code:

    • cd ~/src
      svn co https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-4-x-prod-001
      

      
      
  • Setting up the build environment:
  • ~/build.properties

    • maven.repo.remote=http://source.sakaiproject.org/maven/,https://mware.ucdavis.edu/maven/,http://www.ibiblio.org/maven2/
      maven.tomcat.home=/ucd/opt/sakai/tomcat-staging/
      maven.compile.source = 1.5
      maven.compile.target = 1.5
      

      
      
  • ~/.bash_profile

    • PATH=/ucd/opt/java5/bin:/ucd/opt/maven/bin:$PATH
      MAVEN_HOME=/ucd/opt/maven
      CATALINA_HOME=/ucd/pkg/apache-tomcat
      JAVA_HOME=/ucd/opt/java5
      JAVA_OPTS=" -d64 -server -Xmx6144m -Xms6144m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:PermSize=256m -XX:MaxPermSize=512m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dsakai.home=/ucd/opt/sakai/sakai-home/"
      CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
      MAVEN_OPTS="-Xmx1024m -Xms1024m"
      export PATH MAVEN_HOME CATALINA_HOME JAVA_HOME JAVA_OPTS CATALINA_OPTS MAVEN_OPTS
      

      
      
  • Delete old tomcat content

    • rm -fr /ucd/pkg/apache-tomcat/*
      

      
      
  • Delete old tomcat staging build

    • cd ~/bin
      ./tomcatclean.sh
      

      
      
  • Build the source code

    • cd ~/src/sakai_2-4-x-prod-001
      maven sakai
      

      
      
  • Copy staged tomcat

    • rm -fr /ucd/pkg/apache-tomcat/*
      cp -R /afs/.ucdavis.edu/app/sakai-test/staging/tomcat/apache-tomcat-5.5.23/* /ucd/pkg/apache-tomcat/
      

      
      
• (02:50 Thomas/Emily) Making sure all the configuration files are in place for v2.4.x such as sakai.properties etc.
  • Configure JAVA_OPTS to allow JMX access via JConsole
  • Synchronize all configuration files on the head node (stubing) with the AFS configuration staging area
    • /afs/.ucdavis.edu/app/sakai/staging/config
  • Edit sakai.properties on stubing

    • performChatMigration@org.sakaiproject.chat2.model.impl.ChatDataMigration=true
      chatMigrationExecuteImmediate@org.sakaiproject.chat2.model.impl.ChatDataMigration=true
      auto.ddl=true
      

      
      
  • Start-up head node (stubing) and check catalina.out for any errors
    • This startup will take longer than the usual because of the chat conversion
    • Check catalina.out for DB errors
    • Once the head node started up without any errors and you see the message that the chat conversion has completed, shut it down again
  • Edit sakai.properties on stubing

    • #performChatMigration@org.sakaiproject.chat2.model.impl.ChatDataMigration=false
      #chatMigrationExecuteImmediate@org.sakaiproject.chat2.model.impl.ChatDataMigration=false
      auto.ddl=false
      

  • Configure site creation to send email to smartsite-request@ucdavis.edu when unusual roster requests are made
  • Start-up head node (stubing) again
    • Make sure that there are no errors in catalina.out
  • Access head node and perform any appropriate testing
  • If there are no errors/problems, perform installation on second node (walton)
  • Start-up walton and perform any appropriate testing
• (03:30 Kirk, Sandra, ...) Release SmartSite for initial testing to determine "successful upgrade"
  • Test Phase Issue Handling Process
  • Access it via:
    • https://smartsite.ucdavis.edu:8443/portal
      • https://stubing.ucdavis.edu:8443/portal
      • https://walton.ucdavis.edu:8443/portal
• (06:55 Joncarlo) Adjust iptables on all nodes (stubing, walton) to allow public access
• (07:00) System ready for public access

Fallback Procedure (If needed)

• In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
• (08:00 Thomas/Emily) Shutdown all application servers
• (08:10 Prabhu) Restore Sakai internal production database on caje
• (08:40 Thomas/Emily) Install latest sakai-core-2-3-x-prod snapshot on all application nodes (stubing, walton)
  • Restore v2.3.x configuration files
• (09:30 Thomas/Emily) Start nodes and test
• (09:40 Kirk, Sandra, ...) Start testing
• (10:00 Joncarlo) Adjust iptables
• (10:05) System ready for public access

Post Upgrade TODOs:

• SAK-664 : Verify footer display, content and links are correct for the SmartSite 2.4 upgrade
• SAK-746 : Post-upgrade task for 2.4: Add site.add function to registered user role so that new users can create course and project sites
• SAK-751 : create CM role mapping for waitlisted students