Sakai Upgrade Process v2.1.x to v2.3.x

IRC Channel for Communication

• server: irc.freenode.net
• channel: #ucdsakai
• Windows IRC client:
  • http://www.mirc.com/ (Information on the program)
  • Direct link to MIRC Download
• If you have any questions or would like help getting configured, talk to Emily (emily@ucdavis.edu)

Maintenance Window

• Date: February 24, 2007
• Time: 00:00 to 08:00

Upgrade Process

• (00:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
  • We need to allow access from dedicated IPs for possible pre-upgrade work, such as exporting Melete modules, etc.
  • Adjust iptables on head node (stubing) to allow JMX access to port 9000 for specified IPs
    • PORT 8443:
      • 169.237.11.40 (thomas)
      • 69.12.224.247 (thomas)
      • 169.237.97.91 (zoidberg: vpn, terminal server)
      • 169.237.97.133 (kirk)
      • 169.237.11.220 (james)
      • 169.237.11.23 (sandra)
      • 169.237.11.211 (emily)
      • 169.237.11.6 (Jon)
      • ...
    • PORT 9000:
      • IPs : 169.237.11.40
• (00:10 James) Export all Melete modules
  • On production login to any modules tool, go to Manage, go to Export/Import, click on Bulk Export
  • File will be written to local machine
  • File is about 60 MB and is called Monster.zip
  • Shouldn't take longer than 10 minutes to export
• (01:00 Thomas/Emily) Shutting down Sakai instances on all nodes (stubing, walton)
• (01:10 Prabhu) Backing up (CLONE) production database so that it can be fully restored.
  • This backup/clone should maintain the names for all constraints, indexes, etc.
  • Production SmartSite internal DB server is caje
• (02:00 Thomas/Emily) Perform maintenance on the SmartSite internal DB on caje
  • Remove all existing melete tables :

    • drop table MELETE_CC_LICENSE;
      drop table MELETE_COURSE_MODULE;
      drop table MELETE_MODULE_SHDATES;
      drop table MELETE_MODULE_STUDENT_PRIVS;
      drop table MELETE_SECTION;
      drop table MELETE_USER_PREFERENCE;
      drop table MELETE_MODULE;
      drop table MELETE_MODULE_LICENSE;
      

• (02:10 James) Apply update scripts to Sakai internal DB on caje:
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_1_2-2_2_0_oracle_conversion.sql
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_2_0-2_2_1_oracle_conversion.sql
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_2_1-2_3_0_oracle_conversion.sql
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_3_0-2_3_1_oracle_conversion.sql
  • We still need to figure out how we handle the DB constraint errors
  • Add additional Samigo sam_media_t conversion (thomas)
  • Add script to remove hibernate auto.ddl errors (jon)
• (02:30 Thomas/Emily) Perform maintenance on the SmartSite AFS volumes
  • Backup all existing melete entries
  • Melete volumes are:
    • /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete
    • /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs
    • /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads

    • mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete.bak
      mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs.bak
      mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads.bak
      cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete.bak/
      cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs.bak/
      cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads.bak/
      

  • Delete the content in melete volumes : We only do this if we choose to export/import all Melete modules

    • rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete/*
      rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs/*
      rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads/*
      

• (03:15 Thomas/Emily) Backup all configuration files on production staging area
  • /afs/.ucdavis.edu/app/sakai/staging/config/*

    • mkdir /afs/.ucdavis.edu/app/sakai/staging/config/config-2-1-x.bak
      cp /afs/.ucdavis.edu/app/sakai/staging/config/* /afs/.ucdavis.edu/app/sakai/staging/config/config-2-1-x.bak/
      

• (03:20 Thomas/Emily) Start process of preparing Sakai v2.3.x source on the head node (stubing).
  • Getting the source code:

    • cd ~/src
      svn co https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-3-x-prod-001
      

  • Setting up the build environment:
  • ~/build.properties

    • maven.repo.remote=http://source.sakaiproject.org/maven/,http://svn.ucdavis.edu/maven/,http://www.ibiblio.org/maven2/
      maven.tomcat.home=/ucd/opt/sakai/tomcat-staging/
      maven.compile.source = 1.5
      maven.compile.target = 1.5
      

  • ~/.bash_profile

    • PATH=/ucd/opt/java5/bin:/ucd/opt/maven/bin:$PATH
      MAVEN_HOME=/ucd/opt/maven
      CATALINA_HOME=/ucd/pkg/apache-tomcat
      JAVA_HOME=/ucd/opt/java5
      JAVA_OPTS=" -d64 -Xmx6144m -Xms6144m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseAdaptiveSizePolicy -XX:NewSize=512m -XX:MaxPermSize=200m -XX:PermSize=64m -XX:MaxTenuringThreshold=3 -XX:CMSInitiatingOccupancyFraction=75 -XX:MinHeapFreeRatio=18 -Dsakai.home=/ucd/opt/sakai/sakai-home/ -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
      MAVEN_OPTS=" -d64 -Xmx1024m -Xms1024m"
      export PATH MAVEN_HOME CATALINA_HOME JAVA_HOME JAVA_OPTS MAVEN_OPTS
      

  • Delete old tomcat content

    • rm -fr /ucd/pkg/apache-tomcat/*	
      

  • Delete old tomcat staging build

    • cd ~/bin
      ./tomcatclean.sh
      

  • Build the source code

    • cd ~/src/sakai_2-3-x-prod-001
      maven sakai
      

  • Copy staged tomcat

    • cd /ucd/pkg/apache-tomcat/
      cp -R /afs/.ucdavis.edu/app/sakai/staging/tomcat/apache-tomcat-5.5.17/* .
      

• (03:45 Thomas/Emily) Follow specific Melete installation instructions for the v2.3 release
  • Copy content

    • cp -R ~/src/sakai_2-3-x-prod-001/melete/var/melete/* /var/melete/
      ls -la /var/melete
      

    • After executing "ls -la /var/melete, we should see a directory called "packagefiles" in /var/melete
  • Copy meleteDocs.xml

    • cp ~/src/sakai_2-3-x-prod-001/melete/meleteDocs.xml /ucd/pkg/apache-tomcat/conf/Catalina/localhost/	
      

  • meleteDocs.xml should look like this:

    •  <Context path="/meleteDocs" docBase="/var/melete/meleteDocs">
              
        <Logger className="org.apache.catalina.logger.FileLogger"
                prefix="localhost_admin_log." suffix=".txt"
                timestamp="true"/>          
      </Context>
      

• (03:55 Thomas/Emily) Making sure all the configuration files are in place for v2.3.x such as sakai.properties etc.
  • Configure JAVA_OPTS to allow JMX access via JConsole
  • Synchronize all configuration files on the head node (stubing) with the AFS configuration stating area
    • /afs/.ucdavis.edu/app/sakai/staging/config
  • Start-up head node (stubing) and check catalina.out for any errors
    • Make sure that "auto.ddl=true" for first time startup on head node
    • Check catalina.out for DB errors
  • Once the head node started up without any errors, shut it down again
  • Start-up head node (stubing) again but this time "auto.ddl=false"
    • Make sure that there are no errors in catalina.out
  • Access head node and perform any appropriate testing
  • If there are no errors/problems, perform installation on second node (walton)
  • Start-up walton and perform any appropriate testing
• (04:30 Kirk, Sandra, ...) Release SmartSite for initial testing to determine "successful upgrade"
  • Access it via:
    • https://smartsite.ucdavis.edu:8443/portal
      • https://stubing.ucdavis.edu:8443/portal
      • https://walton.ucdavis.edu:8443/portal
• (07:00 Joncarlo) Adjust iptables on all nodes (stubing, walton) to give public access again
• (07:05) System ready for public access

Fallback Procedure (If needed)

• In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
• (08:00 Thomas/Emily) Shutdown all application servers
• (08:10 Thomas/Emily) Restore deleted Melete AFS volumes
• (08:10 Prabhu) Restore Sakai internal production database on caje
• (08:40 Thomas/Emily) Install latest sakai-core-2-1-x-prod snapshot on all application nodes (stubing, walton)
  • Restore v2.1.x configuration files
• (09:30 Thomas/Emily) Start nodes and test
• (09:40 Kirk, Sandra, ...) Start testing
• (10:00 Joncarlo) Adjust iptables
• (10:05) System ready for public access

Post Upgrade TODOs:

• SAK-377, SAK-396 : Change copy right message in site "SmartSite-Resources"
  • Change resource "Sakai Foundation Copyright Notice"
    • from:

      Copyright 2003, 2004, 2005 The Sakai Foundation.
      
      All rights reserved.
      
      Portions of Sakai are copyrighted by other parties as described in the Acknowledgments screen.
      

    • to:

      Copyright 2003 - 2006 The Sakai Foundation.
      
      All rights reserved.
      
      Portions of Sakai are copyrighted by other parties as described in the Acknowledgments screen.
      

Things that are obvious but need to be said:

– We're not coming in through Bastion Hosts for this upgrade. Most of us do not yet have tokens to access the Bastion Hosts.
– If our remote access goes down or is not stable, we'll physically come into Chiles Road (application development people) and the Data Center (DBA, SysAdmin).
– If our network or other infrastructure support goes down, the upgrade will extend beyond 8am. Sandra and Kirk will contact Operations to learn the prognosis for restoring access and make decisions on how and when to proceed.