Sakai Upgrade Process v2.1.x to v2.3.x

IRC Channel for Communication

• server: irc.freenode.net
• channel: #ucdsakai
• Windows IRC client:
  • http://www.mirc.com/ (Information on the program)
  • Direct link to MIRC Download
• If you have any questions or would like help getting configured, talk to Emily (emily@ucdavis.edu)

Maintenance Window

• Date: February 24, 2007
• Time: 00:00 to 08:00

Upgrade Process

• (00:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
  • We need to allow access from dedicated IPs for possible pre-upgrade work, such as exporting Melete modules, etc.
  • Adjust iptables on head node (stubing) to allow JMX access to port 9000 for specified IPs
    • PORT 8443:
      • 169.237.11.40 (thomas)
      • 69.12.224.247 (thomas)
      • 169.237.97.91 (zoidberg: vpn, terminal server)
      • 169.237.97.133 (kirk)
      • 169.237.11.220 (james)
      • 169.237.11.23 (sandra)
      • 169.237.11.211 (emily)
      • 169.237.11.6 (Jon)
      • ...
    • PORT 9000:
      • IPs : 169.237.11.40
• (00:10 James) Backup the Melete Tables
  • Script is https://mware.ucdavis.edu/svn/ucd-sakai/update/melete_backup_tables.sql
• (00:15 James) Export all Melete modules
  • Detailed Instructions on Melete Bulk Export
  • On production login to any modules tool, go to Manage, go to Export/Import, click on Bulk Export
  • File will be written to local machine
  • File is about 67 MB and is called Monster.zip
  • Shouldn't take longer than 20 minutes to export
  • Prepare smartsite-test, cleaning out all the melete tables, so that we can do a test import
• (01:00 Thomas/Emily) Shutting down Sakai instances on all nodes (stubing, walton)
• (01:10 Prabhu) Backing up (CLONE) production database so that it can be fully restored.
  • This backup/clone should maintain the names for all constraints, indexes, etc.
  • Production SmartSite internal DB server is caje
• (02:00 Thomas/Emily) Perform maintenance on the SmartSite internal DB on caje
  • Remove all existing melete tables:
    • Script (including cascade constraints): https://mware.ucdavis.edu/svn/ucd-sakai/update/melete_drop_tables.sql

    • drop table MELETE_CC_LICENSE;
      drop table MELETE_COURSE_MODULE;
      drop table MELETE_MODULE_SHDATES;
      drop table MELETE_MODULE_STUDENT_PRIVS;
      drop table MELETE_SECTION;
      drop table MELETE_USER_PREFERENCE;
      drop table MELETE_MODULE;
      drop table MELETE_MODULE_LICENSE;
      

• (02:10 James) Apply update scripts to Sakai internal DB on caje:
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_1_2-2_2_0_oracle_conversion.sql
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_2_0-2_2_1_oracle_conversion.sql
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_2_1-2_3_0_oracle_conversion.sql
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_3_0-2_3_1_oracle_conversion.sql
  • Script that corrects the Samigo sam_media_t table media column data type (thomas)
    • https://mware.ucdavis.edu/svn/ucd-sakai/update/sam_media_t-datatype-change.sql
  • Script that removes hibernate auto.ddl errors (jon)
    • https://mware.ucdavis.edu/svn/ucd-sakai/update/create_drop_all_hibernate_fk_script.sql

      - run the sql that is created by that script
      - output from caje prod:
       alter table "CMN_AGENT_GROUP_ASSOC_T" drop constraint FKBE10ED353D1FCFC;         
       alter table "CMN_AGENT_GROUP_ASSOC_T" drop constraint FKBE10ED358C3DFCAA;        
       alter table "CMN_AGENT_GROUP_T" drop constraint FKD858333FA8D3F022;              
       alter table "CMN_AGENT_GROUP_T" drop constraint FKD858333FF7F2E3EA;              
       alter table "CMN_GRP_PRNT_MMBR_MAP_T" drop constraint FKEBB1654487557E9A;        
       alter table "CMN_GRP_PRNT_MMBR_MAP_T" drop constraint FKEBB165448C3DFCAA;        
       alter table "CMN_NODE_PARENT_CHILD_MAP_T" drop constraint FK614EA03B3D1FCFC;     
       alter table "CMN_NODE_PARENT_CHILD_MAP_T" drop constraint FK614EA03B8C3DFCAA;    
       alter table "CMN_NODE_T" drop constraint FK596CE0D2217F38D0;                     
       alter table "CMN_NODE_T" drop constraint FK596CE0D227873A;                       
       alter table "GB_GRADABLE_OBJECT_T" drop constraint FK759996A76AB1529A;           
       alter table "GB_GRADEBOOK_T" drop constraint FK7C870191F11270F8;                 
       alter table "GB_GRADE_MAP_T" drop constraint FKADE112256AB1529A;                 
       alter table "GB_GRADE_RECORD_T" drop constraint FK46ACF752B5399B44;              
       alter table "GB_GRADE_TO_PERCENT_MAPPING_T" drop constraint FKCDEA021164995486;  
       alter table "GB_GRADING_EVENT_T" drop constraint FK4C9D99E0B5399B44;             
       alter table "MELETE_COURSE_MODULE" drop constraint FK5FE498D34995ABCE;           
       alter table "MELETE_MODULE" drop constraint FK54E9B209EBF83B4B;                  
       alter table "MELETE_MODULE_SHDATES" drop constraint FK2EA2E51A4995ABCE;          
       alter table "MELETE_MODULE_STUDENT_PRIVS" drop constraint FK702F80EA4995ABCE;    
       alter table "MELETE_SECTION" drop constraint FK7492E6E84995ABCE;                 
       alter table "MFR_AP_ACCESSORS_T" drop constraint FKC8532ED796792399;             
       alter table "MFR_AP_ACCESSORS_T" drop constraint FKC8532ED721BCC7D2;             
       alter table "MFR_AP_CONTRIBUTORS_T" drop constraint FKA221A1F7737F309B;          
       alter table "MFR_AP_CONTRIBUTORS_T" drop constraint FKA221A1F721BCC7D2;          
       alter table "MFR_AP_MODERATORS_T" drop constraint FK75B43C0D21BCC7D2;            
       alter table "MFR_AP_MODERATORS_T" drop constraint FK75B43C0DC49D71A5;            
       alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDE2AFBA652;               
       alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDEC6FDB1CF;               
       alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDE20D91C10;               
       alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDEFDEB22F9;               
       alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDEAD5AF852;               
       alter table "MFR_CONTROL_PERMISSIONS_T" drop constraint FKA07CF1D1E581B336;      
       alter table "MFR_CONTROL_PERMISSIONS_T" drop constraint FKA07CF1D151C89994;      
       alter table "MFR_CONTROL_PERMISSIONS_T" drop constraint FKA07CF1D117721828;      
       alter table "MFR_LABEL_T" drop constraint FKC6611543EA902104;                    
       alter table "MFR_LABEL_T" drop constraint FKC661154344B127B6;                    
       alter table "MFR_MEMBERSHIP_ITEM_T" drop constraint FKE03761CB6785AF85;          
       alter table "MFR_MEMBERSHIP_ITEM_T" drop constraint FKE03761CBC6FDB1CF;          
       alter table "MFR_MEMBERSHIP_ITEM_T" drop constraint FKE03761CB2AFBA652;          
       alter table "MFR_MEMBERSHIP_ITEM_T" drop constraint FKE03761CB925CE0F4;          
       alter table "MFR_MESSAGE_PERMISSIONS_T" drop constraint FK750F9AFB17721828;      
       alter table "MFR_MESSAGE_PERMISSIONS_T" drop constraint FK750F9AFB51C89994;      
       alter table "MFR_MESSAGE_PERMISSIONS_T" drop constraint FK750F9AFBE581B336;      
       alter table "MFR_MESSAGE_T" drop constraint FK80C1A316FE0789EA;                  
       alter table "MFR_MESSAGE_T" drop constraint FK80C1A3164FDCE067;                  
       alter table "MFR_OPEN_FORUM_T" drop constraint FKC17608474FDCE067;               
       alter table "MFR_OPEN_FORUM_T" drop constraint FKC1760847B88980FA;               
       alter table "MFR_PRIVATE_FORUM_T" drop constraint FKA9EE57544FDCE067;            
       alter table "MFR_PVT_MSG_USR_T" drop constraint FKC4DE0E14FA8620E;               
       alter table "MFR_TOPIC_T" drop constraint FK863DC0BEC6FDB1CF;                    
       alter table "MFR_TOPIC_T" drop constraint FK863DC0BE7AFA22C2;                    
       alter table "MFR_TOPIC_T" drop constraint FK863DC0BE20D91C10;                    
       alter table "SAKAI_SYLLABUS_ATTACH" drop constraint FK4BF41E45A09831E0;          
       alter table "SAKAI_SYLLABUS_DATA" drop constraint FK3BC123AA4FDCE067;            
       alter table "SAM_ANSWERFEEDBACK_T" drop constraint FK58CEF0D810DF4559;           
       alter table "SAM_ANSWER_T" drop constraint FKDD0580938152036E;                   
       alter table "SAM_ANSWER_T" drop constraint FKDD058093CBA347DB;                   
       alter table "SAM_ASSESSACCESSCONTROL_T" drop constraint FKC945448AC07F835D;      
       alter table "SAM_ASSESSEVALUATION_T" drop constraint FK6A6F29F5C07F835D;         
       alter table "SAM_ASSESSFEEDBACK_T" drop constraint FK557D4CFEC07F835D;           
       alter table "SAM_ASSESSMENTGRADING_T" drop constraint FKDAED4C879E4AF02B;        
       alter table "SAM_ASSESSMETADATA_T" drop constraint FK7E6F9A28C07F835D;           
       alter table "SAM_GRADINGSUMMARY_T" drop constraint FKBC88AA279E4AF02B;           
       alter table "SAM_ITEMFEEDBACK_T" drop constraint FK3254E9ED8152036E;             
       alter table "SAM_ITEMGRADING_T" drop constraint FKB68E6756E5D3D24D;              
       alter table "SAM_ITEMMETADATA_T" drop constraint FK5B4737178152036E;             
       alter table "SAM_ITEMTEXT_T" drop constraint FK271D63158152036E;                 
       alter table "SAM_ITEM_T" drop constraint FK3AAC5EA87DA376A0;                     
       alter table "SAM_MEDIA_T" drop constraint FKD4CF5A1971254D1C;                    
       alter table "SAM_PUBLISHEDACCESSCONTROL_T" drop constraint FK2EDF39E0C07F835D;   
       alter table "SAM_PUBLISHEDANSWERFEEDBACK_T" drop constraint FK6CB765A610DF4559;  
       alter table "SAM_PUBLISHEDANSWER_T" drop constraint FKB41EA361CBA347DB;          
       alter table "SAM_PUBLISHEDANSWER_T" drop constraint FKB41EA3618152036E;          
       alter table "SAM_PUBLISHEDEVALUATION_T" drop constraint FK94CB245FC07F835D;      
       alter table "SAM_PUBLISHEDFEEDBACK_T" drop constraint FK1488D9E8C07F835D;        
       alter table "SAM_PUBLISHEDITEMFEEDBACK_T" drop constraint FKB7D03A3B8152036E;    
       alter table "SAM_PUBLISHEDITEMMETADATA_T" drop constraint FKE0C287658152036E;    
       alter table "SAM_PUBLISHEDITEMTEXT_T" drop constraint FK9C790A638152036E;        
       alter table "SAM_PUBLISHEDITEM_T" drop constraint FK53ABDCF67DA376A0;            
       alter table "SAM_PUBLISHEDMETADATA_T" drop constraint FK3D7B2712C07F835D;        
       alter table "SAM_PUBLISHEDSECTIONMETADATA_T" drop constraint FKDF50FC3B7DA376A0; 
       alter table "SAM_PUBLISHEDSECTION_T" drop constraint FK424F87CCC07F835D;         
       alter table "SAM_PUBLISHEDSECUREDIP_T" drop constraint FK1EDEA25BC07F835D;       
       alter table "SAM_SECTIONMETADATA_T" drop constraint FK762AD7497DA376A0;          
       alter table "SAM_SECTION_T" drop constraint FK364450DAC07F835D;                  
       alter table "SAM_SECUREDIP_T" drop constraint FKE8C55FE9C07F835D; 
      

• (02:30 Thomas/Emily) Perform maintenance on the SmartSite AFS volumes
  • Backup all existing melete entries
  • Melete volumes are:
    • /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete
    • /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs
    • /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads

    • mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete.bak
      mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs.bak
      mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads.bak
      cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete.bak/
      cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs.bak/
      cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads.bak/
      

  • Delete the content in melete volumes:

    • rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete/*
      rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs/*
      rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads/*
      

• (03:15 Thomas/Emily) Backup all configuration files on production staging area
  • /afs/.ucdavis.edu/app/sakai/staging/config/*

    • mkdir /afs/.ucdavis.edu/app/sakai/staging/config/config-2-1-x.bak
      cp /afs/.ucdavis.edu/app/sakai/staging/config/* /afs/.ucdavis.edu/app/sakai/staging/config/config-2-1-x.bak/
      

• (03:20 Thomas/Emily) Start process of preparing Sakai v2.3.x source on the head node (stubing).
  • Getting the source code:

    • cd ~/src
      svn co https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-3-x-prod-001
      

  • Setting up the build environment:
  • ~/build.properties

    • maven.repo.remote=http://source.sakaiproject.org/maven/,http://svn.ucdavis.edu/maven/,http://www.ibiblio.org/maven2/
      maven.tomcat.home=/ucd/opt/sakai/tomcat-staging/
      maven.compile.source = 1.5
      maven.compile.target = 1.5
      

  • ~/.bash_profile

    • PATH=/ucd/opt/java5/bin:/ucd/opt/maven/bin:$PATH
      MAVEN_HOME=/ucd/opt/maven
      CATALINA_HOME=/ucd/pkg/apache-tomcat
      JAVA_HOME=/ucd/opt/java5
      JAVA_OPTS=" -d64 -Xmx6144m -Xms6144m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseAdaptiveSizePolicy -XX:NewSize=512m -XX:MaxPermSize=200m -XX:PermSize=64m -XX:MaxTenuringThreshold=3 -XX:CMSInitiatingOccupancyFraction=75 -XX:MinHeapFreeRatio=18 -Dsakai.home=/ucd/opt/sakai/sakai-home/ -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
      MAVEN_OPTS=" -d64 -Xmx1024m -Xms1024m"
      export PATH MAVEN_HOME CATALINA_HOME JAVA_HOME JAVA_OPTS MAVEN_OPTS
      

  • Delete old tomcat content

    • rm -fr /ucd/pkg/apache-tomcat/*	
      

  • Delete old tomcat staging build

    • cd ~/bin
      ./tomcatclean.sh
      

  • Build the source code

    • cd ~/src/sakai_2-3-x-prod-001
      maven sakai
      

  • Copy staged tomcat

    • cd /ucd/pkg/apache-tomcat/
      cp -R /afs/.ucdavis.edu/app/sakai/staging/tomcat/apache-tomcat-5.5.17/* .
      

• (03:45 Thomas/Emily) Follow specific Melete installation instructions for the v2.3 release
  • Copy content

    • cp -R ~/src/sakai_2-3-x-prod-001/melete/var/melete/* /var/melete/
      ls -la /var/melete
      

    • After executing "ls -la /var/melete, we should see a directory called "packagefiles" in /var/melete
  • Copy meleteDocs.xml

    • cp ~/src/sakai_2-3-x-prod-001/melete/meleteDocs.xml /ucd/pkg/apache-tomcat/conf/Catalina/localhost/	
      

  • meleteDocs.xml should look like this:

    •  <Context path="/meleteDocs" docBase="/var/meleteDocs">
              
        <Logger className="org.apache.catalina.logger.FileLogger"
                prefix="localhost_admin_log." suffix=".txt"
                timestamp="true"/>          
      </Context>
      

• (03:55 Thomas/Emily) Making sure all the configuration files are in place for v2.3.x such as sakai.properties etc.
  • Configure JAVA_OPTS to allow JMX access via JConsole
  • Synchronize all configuration files on the head node (stubing) with the AFS configuration stating area
    • /afs/.ucdavis.edu/app/sakai/staging/config
  • Start-up head node (stubing) and check catalina.out for any errors
    • Make sure that "auto.ddl=true" for first time startup on head node
    • Check catalina.out for DB errors
  • Once the head node started up without any errors, shut it down again
  • Start-up head node (stubing) again but this time "auto.ddl=false"
    • Make sure that there are no errors in catalina.out
  • Access head node and perform any appropriate testing
  • If there are no errors/problems, perform installation on second node (walton)
  • Start-up walton and perform any appropriate testing
• (04:15 James) Bulk import monster.zip
  • Detailed Instructions on Melete Bulk Import
  • Do some checking if everything got imported
• (04:45 Kirk, Sandra, ...) Release SmartSite for initial testing to determine "successful upgrade"
  • Access it via:
    • https://smartsite.ucdavis.edu:8443/portal
      • https://stubing.ucdavis.edu:8443/portal
      • https://walton.ucdavis.edu:8443/portal
• (07:00 Joncarlo) Adjust iptables on all nodes (stubing, walton) to give public access again
• (07:05) System ready for public access

Fallback Procedure (If needed)

• In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
• (08:00 Thomas/Emily) Shutdown all application servers
• (08:10 Thomas/Emily) Restore deleted Melete AFS volumes
• (08:10 Prabhu) Restore Sakai internal production database on caje
• (08:40 Thomas/Emily) Install latest sakai-core-2-1-x-prod snapshot on all application nodes (stubing, walton)
  • Restore v2.1.x configuration files
• (09:30 Thomas/Emily) Start nodes and test
• (09:40 Kirk, Sandra, ...) Start testing
• (10:00 Joncarlo) Adjust iptables
• (10:05) System ready for public access

Post Upgrade TODOs:

• SAK-377, SAK-396 : Change copy right message in site "SmartSite-Resources"
  • Change resource "Sakai Foundation Copyright Notice"
    • from:

      Copyright 2003, 2004, 2005 The Sakai Foundation.
      
      All rights reserved.
      
      Portions of Sakai are copyrighted by other parties as described in the Acknowledgments screen.
      

    • to:

      Copyright 2003 - 2006 The Sakai Foundation.
      
      All rights reserved.
      
      Portions of Sakai are copyrighted by other parties as described in the Acknowledgments screen.
      

Things that are obvious but need to be said:

– We're not coming in through Bastion Hosts for this upgrade. Most of us do not yet have tokens to access the Bastion Hosts.
– If our remote access goes down or is not stable, we'll physically come into Chiles Road (application development people) and the Data Center (DBA, SysAdmin).
– If our network or other infrastructure support goes down, the upgrade will extend beyond 8am. Sandra and Kirk will contact Operations to learn the prognosis for restoring access and make decisions on how and when to proceed.