Page Comparison

...

(00:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
- We need to allow access from dedicated IPs for possible pre-upgrade work, such as exporting Melete modules, etc.
- Adjust iptables on head node (stubing) to allow JMX access to port 9000 for specified IPs
  - PORT 8443:
    - 169.237.11.40 (thomas)
    - 69.12.224.247 (thomas)
    - 169.237.97.91 (zoidberg: vpn, terminal server)
    - 169.237.97.133 (kirk)
    - 169.237.11.220 (james)
    - 169.237.11.23 (sandra)
    - 169.237.11.211 (emily)
    - 169.237.11.6 (Jon)
    - ...
  - PORT 9000:
    - IPs : 169.237.11.40
(00:10 James) Backup the Melete Tables
- Script is https://mware.ucdavis.edu/svn/ucd-sakai/update/melete_backup_tables.sql
(00:15 James) Export all Melete modules
- Detailed Instructions on Melete Bulk Export
- On production login to any modules tool, go to Manage, go to Export/Import, click on Bulk Export
- File will be written to local machine
- File is about 60 67 MB and is called Monster.zip
- Shouldn't take longer than 10 20 minutes to export
- Prepare smartsite-test, cleaning out all the melete tables, so that we can do a test import Run melete table backup creation script
(james)(01:00 Thomas/Emily) Shutting down Sakai instances on all nodes (stubing, walton)
(01:10 Prabhu) Backing up (CLONE) production database so that it can be fully restored.
- This backup/clone should maintain the names for all constraints, indexes, etc.
- Production SmartSite internal DB server is caje

(02:00 Thomas/Emily) Perform maintenance on the SmartSite internal DB on caje

Remove all existing melete tables:

Code Block drop tableScript (including cascade constraints): https://mware.ucdavis.edu/svn/ucd-sakai/update/melete_drop_tables.sql

Code Block


drop table MELETE_CC_LICENSE;
drop table MELETE_COURSE_MODULE;
drop table MELETE_MODULE_SHDATES;
drop table MELETE_MODULE_STUDENT_PRIVS;
drop table MELETE_SECTION;
drop table MELETE_USER_PREFERENCE;
drop table MELETE_MODULE;
drop table MELETE_MODULE_LICENSE;

(02:10 James) Apply update scripts to Sakai internal DB on caje:

https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_1_2-2_2_0_oracle_conversion.sql
https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_2_0-2_2_1_oracle_conversion.sql
https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_2_1-2_3_0_oracle_conversion.sql
https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_3_0-2_3_1_oracle_conversion.sql
We still need to figure out how we handle the DB constraint errors

Add additional Script that corrects the Samigo sam_media_t conversion table media column data type (thomas)

Add script to remove

https://mware.ucdavis.edu/svn/ucd-sakai/update/sam_media_t-datatype-change.sql

Script that removes hibernate auto.ddl errors (jon)

https://mware.ucdavis.edu/svn/ucd-sakai/update/create_drop_all_hibernate_fk_script.sql

Code Block


- run the sql that is created by that script
- output from caje prod:
 alter table "CMN_AGENT_GROUP_ASSOC_T" drop constraint FKBE10ED353D1FCFC;         
 alter table "CMN_AGENT_GROUP_ASSOC_T" drop constraint FKBE10ED358C3DFCAA;        
 alter table "CMN_AGENT_GROUP_T" drop constraint FKD858333FA8D3F022;              
 alter table "CMN_AGENT_GROUP_T" drop constraint FKD858333FF7F2E3EA;              
 alter table "CMN_GRP_PRNT_MMBR_MAP_T" drop constraint FKEBB1654487557E9A;        
 alter table "CMN_GRP_PRNT_MMBR_MAP_T" drop constraint FKEBB165448C3DFCAA;        
 alter table "CMN_NODE_PARENT_CHILD_MAP_T" drop constraint FK614EA03B3D1FCFC;     
 alter table "CMN_NODE_PARENT_CHILD_MAP_T" drop constraint FK614EA03B8C3DFCAA;    
 alter table "CMN_NODE_T" drop constraint FK596CE0D2217F38D0;                     
 alter table "CMN_NODE_T" drop constraint FK596CE0D227873A;                       
 alter table "GB_GRADABLE_OBJECT_T" drop constraint FK759996A76AB1529A;           
 alter table "GB_GRADEBOOK_T" drop constraint FK7C870191F11270F8;                 
 alter table "GB_GRADE_MAP_T" drop constraint FKADE112256AB1529A;                 
 alter table "GB_GRADE_RECORD_T" drop constraint FK46ACF752B5399B44;              
 alter table "GB_GRADE_TO_PERCENT_MAPPING_T" drop constraint FKCDEA021164995486;  
 alter table "GB_GRADING_EVENT_T" drop constraint FK4C9D99E0B5399B44;             
 alter table "MELETE_COURSE_MODULE" drop constraint FK5FE498D34995ABCE;           
 alter table "MELETE_MODULE" drop constraint FK54E9B209EBF83B4B;                  
 alter table "MELETE_MODULE_SHDATES" drop constraint FK2EA2E51A4995ABCE;          
 alter table "MELETE_MODULE_STUDENT_PRIVS" drop constraint FK702F80EA4995ABCE;    
 alter table "MELETE_SECTION" drop constraint FK7492E6E84995ABCE;                 
 alter table "MFR_AP_ACCESSORS_T" drop constraint FKC8532ED796792399;             
 alter table "MFR_AP_ACCESSORS_T" drop constraint FKC8532ED721BCC7D2;             
 alter table "MFR_AP_CONTRIBUTORS_T" drop constraint FKA221A1F7737F309B;          
 alter table "MFR_AP_CONTRIBUTORS_T" drop constraint FKA221A1F721BCC7D2;          
 alter table "MFR_AP_MODERATORS_T" drop constraint FK75B43C0D21BCC7D2;            
 alter table "MFR_AP_MODERATORS_T" drop constraint FK75B43C0DC49D71A5;            
 alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDE2AFBA652;               
 alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDEC6FDB1CF;               
 alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDE20D91C10;               
 alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDEFDEB22F9;               
 alter table "MFR_ATTACHMENT_T" drop constraint FK7B2D5CDEAD5AF852;               
 alter table "MFR_CONTROL_PERMISSIONS_T" drop constraint FKA07CF1D1E581B336;      
 alter table "MFR_CONTROL_PERMISSIONS_T" drop constraint FKA07CF1D151C89994;      
 alter table "MFR_CONTROL_PERMISSIONS_T" drop constraint FKA07CF1D117721828;      
 alter table "MFR_LABEL_T" drop constraint FKC6611543EA902104;                    
 alter table "MFR_LABEL_T" drop constraint FKC661154344B127B6;                    
 alter table "MFR_MEMBERSHIP_ITEM_T" drop constraint FKE03761CB6785AF85;          
 alter table "MFR_MEMBERSHIP_ITEM_T" drop constraint FKE03761CBC6FDB1CF;          
 alter table "MFR_MEMBERSHIP_ITEM_T" drop constraint FKE03761CB2AFBA652;          
 alter table "MFR_MEMBERSHIP_ITEM_T" drop constraint FKE03761CB925CE0F4;          
 alter table "MFR_MESSAGE_PERMISSIONS_T" drop constraint FK750F9AFB17721828;      
 alter table "MFR_MESSAGE_PERMISSIONS_T" drop constraint FK750F9AFB51C89994;      
 alter table "MFR_MESSAGE_PERMISSIONS_T" drop constraint FK750F9AFBE581B336;      
 alter table "MFR_MESSAGE_T" drop constraint FK80C1A316FE0789EA;                  
 alter table "MFR_MESSAGE_T" drop constraint FK80C1A3164FDCE067;                  
 alter table "MFR_OPEN_FORUM_T" drop constraint FKC17608474FDCE067;               
 alter table "MFR_OPEN_FORUM_T" drop constraint FKC1760847B88980FA;               
 alter table "MFR_PRIVATE_FORUM_T" drop constraint FKA9EE57544FDCE067;            
 alter table "MFR_PVT_MSG_USR_T" drop constraint FKC4DE0E14FA8620E;               
 alter table "MFR_TOPIC_T" drop constraint FK863DC0BEC6FDB1CF;                    
 alter table "MFR_TOPIC_T" drop constraint FK863DC0BE7AFA22C2;                    
 alter table "MFR_TOPIC_T" drop constraint FK863DC0BE20D91C10;                    
 alter table "SAKAI_SYLLABUS_ATTACH" drop constraint FK4BF41E45A09831E0;          
 alter table "SAKAI_SYLLABUS_DATA" drop constraint FK3BC123AA4FDCE067;            
 alter table "SAM_ANSWERFEEDBACK_T" drop constraint FK58CEF0D810DF4559;           
 alter table "SAM_ANSWER_T" drop constraint FKDD0580938152036E;                   
 alter table "SAM_ANSWER_T" drop constraint FKDD058093CBA347DB;                   
 alter table "SAM_ASSESSACCESSCONTROL_T" drop constraint FKC945448AC07F835D;      
 alter table "SAM_ASSESSEVALUATION_T" drop constraint FK6A6F29F5C07F835D;         
 alter table "SAM_ASSESSFEEDBACK_T" drop constraint FK557D4CFEC07F835D;           
 alter table "SAM_ASSESSMENTGRADING_T" drop constraint FKDAED4C879E4AF02B;        
 alter table "SAM_ASSESSMETADATA_T" drop constraint FK7E6F9A28C07F835D;           
 alter table "SAM_GRADINGSUMMARY_T" drop constraint FKBC88AA279E4AF02B;           
 alter table "SAM_ITEMFEEDBACK_T" drop constraint FK3254E9ED8152036E;             
 alter table "SAM_ITEMGRADING_T" drop constraint FKB68E6756E5D3D24D;              
 alter table "SAM_ITEMMETADATA_T" drop constraint FK5B4737178152036E;             
 alter table "SAM_ITEMTEXT_T" drop constraint FK271D63158152036E;                 
 alter table "SAM_ITEM_T" drop constraint FK3AAC5EA87DA376A0;                     
 alter table "SAM_MEDIA_T" drop constraint FKD4CF5A1971254D1C;                    
 alter table "SAM_PUBLISHEDACCESSCONTROL_T" drop constraint FK2EDF39E0C07F835D;   
 alter table "SAM_PUBLISHEDANSWERFEEDBACK_T" drop constraint FK6CB765A610DF4559;  
 alter table "SAM_PUBLISHEDANSWER_T" drop constraint FKB41EA361CBA347DB;          
 alter table "SAM_PUBLISHEDANSWER_T" drop constraint FKB41EA3618152036E;          
 alter table "SAM_PUBLISHEDEVALUATION_T" drop constraint FK94CB245FC07F835D;      
 alter table "SAM_PUBLISHEDFEEDBACK_T" drop constraint FK1488D9E8C07F835D;        
 alter table "SAM_PUBLISHEDITEMFEEDBACK_T" drop constraint FKB7D03A3B8152036E;    
 alter table "SAM_PUBLISHEDITEMMETADATA_T" drop constraint FKE0C287658152036E;    
 alter table "SAM_PUBLISHEDITEMTEXT_T" drop constraint FK9C790A638152036E;        
 alter table "SAM_PUBLISHEDITEM_T" drop constraint FK53ABDCF67DA376A0;            
 alter table "SAM_PUBLISHEDMETADATA_T" drop constraint FK3D7B2712C07F835D;        
 alter table "SAM_PUBLISHEDSECTIONMETADATA_T" drop constraint FKDF50FC3B7DA376A0; 
 alter table "SAM_PUBLISHEDSECTION_T" drop constraint FK424F87CCC07F835D;         
 alter table "SAM_PUBLISHEDSECUREDIP_T" drop constraint FK1EDEA25BC07F835D;       
 alter table "SAM_SECTIONMETADATA_T" drop constraint FK762AD7497DA376A0;          
 alter table "SAM_SECTION_T" drop constraint FK364450DAC07F835D;                  
 alter table "SAM_SECUREDIP_T" drop constraint FKE8C55FE9C07F835D;

(02:30 Thomas/Emily) Perform maintenance on the SmartSite AFS volumes

Backup all existing melete entries

Melete volumes are:

/afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete
/afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs
/afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads

Code Block

mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete.bak
mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs.bak
mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads.bak
cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete.bak/
cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs.bak/
cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads.bak/

Delete the content in melete volumes:

Code Block

rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete/*
rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs/*
rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads/*

(03:15 Thomas/Emily) Backup all configuration files on production staging area

/afs/.ucdavis.edu/app/sakai/staging/config/*

Code Block
mkdir /afs/.ucdavis.edu/app/sakai/staging/config/config-2-1-x.bak cp /afs/.ucdavis.edu/app/sakai/staging/config/* /afs/.ucdavis.edu/app/sakai/staging/config/config-2-1-x.bak/

(03:20 Thomas/Emily) Start process of preparing Sakai v2.3.x source on the head node (stubing).

Getting the source code:

Code Block
cd ~/src svn co https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-3-x-prod-001

Setting up the build environment:

~/build.properties

Code Block

maven.repo.remote=http://source.sakaiproject.org/maven/,http://svn.ucdavis.edu/maven/,http://www.ibiblio.org/maven2/
maven.tomcat.home=/ucd/opt/sakai/tomcat-staging/
maven.compile.source = 1.5
maven.compile.target = 1.5

~/.bash_profile

Code Block

PATH=/ucd/opt/java5/bin:/ucd/opt/maven/bin:$PATH
MAVEN_HOME=/ucd/opt/maven
CATALINA_HOME=/ucd/pkg/apache-tomcat
JAVA_HOME=/ucd/opt/java5
JAVA_OPTS=" -d64 -Xmx6144m -Xms6144m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseAdaptiveSizePolicy -XX:NewSize=512m -XX:MaxPermSize=200m -XX:PermSize=64m -XX:MaxTenuringThreshold=3 -XX:CMSInitiatingOccupancyFraction=75 -XX:MinHeapFreeRatio=18 -Dsakai.home=/ucd/opt/sakai/sakai-home/ -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
MAVEN_OPTS=" -d64 -Xmx1024m -Xms1024m"
export PATH MAVEN_HOME CATALINA_HOME JAVA_HOME JAVA_OPTS MAVEN_OPTS

Delete old tomcat content
- Code Block
  rm -fr /ucd/pkg/apache-tomcat/*
Delete old tomcat staging build
- Code Block
  cd ~/bin ./tomcatclean.sh

Build the source code

Code Block
cd ~/src/sakai_2-3-x-prod-001 maven sakai

Copy staged tomcat

Code Block
cd /ucd/pkg/apache-tomcat/ cp -R /afs/.ucdavis.edu/app/sakai/staging/tomcat/apache-tomcat-5.5.17/* .

(03:45 Thomas/Emily) Follow specific Melete installation instructions for the v2.3 release

Copy content
- Code Block
  cp -R ~/src/sakai_2-3-x-prod-001/melete/var/melete/* /var/melete/ ls -la /var/melete
- After executing "ls -la /var/melete, we should see a directory called "packagefiles" in /var/melete

Copy meleteDocs.xml

Code Block
cp ~/src/sakai_2-3-x-prod-001/melete/meleteDocs.xml /ucd/pkg/apache-tomcat/conf/Catalina/localhost/

meleteDocs.xml should look like this:

Code Block
<Context path="/meleteDocs" docBase="/var/meleteDocs"> <Logger className="org.apache.catalina.logger.FileLogger" prefix="localhost_admin_log." suffix=".txt" timestamp="true"/> </Context>

(03:55 Thomas/Emily) Making sure all the configuration files are in place for v2.3.x such as sakai.properties etc.
- Configure JAVA_OPTS to allow JMX access via JConsole
- Synchronize all configuration files on the head node (stubing) with the AFS configuration stating area
  - /afs/.ucdavis.edu/app/sakai/staging/config
- Start-up head node (stubing) and check catalina.out for any errors
  - Make sure that "auto.ddl=true" for first time startup on head node
  - Check catalina.out for DB errors
- Once the head node started up without any errors, shut it down again
- Start-up head node (stubing) again but this time "auto.ddl=false"
  - Make sure that there are no errors in catalina.out
- Access head node and perform any appropriate testing
- If there are no errors/problems, perform installation on second node (walton)
- Start-up walton and perform any appropriate testing
(04:15 James) Bulk import monster.zip
- Detailed Instructions on Melete Bulk Import
- Do some checking if everything got imported
(04:45 Kirk, Sandra, ...) Release SmartSite for initial testing to determine "successful upgrade"
- Access it via:
  - https://smartsite.ucdavis.edu:8443/portal
    - https://stubing.ucdavis.edu:8443/portal
    - https://walton.ucdavis.edu:8443/portal
(07:00 Joncarlo) Adjust iptables on all nodes (stubing, walton) to give public access again
(07:05) System ready for public access

...

In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
(08:00 Thomas/Emily) Shutdown all application servers
(08:10 Thomas/Emily) Restore deleted Melete AFS volumes
(08:10 Prabhu) Restore Sakai internal production database on caje
(08:40 Thomas/Emily) Install latest sakai-core-2-1-x-prod snapshot on all application nodes (stubing, walton)
- Restore v2.1.x configuration files
(09:30 Thomas/Emily) Start nodes and test
(09:40 Kirk, Sandra, ...) Start testing
(10:00 Joncarlo) Adjust iptables
(10:05) System ready for public access

Post Upgrade TODOs:

SAK-377, SAK-396 : Change copy right message in site "SmartSite-Resources"

Change resource "Sakai Foundation Copyright Notice"

from:

Code Block
Copyright 2003, 2004, 2005 The Sakai Foundation. All rights reserved. Portions of Sakai are copyrighted by other parties as described in the Acknowledgments screen.

to:

Code Block
Copyright 2003 - 2006 The Sakai Foundation. All rights reserved. Portions of Sakai are copyrighted by other parties as described in the Acknowledgments screen.

Things that are obvious but need to be said:

...

Versions Compared

Old Version 47

New Version Current

Key

Post Upgrade TODOs:

Things that are obvious but need to be said: