DRAFT
IRC Channel for Communication
- server: irc.freenode.net
- channel: #ucdsakai
- Windows IRC client:
- http://www.mirc.com/ (Information on the program)
- Direct link to MIRC Download
- If you have any questions or would like help getting configured, talk to Emily (emily@ucdavis.edu)
Maintenance Window
- Date: February 24, 2007
- Time: 00:00 to 08:00
Upgrade Process
- (00:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
- We need to allow access from dedicated IPs for possible pre-upgrade work, such as exporting Melete modules, etc.
- Adjust iptables on head node (stubing) to allow JMX access to port 9000 for specified IPs
- PORT 8443:
- 169.237.11.40 (thomas)
- 69.12.224.247 (thomas)
- 169.237.97.91 (zoidberg: vpn, terminal server)
- 169.237.97.133 (kirk)
- ...
- PORT 9000:
- IPs : 169.237.11.40
- PORT 8443:
- (00:10 James) Export all Melete modules
- (01:00 Thomas/Emily) Shutting down Sakai instances on all nodes (stubing, walton)
- (01:10 Prabhu) Backing up (CLONE) production database so that it can be fully restored.
- This backup/clone should maintain the names for all constraints, indexes, etc.
- Production SmartSite internal DB server is caje
- (02:00 Thomas/Emily) Perform maintenance on the SmartSite internal DB on caje
- Remove all existing melete tables : We only do this if we choose to export/import all Melete modules
- (02:10 James) Apply update scripts to Sakai internal DB on caje:
- https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_1_2-2_2_0_oracle_conversion.sql
- https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_2_0-2_2_1_oracle_conversion.sql
- https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_2_1-2_3_0_oracle_conversion.sql
- https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2_3_0-2_3_1_oracle_conversion.sql
- We still need to figure out how we handle the DB constraint errors
- Apply additional table creation scripts if we choose to create our tables via DDL scripts instead of using Hibernate's audo.ddl
- The tables in question are:
- UCDSAKAI_COURSE_REQUEST
- UCDSAKAI_META_DATA
- UCDSAKAI_PROJECT_REQUEST
- UCDSAKAI_SITE_REQUEST
- SAKAI_DYNAMIC_CONFIGURATION
- The tables in question are:
- (02:30 Thomas/Emily) Perform maintenance on the SmartSite AFS volumes
- Backup all existing melete entries
- Melete volumes are:
- /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete
- /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs
- /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads
mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete.bak mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs.bak mkdir /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads.bak cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete.bak/ cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs.bak/ cp -R /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads/* /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads.bak/
- Delete the content in melete volumes : We only do this if we choose to export/import all Melete modules
rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/melete/* rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/meleteDocs/* rm -fr /afs/.ucdavis.edu/app/sakai/misc/melete/smartsite/uploads/*
- (03:15 Thomas/Emily) Backup all configuration files on production staging area
- /afs/.ucdavis.edu/app/sakai/staging/config/*
mkdir /afs/.ucdavis.edu/app/sakai/staging/config/config-2-1-x.bak cp /afs/.ucdavis.edu/app/sakai/staging/config/* /afs/.ucdavis.edu/app/sakai/staging/config/config-2-1-x.bak/
- /afs/.ucdavis.edu/app/sakai/staging/config/*
- (03:20 Thomas/Emily) Start process of preparing Sakai v2.3.x source on the head node (stubing).
- Getting the source code:
cd ~/src svn co https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-3-x-prod-001
- Setting up the build environment:
- ~/build.properties
maven.repo.remote=http://source.sakaiproject.org/maven/,http://svn.ucdavis.edu/maven/,http://www.ibiblio.org/maven2/ maven.tomcat.home=/ucd/opt/sakai/tomcat-staging/ maven.compile.source = 1.5 maven.compile.target = 1.5
- ~/.bash_profile
PATH=/ucd/opt/java5/bin:/ucd/opt/maven/bin:$PATH MAVEN_HOME=/ucd/opt/maven CATALINA_HOME=/ucd/pkg/apache-tomcat JAVA_HOME=/ucd/opt/java5 JAVA_OPTS=" -d64 -Xmx6144m -Xms6144m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseAdaptiveSizePolicy -XX:NewSize=512m -XX:MaxPermSize=200m -XX:PermSize=64m -XX:MaxTenuringThreshold=3 -XX:CMSInitiatingOccupancyFraction=75 -XX:MinHeapFreeRatio=18 -Dsakai.home=/ucd/opt/sakai/sakai-home/ -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" MAVEN_OPTS=" -d64 -Xmx1024m -Xms1024m" export PATH MAVEN_HOME CATALINA_HOME JAVA_HOME JAVA_OPTS MAVEN_OPTS
- Delete old tomcat content
rm -fr /ucd/pkg/apache-tomcat/*
- Delete old tomcat staging build
cd ~/bin ./tomcatclean.sh
- Build the source code
cd ~/src/sakai_2-3-x-prod-001 maven sakai
- Copy staged tomcat
cd /ucd/pkg/apache-tomcat/ cp -R /afs/.ucdavis.edu/app/sakai/staging/tomcat/apache-tomcat-5.5.17/* .
- Getting the source code:
- (03:45 Thomas/Emily) Follow specific Melete installation instructions for the v2.3 release
- Copy content
cp -R ~/src/sakai_2-3-x-prod-001/melete/var/melete/* /var/melete/ ls -la /var/melete
- After executing "ls -la /var/melete, we should see a directory called "packagefiles" in /var/melete
- Copy meleteDocs.xml
cp ~/src/sakai_2-3-x-prod-001/melete/meleteDocs.xml /ucd/pkg/apache-tomcat/conf/Catalina/localhost/
- meleteDocs.xml should look like this:
<Context path="/meleteDocs" docBase="/var/melete/meleteDocs"> <Logger className="org.apache.catalina.logger.FileLogger" prefix="localhost_admin_log." suffix=".txt" timestamp="true"/> </Context>
- Copy content
- (03:55 Thomas/Emily) Making sure all the configuration files are in place for v2.3.x such as sakai.properties etc.
- Configure JAVA_OPTS to allow JMX access via JConsole
- Synchronize all configuration files on the head node (stubing) with the AFS configuration stating area
- /afs/.ucdavis.edu/app/sakai/staging/config
- Start-up head node (stubing) and check catalina.out for any errors
- Access head node and perform any appropriate testing
- If there are no errors/problems, perform installation on second node (walton)
- Start-up walton and perform any appropriate testing
- (04:30 Kirk, Sandra, ...) Release SmartSite for initial testing to determine "successful upgrade"
- (07:00 Joncarlo) Adjust iptables on all nodes (stubing, walton) to give public access again
- (07:05) System ready for public access
Fallback Procedure (If needed)
- In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
- (08:00 Thomas/Emily) Shutdown all application servers
- (08:10 Thomas/Emily) Restore deleted Melete AFS volumes
- (08:10 Prabhu) Restore Sakai internal production database on caje
- (08:40 Thomas/Emily) Install latest sakai-core-2-1-x-prod snapshot on all application nodes (stubing, walton)
- Restore v2.1.x configuration files
- (09:30 Thomas/Emily) Start nodes and test
- (09:40 Kirk, Sandra, ...) Start testing
- (10:00 Joncarlo) Adjust iptables
- (10:05) System ready for public access