...
- (08:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
- We need to allow access from dedicated IPs for possible pre-upgrade work, such as exporting Melete modules, etc.
- Adjust iptables on head node (stubing) to allow JMX access to port 9000 for specified IPs
- PORT 8443:
- 169.237.11.246 (thomas)
- 69.12.224.247 (thomas)
- 169.237.97.91 (zoidberg: vpn, terminal server)
- 169.237.97.133 (kirk)
- 169.237.11.220 (james)
- 169.237.11.23 (sandra)
- 169.237.11.211 (emily)
- 169.237.11.6 (jon)
- 169.237.11.242 (mike)
- 169.237.11.30 (lisa)
- 169.237.97.223 (pat)
- ...
- PORT 9000:
- IPs : 169.237.11.246
- PORT 8443:
- (08:10 Prabhu) Backing up (CLONE) production database so that it can be fully restored.
- (09:00 Prabhu) Run sakai_2_3_1-2_4_0_oracle_conversion.sql sakai_2_3_1-2_4_0_oracle_conversiondatabase upgrade scripts
- (0109:15 Thomas) Run SmartSite Extra Updates
- (01:30 Thomas) Run SAKAI_USER_ID_MAP conversion script
- (01:40 Thomas) Run SAKAI_REALM and SAKAI_REALM_PROVIDER conversion script
- (01:50 Thomas) Run SmartSite Extra Updates Part II
- (01:55 Thomas/Emily) Backup all configuration files in production staging area
- /afs/.ucdavis.edu/app/sakai/staging/config/*
Code Block mkdir /afs/.ucdavis.edu/app/sakai/staging/config/config-2-34-x0.bak cp /afs/.ucdavis.edu/app/sakai/staging/config/* /afs/.ucdavis.edu/app/sakai/staging/config/config-2-34-x0.bak/
- /afs/.ucdavis.edu/app/sakai/staging/config/*
- (0209:10 20 Thomas/Emily) Start process of preparing Sakai v2.4.x 1 source on the head node (stubing).
- Getting the source code:
Code Block cd ~/src svn co https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-4-x-prod-001
- Setting up the build environment: ~/build.properties
~/.bash_profileCode Block maven.repo.remote=http://source.sakaiproject.org/maven/,https://mware.ucdavis.edu/maven/,http://www.ibiblio.org/maven2/ maven.tomcat.home=/ucd/opt/sakai/tomcat-staging/ maven.compile.source = 1.5 maven.compile.target = 1.5
Delete old tomcat contentCode Block PATH=/ucd/opt/java5/bin:/ucd/opt/maven/bin:$PATH MAVEN_HOME=/ucd/opt/maven CATALINA_HOME=/ucd/pkg/apache-tomcat JAVA_HOME=/ucd/opt/java5 JAVA_OPTS=" -d64 -server -Xmx6144m -Xms6144m -XX:NewSize=512m -XX:MaxNewSize=512m -XX:PermSize=256m -XX:MaxPermSize=512m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dsakai.home=/ucd/opt/sakai/sakai-home/" CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9000 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false" MAVEN_OPTS="-Xmx1024m -Xms1024m" export PATH MAVEN_HOME CATALINA_HOME JAVA_HOME JAVA_OPTS CATALINA_OPTS MAVEN_OPTS
Delete old tomcat staging buildCode Block rm -fr /ucd/pkg/apache-tomcat/*
Build the source codeCode Block cd ~/bin ./tomcatclean.sh
Copy staged tomcatCode Block cd ~/src/sakai_2-4-x-prod-001 maven sakai
Code Block rm -fr /ucd/pkg/apache-tomcat/* cp -R /afs/.ucdavis.edu/app/sakai-test/staging/tomcat/apache-tomcat-5.5.23/* /ucd/pkg/apache-tomcat/
003
- Cleaning, building, deploying source on head node
- Getting the source code:
- (09:35 Thomas) Making sure all the configuration files are in place for v2.4.x 1 such as sakai.properties etc.
- Configure JAVA_OPTS to allow JMX access via JConsole
- Synchronize all configuration files on the head node (stubing) with the AFS configuration staging area
- /afs/.ucdavis.edu/app/sakai/staging/config
- Edit sakai.properties on stubing
Code Block performChatMigration@org.sakaiproject.chat2.model.impl.ChatDataMigration=true chatMigrationExecuteImmediate@org.sakaiproject.chat2.model.impl.ChatDataMigration=true auto.ddl=true
- Start-up head node (stubing) and check catalina.out for any errors
- This startup will take longer than the usual because of the chat conversion
- Check catalina.out for DB errors. There is one known issue where hibernate auto.ddl creates the following warning:
No Format WARN: Unsuccessful schema statement: alter table GB_LETTERGRADE_MAPPING add LG_MAPPING_ID number(19,0) (2007-08-07 16:07:57,962 main_org.sakaiproject.springframework.orm.hibernate.AddableSessionFactoryBean) java.sql.SQLException: ORA-01430: column being added already exists in table ...
- Once the head node started up without any errors and you see the message that the chat conversion has completed, shut it down again
- Check catalina.out for DB errors. There is one known issue where hibernate auto.ddl creates the following warning:
- Edit sakai.properties on stubing
Code Block #performChatMigration@org.sakaiproject.chat2.model.impl.ChatDataMigration=false #chatMigrationExecuteImmediate@org.sakaiproject.chat2.model.impl.ChatDataMigration=false auto.ddl=false
- Start-up head node (stubing) again
- Make sure that there are no errors in catalina.out
- Access head node and perform any appropriate testing
- If there are no errors/problems, perform installation on second node (walton)
- Start-up walton and perform any appropriate testing
- (0310:30 00 Kirk, Sandra, ...) Release SmartSite for initial testing to determine "successful upgrade"
- (0612:55 00 Joncarlo) Adjust iptables on all nodes (stubing, walton) to allow public access
- (0712:0001) System ready for public access
...
- In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
- (0812:00 Thomas/EmilyJoncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
- (12:01 Thomas) Shutdown all application servers
- (0812:10 Prabhu) Restore Sakai internal production database on caje
- (0812:40 Thomas/Emily) Install latest sakai-core-2-34-x-prod-002 snapshot on all application nodes (stubing, walton)
- Restore v2.34.x 0 configuration files
- (0913:30 Thomas/Emily) Start nodes and test
- (0913:40 Kirk, Sandra, ...) Start testing
- (1014:00 Joncarlo) Adjust iptables
- (1014:0501) System ready for public access
Post Upgrade TODOs:
- SAK-664 : Verify footer display, content and links are correct for the SmartSite 2.4 upgrade SAK-746 : Post-upgrade task for 2.4: Add site.add function to registered user role so that new users can create course and project sites SAK-751 : create CM role mapping for waitlisted students