Sakai Upgrade Process v2.4.0 to v2.4.1
IRC Channel for Communication
- server: irc.freenode.net
- channel: #ucdsakai
- Windows IRC client:
- http://www.mirc.com/ (Information on the program)
- Direct link to MIRC Download
Maintenance Window
- Date: August 18, 2007
- Time: 08:00 to 12:00 (noon)
Upgrade Process
- (08:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
- We need to allow access from dedicated IPs for possible pre-upgrade work, such as exporting Melete modules, etc.
- Adjust iptables on head node (stubing) to allow JMX access to port 9000 for specified IPs
- PORT 8443:
- 169.237.11.246 (thomas)
- 69.12.224.247 (thomas)
- 169.237.97.91 (zoidberg: vpn, terminal server)
- 169.237.97.133 (kirk)
- 169.237.11.220 (james)
- 169.237.11.23 (sandra)
- 169.237.11.6 (jon)
- 169.237.11.242 (mike)
- 169.237.11.30 (lisa)
- PORT 9000:
- IPs : 169.237.11.246
- PORT 8443:
- (08:10 Prabhu) Backing up (CLONE) production database so that it can be fully restored.
- (09:00 Prabhu) Run database upgrade scripts in the following order
- (09:15 Thomas) Backup all configuration files in production staging area
- /afs/.ucdavis.edu/app/sakai/staging/config/*
mkdir /afs/.ucdavis.edu/app/sakai/staging/config/config-2-4-0.bak cp /afs/.ucdavis.edu/app/sakai/staging/config/* /afs/.ucdavis.edu/app/sakai/staging/config/config-2-4-0.bak/
- /afs/.ucdavis.edu/app/sakai/staging/config/*
- (09:20 Thomas) Start process of preparing Sakai v2.4.1 source on the head node (stubing).
- Getting the source code:
cd ~/src svn co https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-4-x-prod-003
- Cleaning, building, deploying source on head node
- Getting the source code:
- (09:35 Thomas) Making sure all the configuration files are in place for v2.4.1 such as sakai.properties etc.
- Configure JAVA_OPTS to allow JMX access via JConsole
- Synchronize all configuration files on the head node (stubing) with the AFS configuration staging area
- /afs/.ucdavis.edu/app/sakai/staging/config
- Edit sakai.properties on stubing
auto.ddl=true
- Start-up head node (stubing) and check catalina.out for any errors
- Check catalina.out for DB errors.
There is one known issue where hibernate auto.ddl creates the following warning:
WARN: Unsuccessful schema statement: alter table GB_LETTERGRADE_MAPPING add LG_MAPPING_ID number(19,0) (2007-08-07 16:07:57,962 main_org.sakaiproject.springframework.orm.hibernate.AddableSessionFactoryBean) java.sql.SQLException: ORA-01430: column being added already exists in table ...
- Once the head node started up without any errors, shut it down again
- Check catalina.out for DB errors.
- Edit sakai.properties on stubing
auto.ddl=false
- Start-up head node (stubing) again
- Make sure that there are no errors in catalina.out
- Access head node and perform any appropriate testing
- If there are no errors/problems, perform installation on second node (walton)
- Start-up walton and perform any appropriate testing
- (10:00 Kirk, Sandra, Lisa, ...) Release SmartSite for initial testing to determine "successful upgrade"
- (12:00 Joncarlo) Adjust iptables on all nodes (stubing, walton) to allow public access
- (11:45 Thomas) Send mail to Joncarlo if we are ready to switch iptables
- (12:01) System ready for public access
Fallback Procedure (If needed)
- In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
- (12:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
- (12:01 Thomas) Shutdown all application servers
- (12:10 Prabhu) Restore Sakai internal production database on caje
- (12:40 Thomas) Install sakai-core-2-4-x-prod-002 snapshot on all application nodes (stubing, walton)
- Restore v2.4.0 configuration files
- (13:30 Thomas) Start nodes and test
- (13:40 Kirk, Sandra, ...) Start testing
- (14:00 Joncarlo) Adjust iptables
- (14:01) System ready for public access
Post Upgrade TODOs:
- ...