Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Current »

IRC Channel for Communication

Maintenance Window

  • Date: August 18, 2007
  • Time: 08:00 to 12:00 (noon)

Upgrade Process

  • (08:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
    • We need to allow access from dedicated IPs for possible pre-upgrade work, such as exporting Melete modules, etc.
    • Adjust iptables on head node (stubing) to allow JMX access to port 9000 for specified IPs
      • PORT 8443:
        • 169.237.11.246 (thomas)
        • 69.12.224.247 (thomas)
        • 169.237.97.91 (zoidberg: vpn, terminal server)
        • 169.237.97.133 (kirk)
        • 169.237.11.220 (james)
        • 169.237.11.23 (sandra)
        • 169.237.11.6 (jon)
        • 169.237.11.242 (mike)
        • 169.237.11.30 (lisa)
      • PORT 9000:
        • IPs : 169.237.11.246
  • (08:10 Prabhu) Backing up (CLONE) production database so that it can be fully restored.
  • (09:00 Prabhu) Run database upgrade scripts in the following order
    1. SAK-869.sql
    2. SAK-878.sql
  • (09:15 Thomas) Backup all configuration files in production staging area
    • /afs/.ucdavis.edu/app/sakai/staging/config/* 
      • mkdir /afs/.ucdavis.edu/app/sakai/staging/config/config-2-4-0.bak
cp /afs/.ucdavis.edu/app/sakai/staging/config/* /afs/.ucdavis.edu/app/sakai/staging/config/config-2-4-0.bak/

  • (09:20 Thomas) Start process of preparing Sakai v2.4.1 source on the head node (stubing).
    • Getting the source code: 
      • cd ~/src
svn co https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-4-x-prod-003

    • Cleaning, building, deploying source on head node
  • (09:35 Thomas) Making sure all the configuration files are in place for v2.4.1 such as sakai.properties etc.
    • Configure JAVA_OPTS to allow JMX access via JConsole
    • Synchronize all configuration files on the head node (stubing) with the AFS configuration staging area
      • /afs/.ucdavis.edu/app/sakai/staging/config
    • Edit sakai.properties on stubing 
      • auto.ddl=true

    • Start-up head node (stubing) and check catalina.out for any errors
      • Check catalina.out for DB errors. (warning) There is one known issue where hibernate auto.ddl creates the following warning: 
        • WARN: Unsuccessful schema statement: alter table GB_LETTERGRADE_MAPPING add LG_MAPPING_ID  number(19,0) (2007-08-07 16:07:57,962
main_org.sakaiproject.springframework.orm.hibernate.AddableSessionFactoryBean) java.sql.SQLException: ORA-01430: column being added already exists in table
...
      • Once the head node started up without any errors, shut it down again
    • Edit sakai.properties on stubing 
      • auto.ddl=false
    • Start-up head node (stubing) again
      • Make sure that there are no errors in catalina.out
    • Access head node and perform any appropriate testing
    • If there are no errors/problems, perform installation on second node (walton)
    • Start-up walton and perform any appropriate testing
  • (10:00 Kirk, Sandra, Lisa, ...) Release SmartSite for initial testing to determine "successful upgrade"
  • (12:00 Joncarlo) Adjust iptables on all nodes (stubing, walton) to allow public access
    • (warning) (11:45 Thomas) Send mail to Joncarlo if we are ready to switch iptables
  • (12:01) System ready for public access

(minus) Fallback Procedure (If needed)

  • In case anything goes wrong that cannot be fixed within the allocated maintenance window we perform the following tasks
  • (12:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
  • (12:01 Thomas) Shutdown all application servers
  • (12:10 Prabhu) Restore Sakai internal production database on caje
  • (12:40 Thomas) Install sakai-core-2-4-x-prod-002 snapshot on all application nodes (stubing, walton)
    • Restore v2.4.0 configuration files
  • (13:30 Thomas) Start nodes and test
  • (13:40 Kirk, Sandra, ...) Start testing
  • (14:00 Joncarlo) Adjust iptables
  • (14:01) System ready for public access

Post Upgrade TODOs:

  • ...
  • No labels