SmartSite Database Upgrade Process from Caje to RAC Single Node

IRC Channel for Communication

• server: irc.freenode.net
• channel: #ucdsakai
• Windows IRC client:
  • http://www.mirc.com/ (Information on the program)
  • Direct link to MIRC Download

Steps for DBA's

Maintenance Window

• Date: February 02, 2008
• Time: 06:00 to 09:00

Upgrade Process

• (06:00 Joncarlo) Prevent any client access to SmartSite via iptables and redirect clients to a maintenance/outage page
  • We need to allow access from dedicated IPs:
    • PORT 8443:
      • 169.237.11.246 (thomas)
      • 69.12.224.247 (thomas)
      • 169.237.97.91 (zoidberg: vpn, terminal server)
      • 169.237.97.133 (kirk)
      • 169.237.11.220 (james)
      • 169.237.11.23 (sandra)
      • 169.237.11.6 (jon)
      • 169.237.11.242 (mike)
      • 169.237.11.211 (DC)
      • 169.237.11.30 (DC)
      • 169.237.205.246 (DC)
    • PORT 9000:
      • IPs : 169.237.11.246
• (06:01 Thomas) Stopping sakai on smartsite production nodes (sakaiprod1, ..., sakaiprod6)
• (06:05 Prabhu)
• Prabhu's DB Migration Tasks
  • Backup Caje Sakai internal DB
  • Restore Caje Sakai internal DB on RAC single node steele
  • Change LDAP entry for ldap://oraldap1.ucdavis.edu:389/sakai and ldap://oraldap2.ucdavis.edu:389/sakai to return the connect string for steele instead of caje
  • local.properties JDBC connect string:

    • jdbc:oracle:thin:@ldap://oraldap1.ucdavis.edu:389/sakai,cn=OracleContext,dc=ucdavis,dc=edu
      ldap://oraldap2.ucdavis.edu:389/sakai,cn=OracleContext,dc=ucdavis,dc=edu
      

• (07:45 Thomas) Startup sakai on smartsite production nodes (sakaiprod1, ..., sakaiprod6)
  • ldap service name change in local.properties: '/sakai'
• (08:00 Kirk, Sandra, ...) Start testing/verifying SmartSite
  • Monitor catalina.out for errors on all application nodes during this step
• (08:45 Kirk, Sandra, ...) Decide if we go into production with RAC single node (steele)
• (09:00 DC Operations) Adjust iptables on all nodes (sakaiprod1, ..., sakaiprod6) to allow public access
  • From Joncarlo: Once maintenance has finished, please contact Operations (530-752-1566).
    You should only need to ask them to return the app. servers to normal
    status. They should be expecting this call and know to restore service
    as well as change the public status page. If they need help, our on-call
    person will be available.

Fallback Procedure (If needed)

• (08:45 Thomas)
  • If we decide to go back to caje, stop sakai on smartsite production nodes (sakaiprod1, ..., sakaiprod6)
  • Revert configuration file "local.properties" so that we point at caje
  • Startup sakai on smartsite production nodes (sakaiprod1, ..., sakaiprod6)
• (09:00 DC Operations) Adjust iptables on all nodes (sakaiprod1, ..., sakaiprod6) to allow public access
  • From Joncarlo: Once maintenance has finished, please contact Operations (530-752-1566).
    You should only need to ask them to return the app. servers to normal
    status. They should be expecting this call and know to restore service
    as well as change the public status page. If they need help, our on-call
    person will be available.

TODOs

• Steele needs to be monitored as a production DB starting on Saturday morning
• The School of Medicine is access caje's external DB, we need to inform them about the change so that they can access the new system as well
• Earl at ATS is accessing caje's external DB to populate Adobe Connect (Breeze) collaboration sessions. He will need access to the new system
• We also need to notify Ray Tai of Vet.Med. He access the EXTERNALSCHEMA.

Important Notes:

• We will keep caje running two weeks into the winter quarter. So that we can use it as a potential fall-back DB
  • (From Patrick) It is usual practice to shut down the original database (caje) or put the database in single-user (maintenance) mode to prevent any users from accidentally accessing the old data store. I recommend we add this step
    to the procedures.