Page Comparison

Personnel

...

• QC is Sandra
• DBA is Prabhu
• Sysadmin Joncarlo

Preparation for

...

Swith Over

• (Programmer) Verify that sakai.properties is set to the correct version and includes the following property:
  • gb2.gradable.role.names=Student,Participant,Open Campus,access
• (Programmer) Copy over sakai, gb2, and properties tags to non-release-candidate version
  • svn copy https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-5-x-prod-022-RC2 https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-5-x-prod-022
  • svn copy https://mware.ucdavis.edu/svn/ucd-sakai/sakai-properties/tags/sakai-properties-2-5-x-prod-022-RC2 [https://mware.ucdavis.edu/svn/ucd-sakai/sakai-properties/tags/sakai-properties-2-5-x-prod-022
    • I used the following copy command because the sakai-properties-2-5-x-prod-022-RC2 tag has a missing GB2 property
      • svn copy https://svn.ucdavis.edu/svn/ucd-sakai/sakai-properties/branches/sakai_2-5-x/smartsite-prod/sakai.properties https://svn.ucdavis.edu/svn/ucd-sakai/sakai-properties/tags/sakai-properties-2-5-x-prod-022
  • svn copy https://source.sakaiproject.org/contrib//gradebook2/tags/1.1.1-rc1/https://source.sakaiproject.org/contrib//gradebook2/tags/1.1.1
  • on sakaiprod1:~/config-staging do: cp sakai.properties and toolOrder.xml to sakai.properties-prod-021 and toolOrder.xml-prod-021, respectively
• (Programmer) Export sakai.properties and toolOrder.xml from https://mware.ucdavis.edu/svn/ucd-sakai/sakai-properties/tags/sakai-properties-2-5-x-prod-022 onto sakaiprod1 ~/config-staging
• (Programmer) Check out https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-5-x-prod-022 onto sakaiprod1 ~/src and build with patchesPrabhu) Copy all the cron jobs from Hammer to Steele.
• (Prabhu) Copy all the scripts from Hammer to Steele and edit the necessary parameters.
• (Prabhu) Verify with sysadmin about firewall rules, processes limits, Kernel parameters, and oracle limits are the same on hammer and steele.

During maintenance window

S = maintenance window + 30 mins 2 Hrs 

• (S - 15 min m ProjManmgt/Prabhu) Set database marker.Make sure database marker is set prior to start of the patching.
• (S - m Prabhu) Verify with sysadmin that patching of hammer is completed
• (S - 10 min Programmer) Put m Prabhu) Make sure all application servers into maintenance mode and shutdown Sakai on each oneare offline.
• (S + 5m m Prabhu) Backup the full database on hammer.
• (S + 35m m Prabhu) Apply DB scripts (These are both alter table commands)
  • SQL: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/SAK-2970.sql
  • SQL: https://mware.ucdavis.edu/svn/ucd-sakai/maintenance/db-scripts/GRBK-367.sql
• (S + 40m Programmer) Deploy sakai release tag via setup-sakai-node.sh to sakaiprod1.
• (S + 45m Programmer) Restart sakaiprod1 sakai instance
• (S + 50m QC) Verification
  • ?
• (S + 70m Programmer) Deploy release tag via setup-sakai-node.sh to sakaiprod2 . . . sakaiprod6, and sakaiprodmail1.
• (S + 80m Programmer) Start Sakaiprod2 . . . sakaiprod6, and sakaiprodmail1.
• (S + 85m Programmer) Take smartsite-prod out of maintenance mode 
• (S + 90m Sandra) Send Release Management email
• Make sure backup of full database on hammer is complete.
• (S + m Prabhu) Shutdown the standby database on Steele and Primary database on Hammer
• (S + m Prabhu) Inform Backupadmin/Sysadmin to start filesystem backup/snapshot on Hammer and Steele.
• (S + m Prabhu) Make sure filesystem snapshot is complete and Start the standby database on Steele and then Primary database on Hammer.
• (S + m Prabhu) Execute switchover procedure
• (S + m Sysadmin)  Restart sakai on head node
• (S + m Prabhu/Sysadmin) Verify the application login
• (S + m Sysadmin) Restart sakai on remaining nodes
• (S + m Sysadmin) Take application servers out of maintenance mode

Fall-back Procedure (If needed) Time D is the decision time made to fallback

• (D + 0m Programmerm Sysadmin) Shutdown Sakai on smartsite-prod
• (D + 2m m Prabhu )   Restore the following tables:
  • SAKAI_SITE_TOOL_PROPERTY
  • GB_GRADEBOOK_T
• (D + 2m Programmer) Remove sakai.properties from config-staging and rename sakai.properties.021 to sakai.properties
• (D + 4m Programmer) Run mvn sakai:deploy from previous tag directory
• (D + 15m Programmer) Deploy previous tag on head node and all other app servers via setup-sakai-node.sh
• (D + 20m Programmer) Execute Switch back procedure
• (D + m Sysadmin) Restart sakai on head node
• (D + 25m QCm Prabhu / Sysadmin) Verify the database marker is in placeapplication login
• (D + 40m Programmerm Sysadmin) Restart sakai on remaining nodes
• (D + 25m Programmerm Sysadmin) Take application servers out of maintenance mode