Implementation Plan November 21, 2009

Owned by Mike Wenk
Last updated: Nov 21, 2009 by Thomas AmslerVersion comment

Personnel

  • Programmer is Mike Wenk
  • DBA is Prabhu
  • QC is Sandra

Preparation for production release

  • (Programmer) On local system: copy over sakai, gb2, and properties tags to non-release-candidate version 
    svn copy https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-5-x-prod-025-RC1 https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-5-x-prod-025 -m "NOJIRA : Cutting sakai_2-5-x-prod-025 from release candidate 1"
svn copy https://mware.ucdavis.edu/svn/ucd-sakai/sakai-properties/tags/sakai-properties-2-5-x-prod-025-RC1 https://mware.ucdavis.edu/svn/ucd-sakai/sakai-properties/tags/sakai-properties-2-5-x-prod-025 -m "NOJIRA : Cutting sakai-properties-2-5-x-prod-025 from release candidate 1"
  • (Programmer) On sakaiprod1: backup sakai.properties and toolOrder.xml, export new files, checkout tag, build 
    cd ~/config-staging
./tomcatclean.sh &
mv sakai.properties sakai.properties-prod-024
mv toolOrder.xml toolOrder.xml-prod-024
svn export https://mware.ucdavis.edu/svn/ucd-sakai/sakai-properties/tags/sakai-properties-2-5-x-prod-025/sakai.properties
svn export https://mware.ucdavis.edu/svn/ucd-sakai/sakai-properties/tags/sakai-properties-2-5-x-prod-025/toolOrder.xml
cd ~/src
rm -rf sakai_2-5-x-prod-023 &
svn co https://mware.ucdavis.edu/svn/ucd-sakai/sakai/tags/sakai_2-5-x-prod-025 > ~/co.log
^Z
bg
tail -f ~/co.log
[wait until tinyurl is checked out]
^C
cd sakai_2-5-x-prod-025
patches/scripts/apply_patches.pl all
mvn install sakai:deploy -Pwith-jsmath > ~/build.log &
tail -f ~/build.log
[wait until build successful]
^C

During maintenance window

S = maintenance window

  • (S + 0m Programmer) Put all application servers into maintenance mode and shutdown Sakai on each one
  • (S + 1m DBA) Run SQL script (https://mware.ucdavis.edu/svn/ucd-sakai/ucd-database/branches/sakai_2-5-x/mviews/ucd_cm_enrollmentstatus_mv.sql) to modify external DB
  • (S + 10m Programmer) Deploy sakai release tag via setup-sakai-node.sh to sakaiprod1 and restart sakai instance 
    ~/config-staging/stop.sh
jps
[verify that sakai instance is down]
~/config-staging/setup-sakai-node.sh
cd $CATALINA_HOME/logs
~/config-staging/start.sh ; tail -f catalina.out
[wait until server has started up successfully]
  • (S + 17m QC) Verification--QA will use two of the following cases:
    • ANT 001 A01-A23 SQ 2009
      Webb, Derek    dwebb    dwebb@ucdavis.edu
      ANT 120 001 FQ 2009
      Clark, Jonathan    jcdclark    jcdclark@ucdavis.edu
      ARE 112 001 FQ 2009
      Moghbel, Neda    nmoghbel    nmoghbel@ucdavis.edu
      CHE 008A 001 FQ 2009
      Gerken, Alison    aegerken    aegerken@ucdavis.edu
      pol 150 001 sq 2009
      Mckinnon, Tara    tcmckinnon    tcmckinnon@ucdavis.edu
      por 003 001 sq 2009
      Leonard, Michael    maleonard    maleonard@ucdavis.edu
  • (S + 27m Programmer) Deploy release tag via setup-sakai-node.sh to sakaiprod2 . . . sakaiprod6, and sakaiprodmail1.
  • (S + 37m Programmer) Start Sakaiprod2 . . . sakaiprod6, and sakaiprodmail1.
  • (S + 41m Programmer) Take smartsite-prod out of maintenance mode 
  • (S + 50m Sandra) Send Release Management email

Fall-back Procedure (If needed) Time D is the decision time made to fallback

  • (D + 0m DBA) No clue on rolling back Mview change
  • (D + 0m Programmer) Shutdown Sakai on smartsite-prod and clean out catalina staging 
    ~/config-staging/stop.sh
jps
[verify that sakai instance is down]
~/config-staging/tomcatclean.sh
  • (D + 2m Programmer) On sakaiprod1 remove sakai.properties from config-staging and rename sakai.properties.024 to sakai.properties 
    cd ~/config-staging
rm sakai.properties
rm toolOrder.xml
cp sakai.properties-prod-024 sakai.properties
cp toolOrder.xml-prod-024 toolOrder.xml
  • (D + 4m Programmer) Run mvn sakai:deploy from previous tag directory 
    cd ~/src/sakai_2-5-x-prod-024
mvn sakai:deploy
  • run previous revision of enrollmentstatus mview DDL 
    svn export -r7837 https://mware.ucdavis.edu/svn/ucd-sakai/ucd-database/branches/sakai_2-5-x/mviews/ucd_cm_enrollmentstatus_mv.sql
  • (D + 15m Programmer) Deploy previous tag on head node and restart sakai 
    ~/config-staging/setup-sakai-node.sh
cd $CATALINA_HOME/logs
~/config-staging/start.sh ; tail -f catalina.out
  • (D + 25m QC) Verify correct version of sakai is running on head node
  • (D + 40m Programmer) Run setup-sakai-node.sh and restart sakai on remaining nodes
  • (D + 50m Programmer) Take application servers out of maintenance mode