Sandra Experimentations

Owned by Former user (Deleted)
Last updated: Apr 16, 2010Version comment
4 min read


Implementation Plan April 13, 2010

[Note, the fallback release is April 17, 2010]

Start time: 4:00am

End time: 7:00am

Personnel:

  • Kirk Alexander
  • Sandra Stewart
  • UC Davis programming team
  • rSmart programming team

Communication

  • server: irc.freenode.net
  • channel: #ucdsakai

During Maintenance Window

Pre Implementation tasks

(March 16, rSmart team) check out SAK-3163.patch for Gradebook 2 from our subversion patches director.

(March 16, rSmart team) build Release Candidate 1 on smartsite-test.rsmart.com using the SAK-3163 patch.

(March 16, rSmart team) add -Dsakai2.6 to the mvn command. This will tell gb2 to build using kernel but not edu-services.

(March 16, UCDavis team) attach code for duplicate sections fix patch (sak-3144) to RSN: 3986-8306816 ticket.      

(March 16, rSmart team) build patch for RSN: 3986-8306816 on smartsite-test.rsmart.com.

(March 22, UCDavis team) Check in SAK-3164.patch for Gradebook 2 into our subversion patches directory.

(March 22, rSmart team) check out Release Candidate 2 from our subversion.

(March 22, rSmart team) build Release Candidate 2, on smartsite-test.rsmart.com, for Gradebook 2.

(March 29, rSmart team) check out Final Release Candidate from our subversion.

(March 29, rSmart team) build Final Release Candidate on smartsite-test.rsmart.com for Gradebook 2.

[note: we may have several gradebook release candidates prior to the production release on April 13.]

Prepare Prod local.properties file for tag 2.6.2.13 wrt RSN-3986-8307456

During the Implementation rollout

  • Add database markers.
  • (3:45am, rSmart) Place SmartSite-prod into maintenance mode and stop ALL TomCats. #1
  • (3:45am to 5:00am, already scheduled run) Allow already scheduled back up of internal database to run.
  • (5:00am, rSmart) rSmart will
    • (5:00am to 5:02, rSmart) run the script for RSN-3986-8306969 to change the MOTD tool.  See bottom of doc for sql.  This SQL also exists on the production db server in /home/oracle/apr13.motd.sql AND run April 13th GB2 SQL in same location (see sql at end of doc)
    • (4:00am to 5:30, rSmart) for RSN-3986-8306769, change the number of qmail connection to 200 and recompile qmail.
    • (5:00a to 5:30am, rSmart) Deploy tag 2.6.2.13 on SmartSite-Prod.  Code will be compiled beforehand and just deployed at this point.  Total time is 15 minutes, but it can be started any time after 3:45am after tomcat is stopped.
    • (N/A, rSmart) Retrieve attachment from RSN: 3986-8306816 ticket and apply to SmartSite prod. This code is already in the 2.6.2.12 tag
    • (5:31am to     , rSmart) RSN-3986-8307016, repair tools that need OSIV per lazy loading issues: Lazy loading exception in WebContent tool
    • (                     , rSmart)  RSN-3986-8307456, Check out properties to Prod local.properties file.
    • (                      ,rSmart) RSN-3986-8307376, gradebook.editAssignments permission needs to be removed from TA Role.  This is database sql.
    • (5:30am to 5:35am, rSmart)  Bring up one node, still in maintenance mode, for QA verification.

 

Content and Function Verification Testing

  • (5:35am to 5:55***am, UCDavis) QA  will verify that the system has been successfully upgraded by doing the following:
    • Verify that we can search on a student name in Gradebook (on the spreadsheet side)
    • Verify that TA’s do not have permission to change category and item set up in Gradebook.
    • Verify that a new site with gradebook will open immediately and not require refreshing for it to open.
    • Verify that duplicate sections is fixed
  •  (5:55am--6:00am, rSmart and UCDavis) Team will decide go or roll back.

Vote on our readiness to bring the system back up

  • (5:55am – 6:00am, rSmart and UCDavis) Team to vote on readiness to bring the system back up assuming we’re going forward.  See rollback section for retreating.

 

Release the system to public

  • (6:00am---6:15am, rSmart) bring up all application servers.
  • (6:15am---6:20am, rSmart) will release application to public by reapplying normal mode.
  • (6:20am--6:23am, UCDavis)  QA/PM will send email notification to our SmartSite community to say the system is back up.  (6:23am--6:25am, UCDavis)   Project or Program Manager will change the MOTD.

 

Rollback

  • (6:00am---6:45am, rSmart) Restore the internal database.
    • Deploy prior code tag (2.6.2.10) to all 7 tomcats.
    • Bring up one node to test .
  • (6:45am---6:50am, UCDavis)  QA will log in and verify that the fix has been removed by doing the following:
    • Check database markers.
    • Check duplicate section issue.
    • Check gradebook menu and tree configuration.
  •  (6:50am---7:05am, rSmart) will bring up remaining nodes
  • (7:05am---7:10am, rSmart)  will release application to public
  • (7:10am---7:13am, UCDavis)  QA/PM will send email notification to our SmartSite community to say the system is back up
  • (7:13am---7:15am, UCDavis)   Project or Program Manager will change the MOTD. 


-- sql from 8306969

spool apr13.motd.output;



INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, 'false', 'show-subject' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '3', 'items' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '/announcement/channel/!site/motd', 'channel' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, 'true', 'show-newlines' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '50000', 'length' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '1000', 'days' FROM rsn_tool WHERE tool_id = 'sakai.motd';

spool off;



GB2 SQL:



spool apr06



select count(*) cnt from sakaitest.sakai_realm_rl_fn where function_key =

(select function_key

from sakaitest.sakai_realm_function

where

function_name = 'gradebook.editAssignments'

)

and

role_key in (select role_key from sakaitest.sakai_realm_role where

role_name='Teaching Assistant')



/

delete from sakaitest.sakai_realm_rl_fn where function_key =

(select function_key

from sakaitest.sakai_realm_function

where

function_name = 'gradebook.editAssignments'

)

and

role_key in (select role_key from sakaitest.sakai_realm_role where

role_name='Teaching Assistant')



/

commit

/

select count(*) cnt from sakaitest.sakai_realm_rl_fn where function_key =

(select function_key

from sakaitest.sakai_realm_function

where

function_name = 'gradebook.editAssignments'

)

and

role_key in (select role_key from sakaitest.sakai_realm_role where

role_name='Teaching Assistant')

/

spool off

/



#1

Placing Prod servers into Scheduled Maintenance mode & regular operating mode (called 'outage' mode)

    * Place Apache into scheduled maintenance mode when there is scheduled downtime as follows:


â—¦              as user ucdavis on mach execute /var/www/html/ucdavis/apache/maint_mode.sh


â—¦              as user 'root' on mach, edit /etc/httpd/conf/httpd_ucdavis.conf and uncomment the Proxy * stuff on lines 36-40


â—¦              restart apache: /etc/init.d/httpd_ucdavis restart

    * to restore regular functioning (normal "outage" mode is when there is no downtime planned, so any downtime is an "outage")


â—¦              as user ucdavis on mach execute /var/www/html/ucdavis/apache/outage_mode.sh


â—¦              as user root on mach, comment out Proxy * stuff & restart apache