Start time: 3:45am
End time: 8:00am

Personnel:

• Kirk Alexander
• Sandra Stewart
• Pete Peterson
• David Woods
• UC Davis Programming Team
• rSmart Programming Team
• rSmart Support Team

Communication

• server: irc.freenode.net
• channel: #ucdsakai

Pre Implementation tasks

(March 16, rSmart team) check out SAK-3163.patch for Gradebook 2 from our subversion patches director.

(March 16, rSmart team) build Release Candidate 1 on smartsite-test.rsmart.com using the SAK-3163 patch.

(March 16, rSmart team) add -Dsakai2.6 to the mvn command. This will tell gb2 to build using kernel but not edu-services.

(March 16, UCDavis team) attach code for duplicate sections fix patch (sak-3144) to RSN: 3986-8306816 ticket.      

(March 16, rSmart team) build patch for RSN: 3986-8306816 on smartsite-test.rsmart.com.

(March 22, UCDavis team) Check in SAK-3164.patch for Gradebook 2 into our subversion patches directory.

(March 22, rSmart team) check out Release Candidate 2 from our subversion.

(March 22, rSmart team) build Release Candidate 2, on smartsite-test.rsmart.com, for Gradebook 2.

(March 29, rSmart team) check out Final Release Candidate from our subversion.

(March 29, rSmart team) build Final Release Candidate on smartsite-test.rsmart.com for Gradebook 2.

[note: we may have several gradebook release candidates prior to the production release on April 13.]

Prepare Prod local.properties file for tag 2.6.2.13 wrt RSN-3986-8307456

During the Implementation rollout

• Add database markers.
• (3:45am, rSmart) Place SmartSite-prod into maintenance mode and stop ALL TomCats. #1
• (3:45am to 5:00am, already scheduled run) Allow already scheduled back up of internal database to run.
• (5:00am to 5:30am, rSmart) rSmart will
  • (5:00am to 5:02, rSmart) run the script for RSN-3986-8306969 to change the MOTD tool.  See bottom of doc for sql.  This SQL also exists on the production db server in /home/oracle/apr13.motd.sql AND run April 13th GB2 SQL in same location (see sql at end of doc)
  • ( rSmart) for RSN-3986-8306769, change the number of qmail connection to 200 and recompile qmail.
  • (, rSmart) Deploy tag 2.6.2.13 on SmartSite-Prod.  Code will be compiled beforehand and just deployed at this point.  Total time is 15 minutes, but it can be started any time after 3:45am after tomcat is stopped.
  • (N/A, rSmart) Retrieve attachment from RSN: 3986-8306816 ticket and apply to SmartSite prod. This code is already in the 2.6.2.12 tag
  • (rSmart) RSN-3986-8307016, repair tools that need OSIV per lazy loading issues: Lazy loading exception in WebContent tool
  • ( , rSmart)  RSN-3986-8307456, Check out properties to Prod local.properties file.
  • ( ,rSmart) RSN-3986-8307376, gradebook.editAssignments permission needs to be removed from TA Role.  This is database sql.
  • (5:30am to 5:35am, rSmart)  Bring up one node, still in maintenance mode, for QA verification.

Content and Function Verification Testing

• (5:35am to 5:55***am, UCDavis) QA  will verify that the system has been successfully upgraded by doing the following:
  • Verify that we can search on a student name in Gradebook (on the spreadsheet side)
  • Verify that TA’s do not have permission to change category and item set up in Gradebook.  TA should not be able to change IOR's categories and item.  Should have only permissions IOR specifically grants.
  • Verify that a new site with gradebook will open immediately and not require refreshing for it to open.
  • Verify that duplicate sections is fixed
  • Verify that there are no lazy loading system errors.  rSmart can do this.
• (5:55am--6:00am, rSmart and UCDavis) Team will decide go or roll back.

Vote on our readiness to bring the system back up

• (5:55am – 6:00am, rSmart and UCDavis) Team to vote on readiness to bring the system back up assuming we’re going forward.  See rollback section for retreating.

Release the system to public

• (6:00am to 6:15am, rSmart) bring up all application servers.
• (6:15am to 6:20am, rSmart) will release application to public by reapplying normal mode.
• (6:20am to 6:23am, UCDavis)  QA/PM will send email notification to our SmartSite community to say
• (6:23am to 6:25am, UCDavis)   Project or Program Manager will change the MOTD.

Rollback

• (6:00am to 6:45am, rSmart) Restore the internal database.
  • Deploy prior code tag (2.6.2.10) to all 7 tomcats.
  • Bring up one node to test .
• (6:45am to 6:50am, UCDavis)  QA will log in and verify that the fix has been removed by doing the following:
  • Check database markers.
  • Check duplicate section issue.
  • Check gradebook menu and tree configuration.
• (6:50am to 7:05am, rSmart) will bring up remaining nodes
• (7:05am to 7:10am, rSmart)  will release application to public
• (7:10am to 7:13am, UCDavis)  QA/PM will send email notification to our SmartSite community to say the system is back up
• (7:13am to 7:15am, UCDavis)   Project or Program Manager will change the MOTD.

Placing Prod servers into Scheduled Maintenance mode & regular operating mode (called 'outage' mode)

• Apache into scheduled maintenance mode when there is scheduled downtime as follows:
• as user ucdavis on mach execute /var/www/html/ucdavis/apache/maint_mode.sh
• as user 'root' on mach, edit /etc/httpd/conf/httpd_ucdavis.conf and uncomment the Proxy * stuff on lines 36-40
• restart apache: /etc/init.d/httpd_ucdavis restart
• to restore regular functioning (normal "outage" mode is when there is no downtime planned, so any downtime is an "outage")
• as user ucdavis on mach execute /var/www/html/ucdavis/apache/outage_mode.sh
• as user root on mach, comment out Proxy * stuff & restart apache

-- sql from 8306969

spool apr13.motd.output;



INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, 'false', 'show-subject' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '3', 'items' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '/announcement/channel/!site/motd', 'channel' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, 'true', 'show-newlines' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '50000', 'length' FROM rsn_tool WHERE tool_id = 'sakai.motd';

INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '1000', 'days' FROM rsn_tool WHERE tool_id = 'sakai.motd';

spool off;



GB2 SQL:



spool apr06



select count(*) cnt from sakaitest.sakai_realm_rl_fn where function_key =

(select function_key

from sakaitest.sakai_realm_function

where

function_name = 'gradebook.editAssignments'

)

and

role_key in (select role_key from sakaitest.sakai_realm_role where

role_name='Teaching Assistant')



/

delete from sakaitest.sakai_realm_rl_fn where function_key =

(select function_key

from sakaitest.sakai_realm_function

where

function_name = 'gradebook.editAssignments'

)

and

role_key in (select role_key from sakaitest.sakai_realm_role where

role_name='Teaching Assistant')



/

commit

/

select count(*) cnt from sakaitest.sakai_realm_rl_fn where function_key =

(select function_key

from sakaitest.sakai_realm_function

where

function_name = 'gradebook.editAssignments'

)

and

role_key in (select role_key from sakaitest.sakai_realm_role where

role_name='Teaching Assistant')

/

spool off

/




#1