STILL UNDER CONSTRUCTION
Start time: 3:45am
End time: 8:00am
...
Personnel:
...
- Kirk Alexander
- Sandra Stewart
- Pete Peterson
- David Woods
- UC Davis programming teamProgramming Team
- rSmart Programming Team
- rSmart programming teamSupport Team
...
Communication
...
- server: irc.freenode.net
- channel: #ucdsakai
...
Pre Implementation tasks
...
(March 16, rSmart team) check out SAK-3163.patch for Gradebook 2 from our subversion patches director.
...
Prepare Prod local.properties file for tag 2.6.2.13 wrt RSN-3986-8307456
...
During the Implementation rollout
...
- Add database markers.
- (3:45am, rSmart) Place SmartSite-prod into maintenance mode and stop ALL TomCats. #1
- (3:45am to 5:00am, already scheduled run) Allow already scheduled back up of internal database to run.
- (5:00am to 5:30am, rSmart) rSmart will
- (5:00am to 5:02, rSmart) run the script for RSN-3986-8306969 to change the MOTD tool. See bottom of doc for sql. This SQL also exists on the production db server in /home/oracle/apr13.motd.sql AND run April 13th GB2 SQL in same location (see sql at end of doc)
- ( rSmart) for RSN-3986-8306769, change the number of qmail connection to 200 and recompile qmail.
- (, rSmart) Deploy tag 2.6.2.13 on SmartSite-Prod. Code will be compiled beforehand and just deployed at this point. Total time is 15 minutes, but it can be started any time after 3:45am after tomcat is stopped.
- (N/A, rSmart) Retrieve attachment from RSN: 3986-8306816 ticket and apply to SmartSite prod. This code is already in the 2.6.2.12 tag
- (rSmart) RSN-3986-8307016, repair tools that need OSIV per lazy loading issues: Lazy loading exception in WebContent tool
- ( , rSmart) RSN-3986-8307456, Check out properties to Prod local.properties file.
- ( ,rSmart) RSN-3986-8307376, gradebook.editAssignments permission needs to be removed from TA Role. This is database sql.
- (5:30am to 5:35am, rSmart) Bring up one node, still in maintenance mode, for QA verification.
...
Content and Function Verification Testing
...
...
- (5:35am to 5:55***am, UCDavis) QA will verify that the system has been successfully upgraded by doing the following:
...
- Verify that we can search on a student name in Gradebook (on the spreadsheet side)
...
- Verify that TA’s do not have permission to change category and item set up in Gradebook. TA should not be able to change IOR's categories and item.
...
- Should have only permissions IOR specifically grants.
- Verify that a new site with gradebook will open immediately and not require refreshing for it to open.
...
- Verify that duplicate sections is fixed
...
- Verify that there are no lazy loading system errors. rSmart can do this.
- (5:55am--6:00am, rSmart and UCDavis) Team will decide go or roll back.
...
Vote on our readiness to bring the system back up
...
...
- (5:55am – 6:00am, rSmart and UCDavis) Team to vote on readiness to bring the system back up assuming we’re going forward. See rollback section for retreating.
...
Release the system to public
...
...
- (6:00am
...
- to 6:15am, rSmart) bring up all application servers.
...
- (6:15am
...
- to 6:20am, rSmart) will release application to public by reapplying normal mode.
...
- (6:20am
...
- to 6:23am, UCDavis) QA/PM will send email notification to our SmartSite community to say
...
- (6:23am
...
- to 6:25am, UCDavis) Project or Program Manager will change the MOTD.
...
Rollback
...
...
- (6:00am
...
- to 6:45am, rSmart) Restore the internal database.
...
- Deploy prior code tag (2.6.2.10) to all 7 tomcats.
...
- Bring up one node to test .
...
- (6:45am
...
- to 6:50am, UCDavis) QA will log in and verify that the fix has been removed by doing the following:
...
- Check database markers.
...
- Check duplicate section issue.
...
- Check gradebook menu and tree configuration.
...
- (6:50am
...
- to 7:05am, rSmart) will bring up remaining nodes
...
- (7:05am
...
- to 7:10am, rSmart) will release application to public
...
- (7:10am
...
- to 7:13am, UCDavis) QA/PM will send email notification to our SmartSite community to say the system is back up
...
- (7:13am
...
- to 7:15am, UCDavis) Project or Program Manager will change the MOTD.
...
Placing Prod servers into Scheduled Maintenance mode & regular operating mode (called 'outage' mode)
...
...
- Apache into scheduled maintenance mode when there is scheduled downtime as follows:
...
- as user ucdavis on mach execute /var/www/html/ucdavis/apache/maint_mode.sh
...
- as user 'root' on mach, edit /etc/httpd/conf/httpd_ucdavis.conf and uncomment the Proxy * stuff on lines 36-40
...
- restart apache: /etc/init.d/httpd_ucdavis restart
...
- to restore regular functioning (normal "outage" mode is when there is no downtime planned, so any downtime is an "outage")
...
- as user ucdavis on mach execute /var/www/html/ucdavis/apache/outage_mode.sh
...
- as user root on mach, comment out Proxy * stuff & restart apache
| Code Block |
|---|
-- sql from 8306969
spool apr13.motd.output;
INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, 'false', 'show-subject' FROM rsn_tool WHERE tool_id = 'sakai.motd';
INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '3', 'items' FROM rsn_tool WHERE tool_id = 'sakai.motd';
INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '/announcement/channel/!site/motd', 'channel' FROM rsn_tool WHERE tool_id = 'sakai.motd';
INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, 'true', 'show-newlines' FROM rsn_tool WHERE tool_id = 'sakai.motd';
INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '50000', 'length' FROM rsn_tool WHERE tool_id = 'sakai.motd';
INSERT INTO RSN_TOOL_MUTABLE_CONFIG ( TOOL_ID, PROP_VALUE, PROP_KEY) SELECT id, '1000', 'days' FROM rsn_tool WHERE tool_id = 'sakai.motd';
spool off;
GB2 SQL:
spool apr06
select count(*) cnt from sakaitest.sakai_realm_rl_fn where function_key =
(select function_key
from sakaitest.sakai_realm_function
where
function_name = 'gradebook.editAssignments'
)
and
role_key in (select role_key from sakaitest.sakai_realm_role where
role_name='Teaching Assistant')
/
delete from sakaitest.sakai_realm_rl_fn where function_key =
(select function_key
from sakaitest.sakai_realm_function
where
function_name = 'gradebook.editAssignments'
)
and
role_key in (select role_key from sakaitest.sakai_realm_role where
role_name='Teaching Assistant')
/
commit
/
select count(*) cnt from sakaitest.sakai_realm_rl_fn where function_key =
(select function_key
from sakaitest.sakai_realm_function
where
function_name = 'gradebook.editAssignments'
)
and
role_key in (select role_key from sakaitest.sakai_realm_role where
role_name='Teaching Assistant')
/
spool off
/
#1
|