Start time: 3:45am
End time: 8:00am
...
Personnel:
...
- Kirk Alexander
- Sandra Stewart
- Pete Peterson
- David Woods
- UC Davis programming team
- rSmart programming team
...
Communication
...
- server: irc.freenode.net
- channel: #ucdsakai
During Maintenance Window
...
Pre Implementation tasks
...
(March 16, rSmart team) check out SAK-3163.patch for Gradebook 2 from our subversion patches director.
...
Prepare Prod local.properties file for tag 2.6.2.13 wrt RSN-3986-8307456
...
During the Implementation rollout
...
- Add database markers.
- (3:45am, rSmart) Place SmartSite-prod into maintenance mode and stop ALL TomCats. #1
- (3:45am to 5:00am, already scheduled run) Allow already scheduled back up of internal database to run.
- (5:00am to 5:30am, rSmart) rSmart will
- (5:00am to 5:02, rSmart) run the script for RSN-3986-8306969 to change the MOTD tool. See bottom of doc for sql. This SQL also exists on the production db server in /home/oracle/apr13.motd.sql AND run April 13th GB2 SQL in same location (see sql at end of doc)
- ( rSmart) for RSN-3986-8306769, change the number of qmail connection to 200 and recompile qmail.
- (, rSmart) Deploy tag 2.6.2.13 on SmartSite-Prod. Code will be compiled beforehand and just deployed at this point. Total time is 15 minutes, but it can be started any time after 3:45am after tomcat is stopped.
- (N/A, rSmart) Retrieve attachment from RSN: 3986-8306816 ticket and apply to SmartSite prod. This code is already in the 2.6.2.12 tag
- (rSmart) RSN-3986-8307016, repair tools that need OSIV per lazy loading issues: Lazy loading exception in WebContent tool
- ( , rSmart) RSN-3986-8307456, Check out properties to Prod local.properties file.
- ( ,rSmart) RSN-3986-8307376, gradebook.editAssignments permission needs to be removed from TA Role. This is database sql.
- (5:30am to 5:35am, rSmart) Bring up one node, still in maintenance mode, for QA verification.
...
Content and Function Verification Testing
...
- (5:35am to 5:55***am, UCDavis) QA will verify that the system has been successfully upgraded by doing the following:
- Verify that we can search on a student name in Gradebook (on the spreadsheet side)
- Verify that TA’s do not have permission to change category and item set up in Gradebook.
- Verify that a new site with gradebook will open immediately and not require refreshing for it to open.
- Verify that duplicate sections is fixed
- (5:55am--6:00am, rSmart and UCDavis) Team will decide go or roll back.
...
Vote on our readiness to bring the system back up
...
- (5:55am – 6:00am, rSmart and UCDavis) Team to vote on readiness to bring the system back up assuming we’re going forward. See rollback section for retreating.
...
Release the system to public
...
- (6:00am to 6:15am, rSmart) bring up all application servers.
- (6:15am to 6:20am, rSmart) will release application to public by reapplying normal mode.
- (6:20am to 6:23am, UCDavis) QA/PM will send email notification to our SmartSite community to say
- (6:23am to 6:25am, UCDavis) Project or Program Manager will change the MOTD.
...
Rollback
...
- (6:00am to 6:45am, rSmart) Restore the internal database.
- Deploy prior code tag (2.6.2.10) to all 7 tomcats.
- Bring up one node to test .
- (6:45am to 6:50am, UCDavis) QA will log in and verify that the fix has been removed by doing the following:
- Check database markers.
- Check duplicate section issue.
- Check gradebook menu and tree configuration.
- (6:50am to 7:05am, rSmart) will bring up remaining nodes
- (7:05am to 7:10am, rSmart) will release application to public
- (7:10am to 7:13am, UCDavis) QA/PM will send email notification to our SmartSite community to say the system is back up
- (7:13am to 7:15am, UCDavis) Project or Program Manager will change the MOTD.
...
Placing Prod servers into Scheduled Maintenance mode & regular operating mode (called 'outage' mode)
...
- Apache into scheduled maintenance mode when there is scheduled downtime as follows:
- as user ucdavis on mach execute /var/www/html/ucdavis/apache/maint_mode.sh
- as user 'root' on mach, edit /etc/httpd/conf/httpd_ucdavis.conf and uncomment the Proxy * stuff on lines 36-40
- restart apache: /etc/init.d/httpd_ucdavis restart
- to restore regular functioning (normal "outage" mode is when there is no downtime planned, so any downtime is an "outage")
- as user ucdavis on mach execute /var/www/html/ucdavis/apache/outage_mode.sh
- as user root on mach, comment out Proxy * stuff & restart apache