Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

STILL UNDER CONSTRUCTION

Start time: 3:45am
End time: 8:00am
Personnel:

  • Kirk Alexander
  • Sandra Stewart
  • Pete Peterson
  • David Woods
  • UC Davis programming team
  • rSmart programming team

Communication

  • server: irc.freenode.net
  • channel: #ucdsakai

During Maintenance Window
Pre Implementation tasks
(March 16, rSmart team) check out SAK-3163.patch for Gradebook 2 from our subversion patches director.

(March 16, rSmart team) build Release Candidate 1 on smartsite-test.rsmart.com using the SAK-3163 patch.

(March 16, rSmart team) add -Dsakai2.6 to the mvn command. This will tell gb2 to build using kernel but not edu-services.

(March 16, UCDavis team) attach code for duplicate sections fix patch (sak-3144) to RSN: 3986-8306816 ticket.      

(March 16, rSmart team) build patch for RSN: 3986-8306816 on smartsite-test.rsmart.com.

(March 22, UCDavis team) Check in SAK-3164.patch for Gradebook 2 into our subversion patches directory.

(March 22, rSmart team) check out Release Candidate 2 from our subversion.

(March 22, rSmart team) build Release Candidate 2, on smartsite-test.rsmart.com, for Gradebook 2.

(March 29, rSmart team) check out Final Release Candidate from our subversion.

(March 29, rSmart team) build Final Release Candidate on smartsite-test.rsmart.com for Gradebook 2.

[note: we may have several gradebook release candidates prior to the production release on April 13.]

Prepare Prod local.properties file for tag 2.6.2.13 wrt RSN-3986-8307456

During the Implementation rollout

  • Add database markers.
  • (3:45am, rSmart) Place SmartSite-prod into maintenance mode and stop ALL TomCats. #1
  • (3:45am to 5:00am, already scheduled run) Allow already scheduled back up of internal database to run.
  • (5:00am to 5:30am, rSmart) rSmart will
    • (5:00am to 5:02, rSmart) run the script for RSN-3986-8306969 to change the MOTD tool.  See bottom of doc for sql.  This SQL also exists on the production db server in /home/oracle/apr13.motd.sql AND run April 13th GB2 SQL in same location (see sql at end of doc)
    • ( rSmart) for RSN-3986-8306769, change the number of qmail connection to 200 and recompile qmail.
    • (, rSmart) Deploy tag 2.6.2.13 on SmartSite-Prod.  Code will be compiled beforehand and just deployed at this point.  Total time is 15 minutes, but it can be started any time after 3:45am after tomcat is stopped.
    • (N/A, rSmart) Retrieve attachment from RSN: 3986-8306816 ticket and apply to SmartSite prod. This code is already in the 2.6.2.12 tag
    • (rSmart) RSN-3986-8307016, repair tools that need OSIV per lazy loading issues: Lazy loading exception in WebContent tool
    • ( , rSmart)  RSN-3986-8307456, Check out properties to Prod local.properties file.
    • ( ,rSmart) RSN-3986-8307376, gradebook.editAssignments permission needs to be removed from TA Role.  This is database sql.
    • (5:30am to 5:35am, rSmart)  Bring up one node, still in maintenance mode, for QA verification.

Content and Function Verification Testing
•    (5:35am to 5:55***am, UCDavis) QA  will verify that the system has been successfully upgraded by doing the following:
o    Verify that we can search on a student name in Gradebook (on the spreadsheet side)
o    Verify that TA’s do not have permission to change category and item set up in Gradebook.
o    Verify that a new site with gradebook will open immediately and not require refreshing for it to open.
o    Verify that duplicate sections is fixed
•     (5:55am--6:00am, rSmart and UCDavis) Team will decide go or roll back.

Vote on our readiness to bring the system back up
•    (5:55am – 6:00am, rSmart and UCDavis) Team to vote on readiness to bring the system back up assuming we’re going forward.  See rollback section for retreating.

Release the system to public
•    (6:00am to 6:15am, rSmart) bring up all application servers.
•    (6:15am to 6:20am, rSmart) will release application to public by reapplying normal mode.
•    (6:20am to 6:23am, UCDavis)  QA/PM will send email notification to our SmartSite community to say the system is back up.  (6:23am to 6:25am, UCDavis)   Project or Program Manager will change the MOTD.

Rollback
•    (6:00am to 6:45am, rSmart) Restore the internal database.
♣    Deploy prior code tag (2.6.2.10) to all 7 tomcats.
♣    Bring up one node to test .
•    (6:45am to 6:50am, UCDavis)  QA will log in and verify that the fix has been removed by doing the following:
♣    Check database markers.
♣    Check duplicate section issue.
♣    Check gradebook menu and tree configuration.
•     (6:50am to 7:05am, rSmart) will bring up remaining nodes
•    (7:05am to 7:10am, rSmart)  will release application to public
•    (7:10am to 7:13am, UCDavis)  QA/PM will send email notification to our SmartSite community to say the system is back up
•    (7:13am to 7:15am, UCDavis)   Project or Program Manager will change the MOTD.

Placing Prod servers into Scheduled Maintenance mode & regular operating mode (called 'outage' mode)
•    Place Apache into scheduled maintenance mode when there is scheduled downtime as follows:
â—¦    as user ucdavis on mach execute /var/www/html/ucdavis/apache/maint_mode.sh
â—¦    as user 'root' on mach, edit /etc/httpd/conf/httpd_ucdavis.conf and uncomment the Proxy * stuff on lines 36-40
â—¦    restart apache: /etc/init.d/httpd_ucdavis restart
•    to restore regular functioning (normal "outage" mode is when there is no downtime planned, so any downtime is an "outage")
â—¦    as user ucdavis on mach execute /var/www/html/ucdavis/apache/outage_mode.sh
â—¦    as user root on mach, comment out Proxy * stuff & restart apache

  • No labels