Implementation Plan August 14, 2010 (rev0.4)
Personnel:
â—� Anand Mehta – Application Support Analyst (480-236-4203 cell)
â—� Mark Pankow – DBA (602-369-8714 cell)
â—� Lauren Brodsky – rSmart Project Manager (602-430-5218 cell)
â—� Sandra Stewart – UCD SmartSite Project Manager (530-219-3151 cell)
â—� Joyce Johnstone – UCD SmartSite Project Manager (916-204-9639 cell)
â—� Thomas Amsler – UCD SmartSite Programmer (916-803-2226 cell)
â—� Mike Wenk – UCD SmartSite Programmer (916-764-2312 cell)
â—� David Woods – UCD SmartSite Level 3 Support (530-400-2691 cell, 530-752-4901 office)
â—� Vernon O’Connor – rSmart QA Analyst Support (707-761-0541 cell)
â—� Fernando Socorro – UCD SmartSite Trainer (415-637-4026 cell 530-754-5194 office)
â—� Kirk Alexander – UCDavis Program Manager (530-753-4430 home, 530-902-7352 cell)
Communication:
This level of communication represents communications with the team involved in the rollout, during the roll out, e.g, programmers communicating with the project manager and QA people, etc. We chat using IRC.
â—� server: irc.freenode.net
â—� channel: #ucdsakai
Login for verification while prod is still in maintenance mode:
â—� https://smartsite.ucdavis.edu:8081/xsl-portal
â—� Enter via guest login.
â—� Use your Kerberos login and ID.
Pre Implementation tasks:
On Friday, August 13, 2010:
â—� Sandra and Fernando will set database markers.
â—� Anand Mehta will:
â—‹ Copy the existing Tomcat to tomcat-264 on all seven app servers
â—‹ Delete all CLE files from tomcat-264
â—‹ Build and deploy the 2.6.4 code to tomcat-264
â—‹ Make a copy of all properties file in the sakai directory and update the copies to 2.6.4 level
â—� Anand Mehta to set the Sakai properties:
Panel |
---|
UCD Deployment Notes 1. Commands to remove old CLE files from Tomcat (executed from /home/ucdavis/cle/tomcat) rm -rf shared/lib/* 2. Verify the following setting in sakai.properties: portalPath=/xsl-portal (This is used by GB2 security, and should be already set in sakai.properties) 3. Modify/Add the following settings in local.properties a. version.service=2.6.4.0 b. version.sakai=2.6.2 c. Gradebook2 properties: d. Add to prevent lazy initialization errors: e. Add sakai.kaltura,sakai-itunesu-admin,sakai.iTunesU to the stealth tool setting f. Verify that all the Gradebook2 final grade submission properties are still in place 4. Ensure that auto.ddl is true in the first Tomcat's instance.properties 5. Make the following changes to SystemGlobals.properties a. set cache.engine.implementation=org.etudes.jforum.cache.MemoryServiceCacheEngine b. set html.tags.welcome=a, abbr, acronym, address, b, big, blockquote, br, center, cite, code, dd, del, dir, div, dl, dt, em, font, hr, h1, h2, h3, h4, h5, h6, I, ins,kbd, li, marquee, menu, nobr, noembed, ol, p, pre, q, rt, ruby, rbc, rb, rtc, rp,s, samp, small, span, strike, strong, sub, sup, tt, u, ul, var, xmp, img, embed,object, table, tr, td, th, tbody, caption, thead, tfoot, colgroup, col\ |
Saturday, August 14, 2010 during maintenance window: 5:00 am to 10:30am
1. (5:00 am to 5:10 am) Anand Mehta will place smartsite in maintenance mode and stop all Tomcats.
2. (5:11 am to 5:25 am) Anand Mehta will rename tomcat to tomcat-263 and tomcat-264 to tomcat on all app servers
3. (5:26 am to 5:40 am) Anand Mehta will backup the 2.6.3 properties files and put the 2.6.4 properties files in the sakai directory
4. (5:40am to 6:10 am) Mark Pankow will backup the existing DB.
1. No cron jobs will be disabled or altered. ** Mark will link to actual text document.
2. I will let the normally scheduled hot backup run at 4AM. It will not include a datapump (export) which is part of my "normal" daily backup.
3. At 5:40 I will manually export the SAKAI schema. It should not alter the timeline, but I will be running a test at some point this weekend to verify.
4. After the export, I will manually switch the logfile in production and make note of that.
5. I will then defer log transfer to the standby.
5. (6:10 am to 6:20) Mark Pankow will execute the upgrade scripts 2.6.2 to 2.6.3 and 2.6.3 to 2.6.4 CLE upgrade scripts as follows:
ADD: Talk to Anand: No commits; add commits or have DBA issue commits.
Panel |
---|
/* change Preferences to split out email into different fields that can be controlled individually.*/ /* update the new columns based on the old data */ /* if just requests, set requests to true, confirms to false */ /* if just confirms, set confirms to true, requests to false */ /* if all off, set both false */ /* now drop the old column */ /* change name of profile field to be profile_image since it only controls that now (PRFL-24) */ /* update values in profile_image and search, we no longer have 2 as an option (ie only me) */ /* add my_status column */ /* add the new academic column, default to 0, (PRFL-38) */ /* increase size of UUID columns (PRFL-44) */ /* resize column (PRFL-44) but also change its type (PRFL-45) */ /* add indexes (PRFL-76) */ /* CLE-4417 jforum reset icon missing */ /* CLE-4329 logical delete of sakai site data */ /* CLE-4321 trim space after term name */ |
6. (6:20 am to 6:40 am) Mark Pankow to run rsnfixes.sql [Mark will move to script files] ADD: Talk to Anand: No commits; add commits or have DBA issue commit.
Panel |
---|
-- Cannot Change Config options on some tools (8306969) -- Cannot assign TA in Section Info (8306675) -- Site Editor tool does not show for Participants in Project site (8306970) -- Stack trace when selecting the Content over Text layout during Freeform Portfolio creation (8306957) UCD Thinks this should be removed. 8/6/2010 -- Site Editor on new Course sites is being named Site Setup (8306811) UCD Thinks this should be removed because we don’t have QA results that say it fixed anything.. 8/6/2010 -- Bad data in sakai.GB_CATEGORY_T table causing grade submission failure (8308193) UPDATE GB_CATEGORY_T SET WEIGHT=ROUND(WEIGHT,14) WHERE ROUND(WEIGHT,14) != WEIGHT; |
7. (6:40am to 6:50 am) Mark Pankow to run SQL conversion for GB2. Talk to Thomas: Commit to script (Thomas)
Conversion script is below but source can be taken from:https://source.sakaiproject.org/contrib/gradebook2/tags/1.3.0-rc1/sakai/2-6-x/sql/oracle/conversion_1.2.0_to_1.3.0.sql.
At the time of deployment this link will be active for the source script:https://source.sakaiproject.org/contrib/gradebook2/tags/1.3.0/sakai/2-6-x/sql/oracle/conversion_1.2.0_to_1.3.0.sql
Panel |
---|
update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T update GB_ACTION_RECORD_PROPERTY_T |
8. (6:50 am to 6:55 am) Anand Mehta will check each prod node to make sure that the file /etc/krb5.conf does not exist. Rename existing to /etc/krb5.config.orig.
9. (6:55to 7:00 am) Add the commands to actually deploy the code.
mvn -Dmaven.test.skip=true -Dmaven.tomcat.home=/home/ucdavis/cle/tomcat/ -Dsakai-login-tool.cas -Ducd.cas.mothra.filter -Dsakai2.6 clean install sakai:deploy
10. Content and Function Verification Testing:
1. (7:00 am to 7:05 am) Anand will bring up one server still in maintenance mode for verification testing.
2. (7:05 am to 7:35am) Sandra, Joyce, Dave Woods, Fernando, Steve(?) and Vernon will verify that the system has been successfully upgraded by doing the following:
â—� Verify that webdav works, from course site xxx, from his Mac (10.4 and 10.6) Windows 7 and xp. (Fernando will do this).
â—� Verify that course management work by accessing Site Editor, Rosters, and Mailtool on sites to be determined. Screen shots are stored on RSN ticket, 8307884 for course sites BIS 002C A01-A15,B01-B15 SQ 2010, and BIS 002A A21-A26 SS2 2010 (Sandra will do this).
â—� Verify that adding and dropping a roster for a multi-section site changes the title in the tab (if we get permission from an instructor). Or duplicate a site, unpublish it, then add and drop the rosters and check the site. Then delete the site after the verification (Kirk will do this).
� Verify that, for RSN ticket 8306811 “Site Editor on new Course sites is being named Site Setup� no longer occurs. Create a new course site. Verify that it doesn’t have site info. Should say site editor. (Joyce will do this).
â—� Verify that exporting gradebook using IE8 works without an error. User course site xxx. This will be for letter grades and points (David will do this).
â—� Verify that we can import gradebooks without error, using course site AHI 001e SS1 2010. This will be for letter grades and points (Sandra will do this).
� Verify that we’re using the correct grade scale for letter grade gradebooks. This will use course site xxxxx. Modify the grade scale to be odd. Verify that grades are not looking at grade scale and have not changed. David will do this.
â—� Basic performance test (e.g. Site Editor performance), response time, first time in, should be no greater than 5 seconds. Use course site BIS 002C A01-A15,B01-B15 SQ 2010 with Susan Keen (slkeen), participants > 635. Vernon will do this.
â—� Check to make sure the gradebook sorts on its labels in the spreadsheet view the way we expect, A+ as highest, F as lowest. Will use course sites BIS 002C A01-A15,B01-B15 SQ 2010 & BIS 002C A01-A15,B01-B15 WQ 2010. Vernon will do this. Take a screen shot of this unacceptable sort prior to rollout, attach to GRBK-585.
â—� Check Section Info, verify that we can add TAs. Will use course sites BIS 002C A01-A15,B01-B15 SQ 2010 & BIS 002C A01-A15,B01-B15 WQ 2010. Vernon will do this.
â—� UCD and rSmart programming team will make sure final grades has the correct url
3. (7:35 am to 7:50 am) Anand Mehta will restart Tomcat on remaining servers.
4. (7:50 am to 7:55 am) Team will decide go or roll back.
5. In the case of “GO,� move to “Vote on our readiness to bring the system back up�
6. In the case of “ROLLBACK,� move to “Rollback Section�
Vote on our readiness to bring the system back up:
1. (7:55 am to 8:00 am) Team to vote on readiness to bring the system back up assuming we’re going forward. See rollback section for retreating.
Release the system to public:
1. (8:00 am to 8:15 am) Anand Mehta Release servers to the public.
2. (8:15 am to 8:30 am) UCD - PM will send email notification to our SmartSite community to say the system is back up.
3. (8:30 am to 8:45 am) UCD - Project or Program Manager will change the MOTD.
4. ADD (9:45) : Wait one hour; reconnect primary database to hot standby
a. Details to be added by Mark
Post release actions:
1. GRBK-646: Check production DB if we have duplicate (same realm_id) entries in the GB_USER_DEREF_RM_UPDATE_T
a. select realm_id, count as NUM_DUPS from GB_USER_DEREF_RM_UPDATE_T group by REALM_ID having count > 1;
Rollback:
â—� Anand Mehta rolls back the upgrade by doing the following:
1. (8:00 am to 8:15 am) Anand Mehta stops any running Tomcats.
2. (8:15 am to 8:25 am) Anand Mehta renames tomcat to tomcat-264 and tomcat-263 to tomcat on all app servers.
3. (8:25 am to 8:40 am) Anand Mehta backs up the existing properties files and puts the 2.6.2 properties files in the sakai directory.
4. (8:40 am to 9:40 am) Mark Pankow
1. I would import the SAKAI schema from the export taken manually. That should not take more than the 40 minutes alloted.
2. Additionally, measures above would allow for a restore and recovery to the point right before the upgrade using the 4AM hot backup.
3. Lastly, the standby base is there and ready to be converted to the primary in the event of catastrophic failure.
4. ADD: log transfer; reconnect primary database to hot standby
5. ADD: Any syntax, commands, sql scripts for these steps above
5. (9:40 am to 9:50am)Anand Mehta starts one Tomcat.
...
Iframe | ||||||
---|---|---|---|---|---|---|
|