Page Comparison

...

DB Conversion Scripts : SAK-1871

• Modified Sakai Foundation OOTB upgrade script so that it cleanly applies on top of a current 2-4-x-HEAD production DB snapshot
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2.4.X-TO-sakai_2.5.X/sakai_2_4_x-2_5_x_oracle_conversion_ucd.sql
• While creating sakai_2_4_x-2_5_x_oracle_conversion_ucd.sql, we also started to create a document that lists all the DDL that is currently not generated by the 2-5-x Hibernate auto.ddl=on
• Comparing UCD 2-5-x OOTB against "sakai_2_4_x-2_5_x_oracle_conversion_ucd.sql" applied to a UCD 2-4-x-HEAD prod snapshot
  • https://mware.ucdavis.edu/svn/ucd-sakai/update/sakai_2.4.X-TO-sakai_2.5.X/sakai_diff_sakai_test_compare_20080514.sql
• Next step is to consolidate sakai_2_4_x-2_5_x_oracle_conversion_ucd.sql and sakai_diff_sakai_test_compare_20080514.sql

Duplicate tables

• We are going to Remove Duplicate and Temporary Tables from the Production DB

Gradebook : SAK-1832

Anchor
Gradebook Gradebook

• Need to evaluate whether to port/refactor the following changes
  • SAK-1746 Adhoc groups interfere with grading
  • SAK-764Gradebook : GradeDownloadConfig.properties points to ips instead of classes
  • SAK-1062 Error generated in Gradebook when submitting a points grade to a gb item which will not be included in the course grade.
  • SAK-1348 Hide Begin Final Grading button from TA's
  • SAK-935 Refactor gradebook modifications for 2.4
  • SAK-936 add site memberships (back) into default (all sections) views of roster and course grades

• One of the first steps to accomplishing this overall objective is to inject our CourseGradesToFinalGradingToolConverter into the Gradebook in place of the sample one that injects out of the box. This is accomplished by modifying spring-facades.xml under the gradebook project. SAK-1870 is an asset for that end.
• Unfortunately, because the CourseGradesToFinalGradingToolConverter does not execute correctly unless the patch is applied to EnrollmentTableBean and this patch does not apply cleanly, it's necessary to get into the guts of that bean and mess around.

SAK-1746 "Adhoc groups interfere with grading"

...

Content Hosting, Resource Conversion : SAK-????

• From the 2-5-x installation guide:

  No Format

  Moving from 2.4 to 2.5 requires adding several columns to the database tables used by the Content Hosting Service (the service used by the Resources tool, Dropbox, filepicker and webDav). The conversion scripts contain DDL statements to accomplish those changes. The new columns enable a switch from XML serialization to "binary-entity" serialization, which is faster and requires less memory. They also enable improved performance of quota calculations. You can get the benefits of binary-entity serialization without running the conversion utility, but you will never get the quota-calculation improvements without doing the full conversion. There are two start up modes: 1. Dual mode - if the code detects the binary entity upon start up, it will run in dual mode. This means it will be capable of reading xml but will only writing to binary. This would give you the benefits of binary-entity serialization without running the conversion utility, but you will never get the quota-calculation improvements without doing the full conversion. 2. Binary only - The system will only reads and write binary and occurs only after a full database migration. Only once this is done, will a system be able to fully realize all the performance enhancements that binary entity serialization offers. This occurs upon start up when it's detected that all of the xml fields are null. There are two methods to get a fully database conversion. 1. Run the conversion utility which can be run on a live system. Systems running oracle with large amounts should not attempt this right now. 2. Run the conversion upon start up. This is only recommended for implementations with smaller datasets.

• 2008-04-07 : Sent mail to Jim Eng requesting information/status about method #1, which seems not quite ready for large oracle installations
  • Jim's Reply

    No Format

    Here's what I recommend for oracle systems right now (until the oracle queries get updated in trunk and 2.5.x): I'd recommend you do a separate build to generate the conversion utility, rather than building it as part of building sakai for your production servers. The best way to do that for oracle at this point is to base it on 2.4.x rather than 2.5.x. The conversion utility will be essentially the same. Here's how to do that: Check out Sakai 2.4.x. You could start with the cafe checkout of 2.4.x, I think. That should give everything needed. Replace the content, db and entity with the SAK-12239 branches of those projects. https://source.sakaiproject.org/svn/content/branches/SAK-12239 https://source.sakaiproject.org/svn/db/branches/SAK-12239 https://source.sakaiproject.org/svn/entity/branches/SAK-12239 You need to select a project.xml file for the content/content-conversion project. I do that by going to dev/sakai/content/content-conversion and making a copy of project.xml.oracle named project.xml in that folder. Then I go back to dev/sakai/ and run a maven-1 build of that entire sakai. You will need the oracle jdbc driver to build this project. We can't provide a mechanism to include that automatically because you need to agree to the copyright protection. So you need to get the latest oracle driver and put it in your local maven-1 repository. Here's the maven dependency: <dependency> <groupId>oracle</groupId> <artifactId>ojdbc14</artifactId> <jar>ojdbc14.jar</jar> <type>jar</type> <url>http://download.oracle.com/otn/utilities_drivers/jdbc/10203/ojdbc14.jar</url> <properties> <war.bundle>true</war.bundle> </properties> </dependency> That indicates that the jar can be downloaded from http://download.oracle.com/otn/utilities_drivers/jdbc/10203/ojdbc14.jar, which was the version we used. If you have an earlier version of the driver, it may not work. If there's a newer version, I'd use that instead. The default location for the maven-1 repo on my laptop is ~/.maven/repository, so I put a copy of that file at this location: /.maven/repository/oracle/jars/ojdbc14.jar If you don't do that, your maven-1 build will fail and display the URL from which you can get the oracle jdbc driver. Once you've built the war file, you can find it in dev/sakai/content/content-conversion/target. That war file contains everything needed to run the conversion utility. You will need to do a little more configuration before running the conversion utility. You could start up the conversion utility without any other preparation, but I recommend you first run the 2.4.x to 2.5.x conversion script that is in reference and then you can start up the 2.5.x code. If you haven't run the conversion script, you can find a script named "init1.sql" that contains the ddl statements to add the required new columns and indexes. Here are the steps for running the conversion (using a war file built from the latest code from subversion, r45197 or later): 1) Expand the war file, which creates a folder named "sakai-content-conversion" 2) Navigate down to the sakai-content-conversion/WEB-INF/classes directory 3) Edit the file sakai-content-conversion/WEB-INF/classes/runconversion.sh and provide the absolute path for the java runtime 4) Edit the file named sakai-content-conversion/WEB-INF/classes/upgradeschema-step1.config and provide the connection info (i.e. supply values for dbURL, dbUser and dbPass) 5) Start the conversion utility for part 1 of the conversion by running the shell script named runconversion.sh in no-hangup mode with the upgradeschema-step1.config file as a parameter and piping output to log files. For example, after changing directories to sakai-content-conversion/WEB-INF/classes/, from the shell prompt, issue a command with this form (where $LOGS is the url for a directory where you have permission to write log files): # nohup ./runconversion.sh ./upgradeschema-step1.config 1>> $LOGS/conversion1.log 2>> $LOGS/conversion1.errors & 6) Allow part 1 of the conversion to be completed without intervention, if possible. It is possible to stop the conversion by creating a file named "quit.txt" in the sakai-content-conversion/WEB-INF/classes directory. If this part of the conversion is stopped, it can be restarted by removing the quit.txt file from the sakai-content-conversion/WEB-INF/classes directory and then repeating step 5. The conversion will pick up where it left off. 7) Edit the file named sakai-content-conversion/WEB-INF/classes/upgradeschema-step2.config and provide the connection info (i.e. supply values for dbURL, dbUser and dbPass) 8) Start the conversion utility for part 2 of the conversion by running the shell script named runconversion.sh in no-hangup mode with the upgradeschema-step2.config file as a parameter and piping output to log files. For example, after changing directories to sakai-content-conversion/WEB-INF/classes/, from the shell prompt, issue a command with this form: # nohup ./runconversion.sh ./upgradeschema-step2.config 1>> $LOGS/conversion2.log 2>> $LOGS/conversion2.errors & 9) Allow part 2 of the conversion to be completed without intervention, if possible. It is possible to stop the conversion by creating a file named "quit.txt" in the sakai-content-conversion/WEB-INF/classes directory. If this part of the conversion is stopped, it can be restarted by removing the quit.txt file from the sakai-content-conversion/WEB-INF/classes directory and then repeating step 8. The conversion will pick up where it left off. It is not necessary to convert the CONTENT_RESOURCE_DELETE table. The current code in this branch is very similar to what we used at Michigan to do the conversion last month. The primary difference is that we separated out the ddl statements a little more and didn't allow the conversion utility to do any ddl. Letting the conversion utility handle some of the ddl simplifies the process but uses the same sql statements. I'm available to answer questions or provide advice.

    • Jim's Reply to the Reply

      No Format

      I'm not sure whether you're referring to the conversion scripts or the conversion utility when you say "scripts", so I will try to answer about both. When I referred to "scripts" I meant the sql conversion scripts (from /reference/). The part of those scripts related to Content Hosting can be run while 2.4.x is still running. They add columns and indexes that will be ignored by the 2.4.x code. Then you can shut down 2.4.x and bring up 2.5.0. The 2.5.0 code will convert a few of the existing content collection records during startup (the top-level folders). It will also convert a limited number of other records as people create or revise resources and collections. Then there's the piece Ian wrote that you start up using the runconversion.sh shell script. I refer to that as the "conversion utility" in an ineffective attempt to distinguish it from the sql conversion scripts. The conversion utility can be run while the new version of sakai (2.5.0 or later) is running. Once the conversion utility finishes migrating the data, the running instances of sakai will automatically start using the new columns that were added.

– Owen's Reply

We discovered last week that runconversion.sh upgradeschema-oracle.config
doesn't work for us.

Jim then very helpfully explained that he too found himself needing to
modify both the config and the script extensively. I think he is in the
midst of preparing a better version to share.

Side note: the switch to 2.5 also uncovered our local need to upgrade the
Oracle driver we were using. This needs to happen just to get things
working at all -- even prior to any conversion. The symptoms there were
problems uploading files (e.g., exceptions and seeing lots of SAX parsing
errors in catalina.out). See these for background:

http://bugs.sakaiproject.org/jira/browse/SAK-11960
http://article.gmane.org/gmane.comp.cms.sakai.devel/17574

JSR-170 : SAK-???

• Sakai FND Confluence page: http://confluence.sakaiproject.org/confluence/x/kno
• 2008-04-07 : Sent mail to sakai-dev and production mailing list asking about the JSR-170 Content Hosting status. Who planning on running it, who is running it?
  • Omer@rice.edu replied indicating that Rice is switching to JSR-170 this summer