awsedx

Upgrade Open edX

NOTE: Before proceeding to upgrade the Bitnami edX stack, ensure that you have performed a complete backup.

Upgrade edX Ginkgo to Hawthorn.2

The recommended approach to upgrade edX Ginkgo to Hawthorn is to migrate the data to a fresh installation of the Bitnami edX Hawthorn stack, and copy data and settings to it.

In order to proceed, log in to your edX Ginkgo machine and follow the steps below:

  • Create a directory where to store all the data:

    $ mkdir edx-upgrade-data

  • Dump databases required by edX:

    • MySQL:

      $ /opt/bitnami/mysql/bin/mysqldump -u root -p --databases bitnami_edxapp bitnami_edxapp_csmh bitnami_xqueue --add-drop-database > edx-upgrade-data/bitnami_edx_mysql.sql

    • MongoDB:

      $ /opt/bitnami/mongodb/bin/mongodump --authenticationDatabase admin -u root -p -d bitnami_edxapp --out edx-upgrade-data/bitnami_edx_mongodb
$ /opt/bitnami/mongodb/bin/mongodump --authenticationDatabase admin -u root -p -d bitnami_cs_comments_service --out edx-upgrade-data/bitnami_edx_mongodb

  • Copy edX Platform configuration and data files (which contain uploads, courses, etc.):

    $ cp /opt/bitnami/apps/edx/conf/*.json edx-upgrade-data
$ cp -R /opt/bitnami/apps/edx/var edx-upgrade-data

  • Copy edX XQueue configuration files:

    $ cp /opt/bitnami/apps/edx/conf/*.json edx-upgrade-data

  • Copy edX Forum configuration files:

    $ cp /opt/bitnami/apps/forum/conf/httpd-forum.conf edx-upgrade-data

  • Compress the generated files:

    $ tar czf edx-upgrade-data.tar.gz edx-upgrade-data

  • Copy edx-upgrade-data.tar.gz over to the new edX Hawthorn machine.

Next, log in to your edX Hawthorn machine, and navigate to the directory where you copied edx-upgrade-data.tar.gz. Then, proceed with the following steps:

  • Extract the compressed file:

    $ tar xzf edx-upgrade-data.tar.gz

  • Migrate databases:

    • MySQL:

      $ /opt/bitnami/mysql/bin/mysql -u root -p < edx-upgrade-data/bitnami_edx_mysql.sql

    • MongoDB:

      $ /opt/bitnami/mongodb/bin/mongorestore -h localhost --authenticationDatabase admin -u root --drop -d bitnami_edxapp edx-upgrade-data/bitnami_edx_mongodb/bitnami_edxapp
$ /opt/bitnami/mongodb/bin/mongorestore -h localhost --authenticationDatabase admin -u root --drop -d bitnami_cs_comments_service edx-upgrade-data/bitnami_edx_mongodb/bitnami_cs_comments_service

  • Migrate edX Platform data:

    • Replace the edX Platform data directory with the one copied from your edX Ginkgo installation (note that you need to clear the assets):

      $ sudo rm -rf /opt/bitnami/apps/edx/var
$ sudo cp -R edx-upgrade-data/var /opt/bitnami/apps/edx/var
$ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var
$ sudo rm -rf /opt/bitnami/apps/edx/var/staticfiles/*

    • (Optional) Review edX Platform configuration for custom changes in the previous installation (e.g. themes, SMTP configuration, etc.):

      • LMS configuration files: /opt/bitnami/apps/edx/conf/lms.auth.json and /opt/bitnami/apps/edx/conf/lms.env.json.
      • CMS configuration files: /opt/bitnami/apps/edx/conf/cms.auth.json and /opt/bitnami/apps/edx/conf/cms.env.json.

    • In order to upgrade from Ginkgo to Hawthorn, you need to drop tables used by djcelery (which should already be empty in your Ginkgo data):

      $ sudo /opt/bitnami/apps/edx/bin/python.edxapp manage.py lms drop_djcelery_tables --settings=bitnami

    • Run edX Platform migrations, which will update your Ginkgo data to be valid for Hawthorn:

      $ sudo /opt/bitnami/apps/edx/bin/edxapp-migrate-lms
$ sudo /opt/bitnami/apps/edx/bin/edxapp-migrate-cms

    • Update assets for edX Platform:

      $ sudo /opt/bitnami/apps/edx/bin/edxapp-update-assets-lms
$ sudo /opt/bitnami/apps/edx/bin/edxapp-update-assets-cms

  • Migrate edX XQueue data:

    • (Optional) Check the edX XQueue configuration file for any changes for customizing your application: /opt/bitnami/apps/xqueue/conf/xqueue.yml.

      NOTE: Since edX Hawthorn, a couple of major changes have happened: The XQueue configuration files xqueue.auth.json and xqueue.env.json have been merged into xqueue.yml, and it does not use RabbitMQ anymore.

    • Run edX XQueue migrations, which will update your Ginkgo data to be valid for Hawthorn:

      $ sudo /opt/bitnami/apps/xqueue/bin/django-admin.xqueue migrate --noinput

  • Restart Apache, edX Celery Workers and XQueue:

    $ sudo /opt/bitnami/ctlscript.sh restart apache
$ sudo /opt/bitnami/ctlscript.sh restart edx
$ sudo /opt/bitnami/ctlscript.sh restart xqueue

Upgrade edX Ficus.4 to Gingko.2

IMPORTANT: In order to upgrade any edX Ficus version to edX Ginkgo, you must first upgrade to edX Ficus.4

To upgrade edX Ficus.4 to Gingko.2, follow these steps:

  • Load the Bitnami stack environment, if it is not already loaded:

    $ sudo /opt/bitnami/use_edx
$ . /opt/bitnami/scripts/setenv.sh

  • Stop all edX services:

    $ sudo /opt/bitnami/ctlscript.sh stop

  • Install the GCC compiler.

    • Debian/Ubuntu:

      $ sudo apt-get update
$ sudo apt-get install gcc g++

    • RHEL/CentOS:

      $ sudo yum group install "Development Tools"

  • Navigate to the application directory at /opt/bitnami/apps/edx/edx-platform and save the current state of the Git repository:

    $ cd /opt/bitnami/apps/edx/edx-platform
$ git diff -- . ':!requirements' > ~/bitnami_edx-platform_changes.txt

  • Check out the Gingko.2 branch and apply the Bitnami changes to the Git repository:

    $ git fetch
$ git checkout -f open-release/ginkgo.2
$ git apply ~/bitnami_edx-platform_changes.txt

  • Install the necessary modules:

    $ npm install

  • Install edX’s Python dependencies in this order:

    $ . ../venvs/edxapp/bin/activate
$ pip install --exists-action=w -r requirements/edx/setuptools.txt
$ pip install --exists-action=w -r requirements/edx/pre.txt
$ pip install --exists-action=w -r requirements/edx/github.txt
$ pip install --exists-action=w -r requirements/edx/local.txt
$ pip install --exists-action=w -r requirements/edx/base.txt
$ pip install --exists-action=w -r requirements/edx/paver.txt
$ pip install --exists-action=w -r requirements/edx/post.txt
$ deactivate

  • Install edX Sandbox’s Python dependencies and fix ownerships:

    $ . ../venvs/edxapp-sandbox/bin/activate
$ sudo pip install -r requirements/edx-sandbox/setuptools.txt
$ sudo pip install -r requirements/edx-sandbox/base.txt --ignore-installed
$ sudo pip install -r requirements/edx-sandbox/local.txt
$ sudo pip install -r requirements/edx-sandbox/post.txt
$ deactivate
$ sudo chown -R sandbox:sandbox ../venvs/edxapp-sandbox/

  • Start databases:

    $ sudo /opt/bitnami/ctlscript.sh start mysql
$ sudo /opt/bitnami/ctlscript.sh start mongodb
$ sudo /opt/bitnami/ctlscript.sh start memcached
$ sudo /opt/bitnami/ctlscript.sh start rabbitmq
$ sudo /opt/bitnami/ctlscript.sh start elasticsearch

  • Update edX assets. Note that this may take a significant amount of time for each step:

    $ ../bin/edxapp-update-assets-cms
$ ../bin/edxapp-update-assets-lms

  • Drop the djcelery database tables:

    $ sudo ../bin/python.edxapp manage.py lms drop_djcelery_tables --settings=aws

  • Run edX migrations (this may also take some time):

    $ sudo ../bin/edxapp-migrate-cms
$ sudo ../bin/edxapp-migrate-lms

  • Fix log file ownership:

    $ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var/log

  • Edit the /opt/bitnami/apps/edx/scripts/ctl-edx-celery-workers.sh script and replace this:

    start_worker() {
    ...
}

    with this:

    start_worker() {
    EDX_HOME="/opt/bitnami/apps/edx"
    cd $EDX_HOME/edx-platform
    APP=$(echo $1 | cut -d. -f2)
    ADDITIONAL_PARAMS=""
    if [ "$APP" = "cms" ]; then
        ADDITIONAL_PARAMS="-I contentstore.tasks"
    fi
    if [ `id|sed -e s/uid=//g -e s/\(.*//g` -eq 0 ]; then
        su -s /bin/sh daemon -c "$EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &"
    else
        $EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &
    fi
    cd - > /dev/null
}

  • Edit the lms.env.json file and edit the CELERY_QUEUES section so that it looks like this:

    "CELERY_QUEUES": [
    "edx.lms.core.low",
    "edx.lms.core.default",
    "edx.lms.core.high",
    "edx.lms.core.high_mem"
],
...

  • Upgrade xqueue:

    $ cd /opt/bitnami/apps/xqueue/xqueue
$ git diff -- . ':!*requirements.txt' ':!xqueue/test_settings.py' > ~/bitnami_xqueue_changes.txt
$ git fetch
$ git checkout -f open-release/ginkgo.2
$ git apply ~/bitnami_xqueue_changes.txt

  • Migrate and update users:

    $ sudo ../bin/django-admin.xqueue migrate --noinput
$ sudo ../bin/django-admin.xqueue update_users

  • Upgrade the comments service:

    $ cd /opt/bitnami/apps/forum/cs_comments_service
$ git fetch
$ git checkout -f open-release/ginkgo.2
$ sed -i 's/^ruby ".*"$//' Gemfile
$ bundle install

  • Start the remaining services:

    $ sudo /opt/bitnami/ctlscript.sh start apache
$ sudo /opt/bitnami/ctlscript.sh start edx
$ sudo /opt/bitnami/ctlscript.sh start xqueue

Your Bitnami edX stack should now be upgraded to Gingko.2. Confirm that the upgrade was successful by logging into edX and verifying the integrity of your data.

Upgrade edX Ficus to Ficus.4

To upgrade edX Ficus to Ficus.4, follow these steps:

  • Load the Bitnami stack environment, if it is not already loaded:

    $ sudo /opt/bitnami/use_edx
$ . /opt/bitnami/scripts/setenv.sh

  • Stop all edX services:

    $ sudo /opt/bitnami/ctlscript.sh stop

  • Install the GCC compiler.

    • Debian/Ubuntu:

      $ sudo apt-get update
$ sudo apt-get install gcc g++

    • RHEL/CentOS:

      $ sudo yum group install "Development Tools"

  • Navigate to the application directory at /opt/bitnami/apps/edx/edx-platform and save the current state of the Git repository:

    $ cd /opt/bitnami/apps/edx/edx-platform
$ git diff -- . ':!requirements' > ~/bitnami_edx-platform_changes.txt

  • Check out the Ficus.4 branch and apply the Bitnami changes to the Git repository:

    $ git fetch
$ git checkout -f open-release/ficus.4
$ git apply ~/bitnami_edx-platform_changes.txt

  • Install the necessary modules:

    $ npm install

  • Install edX’s Python dependencies in this order:

    $ . ../venvs/edxapp/bin/activate
$ pip install --exists-action=w -r requirements/edx/setuptools.txt
$ pip install --exists-action=w -r requirements/edx/pre.txt
$ pip install --exists-action=w -r requirements/edx/github.txt
$ pip install --exists-action=w -r requirements/edx/local.txt
$ pip install --exists-action=w -r requirements/edx/base.txt
$ pip install --exists-action=w -r requirements/edx/paver.txt
$ pip install --exists-action=w -r requirements/edx/post.txt
$ deactivate

  • Install edX Sandbox’s Python dependencies and fix ownerships:

    $ . ../venvs/edxapp-sandbox/bin/activate
$ sudo pip install -r requirements/edx-sandbox/setuptools.txt
$ sudo pip install -r requirements/edx-sandbox/base.txt --ignore-installed
$ sudo pip install -r requirements/edx-sandbox/local.txt
$ sudo pip install -r requirements/edx-sandbox/post.txt
$ deactivate
$ sudo chown -R sandbox:sandbox ../venvs/edxapp-sandbox/

  • Start databases:

    $ sudo /opt/bitnami/ctlscript.sh start mysql
$ sudo /opt/bitnami/ctlscript.sh start mongodb
$ sudo /opt/bitnami/ctlscript.sh start memcached
$ sudo /opt/bitnami/ctlscript.sh start rabbitmq
$ sudo /opt/bitnami/ctlscript.sh start elasticsearch

  • Update edX assets. Note that this may take a significant amount of time for each step:

    $ ../bin/edxapp-update-assets-cms
$ ../bin/edxapp-update-assets-lms

  • Run edX migrations (this may also take some time):

    $ sudo ../bin/edxapp-migrate-cms
$ sudo ../bin/edxapp-migrate-lms

  • Fix log file ownership:

    $ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var/log

  • Edit the /opt/bitnami/apps/edx/scripts/ctl-edx-celery-workers.sh script and replace this:

    start_worker() {
    ...
}

    with this:

    start_worker() {
    EDX_HOME="/opt/bitnami/apps/edx"
    cd $EDX_HOME/edx-platform
    APP=$(echo $1 | cut -d. -f2)
    ADDITIONAL_PARAMS=""
    if [ "$APP" = "cms" ]; then
        ADDITIONAL_PARAMS="-I contentstore.tasks"
    fi
    if [ `id|sed -e s/uid=//g -e s/\(.*//g` -eq 0 ]; then
        su -s /bin/sh daemon -c "$EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &"
    else
        $EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &
    fi
    cd - > /dev/null
}

  • Edit the lms.env.json file and edit the CELERY_QUEUES section so that it looks like this:

    "CELERY_QUEUES": [
    "edx.lms.core.low",
    "edx.lms.core.default",
    "edx.lms.core.high",
    "edx.lms.core.high_mem"
],

  • Start remaining services:

    $ sudo /opt/bitnami/ctlscript.sh start apache
$ sudo /opt/bitnami/ctlscript.sh start edx

Your Bitnami edX stack should now be upgraded to Ficus.4. Confirm that the upgrade was successful by logging into edX and verifying the integrity of your data.

Last modification January 2, 2019