awsedx

Upgrade Open edX

NOTE: Before proceeding to upgrade the Bitnami edX stack, ensure that you have performed a complete backup.


Upgrade edX Ginkgo or edX Hawthorn to Ironwood.1

The recommended approach to upgrade edX Ginkgo/Hawthorn to Ironwood.1 is to migrate the data to a fresh installation of the Bitnami edX Ironwood.1 stack, and copy data and settings to it.

In order to proceed, log in to your edX Ginkgo/Hawthorn machine and follow the steps below:

  • Create a directory where to store all the data:

    $ mkdir edx-upgrade-data
    
  • Dump databases required by edX:

    • MySQL:

      $ /opt/bitnami/mysql/bin/mysqldump -u root -p --databases bitnami_edxapp bitnami_edxapp_csmh bitnami_xqueue --add-drop-database > edx-upgrade-data/bitnami_edx_mysql.sql
      
    • MongoDB:

      $ /opt/bitnami/mongodb/bin/mongodump --authenticationDatabase admin -h localhost -u root -d bitnami_edxapp --out edx-upgrade-data/bitnami_edx_mongodb
      $ /opt/bitnami/mongodb/bin/mongodump --authenticationDatabase admin -h localhost -u root -d bitnami_cs_comments_service --out edx-upgrade-data/bitnami_edx_mongodb
      
  • Copy edX Platform configuration and data files (which contain uploads, courses, etc.). The commands below also clear the assets to reduce the size of the directory:

    $ cp /opt/bitnami/apps/edx/conf/*.json edx-upgrade-data
    $ cp -R /opt/bitnami/apps/edx/var edx-upgrade-data
    $ rm -rf edx-upgrade-data/var/staticfiles/*
    
  • Copy edX XQueue configuration files:

    $ cp /opt/bitnami/apps/edx/conf/*.json edx-upgrade-data
    
  • Copy edX Forum configuration files:

    $ cp /opt/bitnami/apps/forum/conf/httpd-forum.conf edx-upgrade-data
    
  • Compress the generated files:

    $ tar czf edx-upgrade-data.tar.gz edx-upgrade-data
    
  • Copy edx-upgrade-data.tar.gz over to the new edX Ironwood.1 machine.

Next, log in to your edX Ironwood.1 machine, and navigate to the directory where you copied edx-upgrade-data.tar.gz. Then, proceed with the following steps:

  • Extract the compressed file:

    $ tar xzf edx-upgrade-data.tar.gz
    
  • Migrate databases:

    • MySQL:

      $ /opt/bitnami/mysql/bin/mysql -u root -p < edx-upgrade-data/bitnami_edx_mysql.sql
      
    • MongoDB:

      $ /opt/bitnami/mongodb/bin/mongorestore --authenticationDatabase admin -h localhost -u root --drop -d bitnami_edxapp edx-upgrade-data/bitnami_edx_mongodb/bitnami_edxapp
      $ /opt/bitnami/mongodb/bin/mongorestore --authenticationDatabase admin -h localhost -u root --drop -d bitnami_cs_comments_service edx-upgrade-data/bitnami_edx_mongodb/bitnami_cs_comments_service
      
  • Migrate edX Platform data:

    • Replace the edX Platform data directory with the one copied from your edX Ginkgo/Hawthorn installation:

      $ sudo rm -rf /opt/bitnami/apps/edx/var
      $ sudo cp -R edx-upgrade-data/var /opt/bitnami/apps/edx/var
      $ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var
      
    • (Optional) Review edX Platform configuration for custom changes in the previous installation (e.g. themes, SMTP configuration, etc.):

      • LMS configuration files: /opt/bitnami/apps/edx/conf/lms.auth.json and /opt/bitnami/apps/edx/conf/lms.env.json.
      • CMS configuration files: /opt/bitnami/apps/edx/conf/cms.auth.json and /opt/bitnami/apps/edx/conf/cms.env.json.
    • Drop tables used by djcelery (which should already be empty in your Ginkgo data):

      NOTE: Only run the command below if you are upgrading from edX Ginkgo to Ironwood.1. It is not needed for edX Hawthorn to Ironwood.1 upgrades.

      $ sudo /opt/bitnami/apps/edx/bin/python.edxapp manage.py lms drop_djcelery_tables --settings=bitnami
      
    • Run edX Platform migrations, which will update your Ginkgo/Hawthorn data to be valid for Ironwood.1:

      $ sudo /opt/bitnami/apps/edx/bin/edxapp-migrate-lms
      $ sudo /opt/bitnami/apps/edx/bin/edxapp-migrate-cms
      
    • Update assets for edX Platform:

      $ sudo /opt/bitnami/apps/edx/bin/edxapp-update-assets-lms
      $ sudo /opt/bitnami/apps/edx/bin/edxapp-update-assets-cms
      
  • Migrate edX XQueue data:

    • (Optional) Review XQueue configuration for custom changes in your previous installation: /opt/bitnami/apps/xqueue/conf/xqueue.yml.

      NOTE: Since edX Hawthorn, a couple of major changes have happened: The XQueue configuration files xqueue.auth.json and xqueue.env.json have been merged into xqueue.yml, and it does not use RabbitMQ anymore.

    • Run edX XQueue migrations, which will update your Ginkgo/Hawthorn data to be valid for Ironwood.1:

      $ sudo /opt/bitnami/apps/xqueue/bin/django-admin.xqueue migrate --noinput
      
  • Restart Apache, edX Celery Workers and XQueue:

    $ sudo /opt/bitnami/ctlscript.sh restart apache
    $ sudo /opt/bitnami/ctlscript.sh restart edx
    $ sudo /opt/bitnami/ctlscript.sh restart xqueue
    

Upgrade edX Ficus.4 to Gingko.2

IMPORTANT: In order to upgrade any edX Ficus version to edX Ginkgo, you must first upgrade to edX Ficus.4

To upgrade edX Ficus.4 to Gingko.2, follow these steps:

  • Load the Bitnami stack environment, if it is not already loaded:

    $ sudo /opt/bitnami/use_edx
    $ . /opt/bitnami/scripts/setenv.sh
    
  • Stop all edX services:

    $ sudo /opt/bitnami/ctlscript.sh stop
    
  • Install the GCC compiler.

    • Debian/Ubuntu:

      $ sudo apt-get update
      $ sudo apt-get install gcc g++
      
    • RHEL/CentOS:

      $ sudo yum group install "Development Tools"
      
  • Navigate to the application directory at /opt/bitnami/apps/edx/edx-platform and save the current state of the Git repository:

    $ cd /opt/bitnami/apps/edx/edx-platform
    $ git diff -- . ':!requirements' > ~/bitnami_edx-platform_changes.txt
    
  • Check out the Gingko.2 branch and apply the Bitnami changes to the Git repository:

    $ git fetch
    $ git checkout -f open-release/ginkgo.2
    $ git apply ~/bitnami_edx-platform_changes.txt
    
  • Install the necessary modules:

    $ npm install
    
  • Install edX’s Python dependencies in this order:

    $ . ../venvs/edxapp/bin/activate
    $ pip install --exists-action=w -r requirements/edx/setuptools.txt
    $ pip install --exists-action=w -r requirements/edx/pre.txt
    $ pip install --exists-action=w -r requirements/edx/github.txt
    $ pip install --exists-action=w -r requirements/edx/local.txt
    $ pip install --exists-action=w -r requirements/edx/base.txt
    $ pip install --exists-action=w -r requirements/edx/paver.txt
    $ pip install --exists-action=w -r requirements/edx/post.txt
    $ deactivate
    
  • Install edX Sandbox’s Python dependencies and fix ownerships:

    $ . ../venvs/edxapp-sandbox/bin/activate
    $ sudo pip install -r requirements/edx-sandbox/setuptools.txt
    $ sudo pip install -r requirements/edx-sandbox/base.txt --ignore-installed
    $ sudo pip install -r requirements/edx-sandbox/local.txt
    $ sudo pip install -r requirements/edx-sandbox/post.txt
    $ deactivate
    $ sudo chown -R sandbox:sandbox ../venvs/edxapp-sandbox/
    
  • Start databases:

    $ sudo /opt/bitnami/ctlscript.sh start mysql
    $ sudo /opt/bitnami/ctlscript.sh start mongodb
    $ sudo /opt/bitnami/ctlscript.sh start memcached
    $ sudo /opt/bitnami/ctlscript.sh start rabbitmq
    $ sudo /opt/bitnami/ctlscript.sh start elasticsearch
    
  • Update edX assets. Note that this may take a significant amount of time for each step:

    $ ../bin/edxapp-update-assets-cms
    $ ../bin/edxapp-update-assets-lms
    
  • Drop the djcelery database tables:

    $ sudo ../bin/python.edxapp manage.py lms drop_djcelery_tables --settings=aws
    
  • Run edX migrations (this may also take some time):

    $ sudo ../bin/edxapp-migrate-cms
    $ sudo ../bin/edxapp-migrate-lms
    
  • Fix log file ownership:

    $ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var/log
    
  • Edit the /opt/bitnami/apps/edx/scripts/ctl-edx-celery-workers.sh script and replace this:

    start_worker() {
        ...
    }
    

    with this:

    start_worker() {
        EDX_HOME="/opt/bitnami/apps/edx"
        cd $EDX_HOME/edx-platform
        APP=$(echo $1 | cut -d. -f2)
        ADDITIONAL_PARAMS=""
        if [ "$APP" = "cms" ]; then
            ADDITIONAL_PARAMS="-I contentstore.tasks"
        fi
        if [ `id|sed -e s/uid=//g -e s/\(.*//g` -eq 0 ]; then
            su -s /bin/sh daemon -c "$EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &"
        else
            $EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &
        fi
        cd - > /dev/null
    }
    
  • Edit the lms.env.json file and edit the CELERY_QUEUES section so that it looks like this:

    "CELERY_QUEUES": [
        "edx.lms.core.low",
        "edx.lms.core.default",
        "edx.lms.core.high",
        "edx.lms.core.high_mem"
    ],
    ...
    
  • Upgrade xqueue:

    $ cd /opt/bitnami/apps/xqueue/xqueue
    $ git diff -- . ':!*requirements.txt' ':!xqueue/test_settings.py' > ~/bitnami_xqueue_changes.txt
    $ git fetch
    $ git checkout -f open-release/ginkgo.2
    $ git apply ~/bitnami_xqueue_changes.txt
    
  • Migrate and update users:

    $ sudo ../bin/django-admin.xqueue migrate --noinput
    $ sudo ../bin/django-admin.xqueue update_users
    
  • Upgrade the comments service:

    $ cd /opt/bitnami/apps/forum/cs_comments_service
    $ git fetch
    $ git checkout -f open-release/ginkgo.2
    $ sed -i 's/^ruby ".*"$//' Gemfile
    $ bundle install
    
  • Start the remaining services:

    $ sudo /opt/bitnami/ctlscript.sh start apache
    $ sudo /opt/bitnami/ctlscript.sh start edx
    $ sudo /opt/bitnami/ctlscript.sh start xqueue
    

Your Bitnami edX stack should now be upgraded to Gingko.2. Confirm that the upgrade was successful by logging into edX and verifying the integrity of your data.

Upgrade edX Ficus to Ficus.4

To upgrade edX Ficus to Ficus.4, follow these steps:

  • Load the Bitnami stack environment, if it is not already loaded:

    $ sudo /opt/bitnami/use_edx
    $ . /opt/bitnami/scripts/setenv.sh
    
  • Stop all edX services:

    $ sudo /opt/bitnami/ctlscript.sh stop
    
  • Install the GCC compiler.

    • Debian/Ubuntu:

      $ sudo apt-get update
      $ sudo apt-get install gcc g++
      
    • RHEL/CentOS:

      $ sudo yum group install "Development Tools"
      
  • Navigate to the application directory at /opt/bitnami/apps/edx/edx-platform and save the current state of the Git repository:

    $ cd /opt/bitnami/apps/edx/edx-platform
    $ git diff -- . ':!requirements' > ~/bitnami_edx-platform_changes.txt
    
  • Check out the Ficus.4 branch and apply the Bitnami changes to the Git repository:

    $ git fetch
    $ git checkout -f open-release/ficus.4
    $ git apply ~/bitnami_edx-platform_changes.txt
    
  • Install the necessary modules:

    $ npm install
    
  • Install edX’s Python dependencies in this order:

    $ . ../venvs/edxapp/bin/activate
    $ pip install --exists-action=w -r requirements/edx/setuptools.txt
    $ pip install --exists-action=w -r requirements/edx/pre.txt
    $ pip install --exists-action=w -r requirements/edx/github.txt
    $ pip install --exists-action=w -r requirements/edx/local.txt
    $ pip install --exists-action=w -r requirements/edx/base.txt
    $ pip install --exists-action=w -r requirements/edx/paver.txt
    $ pip install --exists-action=w -r requirements/edx/post.txt
    $ deactivate
    
  • Install edX Sandbox’s Python dependencies and fix ownerships:

    $ . ../venvs/edxapp-sandbox/bin/activate
    $ sudo pip install -r requirements/edx-sandbox/setuptools.txt
    $ sudo pip install -r requirements/edx-sandbox/base.txt --ignore-installed
    $ sudo pip install -r requirements/edx-sandbox/local.txt
    $ sudo pip install -r requirements/edx-sandbox/post.txt
    $ deactivate
    $ sudo chown -R sandbox:sandbox ../venvs/edxapp-sandbox/
    
  • Start databases:

    $ sudo /opt/bitnami/ctlscript.sh start mysql
    $ sudo /opt/bitnami/ctlscript.sh start mongodb
    $ sudo /opt/bitnami/ctlscript.sh start memcached
    $ sudo /opt/bitnami/ctlscript.sh start rabbitmq
    $ sudo /opt/bitnami/ctlscript.sh start elasticsearch
    
  • Update edX assets. Note that this may take a significant amount of time for each step:

    $ ../bin/edxapp-update-assets-cms
    $ ../bin/edxapp-update-assets-lms
    
  • Run edX migrations (this may also take some time):

    $ sudo ../bin/edxapp-migrate-cms
    $ sudo ../bin/edxapp-migrate-lms
    
  • Fix log file ownership:

    $ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var/log
    
  • Edit the /opt/bitnami/apps/edx/scripts/ctl-edx-celery-workers.sh script and replace this:

    start_worker() {
        ...
    }
    

    with this:

    start_worker() {
        EDX_HOME="/opt/bitnami/apps/edx"
        cd $EDX_HOME/edx-platform
        APP=$(echo $1 | cut -d. -f2)
        ADDITIONAL_PARAMS=""
        if [ "$APP" = "cms" ]; then
            ADDITIONAL_PARAMS="-I contentstore.tasks"
        fi
        if [ `id|sed -e s/uid=//g -e s/\(.*//g` -eq 0 ]; then
            su -s /bin/sh daemon -c "$EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &"
        else
            $EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &
        fi
        cd - > /dev/null
    }
    
  • Edit the lms.env.json file and edit the CELERY_QUEUES section so that it looks like this:

    "CELERY_QUEUES": [
        "edx.lms.core.low",
        "edx.lms.core.default",
        "edx.lms.core.high",
        "edx.lms.core.high_mem"
    ],
    
  • Start remaining services:

    $ sudo /opt/bitnami/ctlscript.sh start apache
    $ sudo /opt/bitnami/ctlscript.sh start edx
    

Your Bitnami edX stack should now be upgraded to Ficus.4. Confirm that the upgrade was successful by logging into edX and verifying the integrity of your data.

Last modification August 8, 2019