Upgrade Open edX

NOTE: Before proceeding to upgrade the Bitnami edX stack, ensure that you have performed a complete backup.

Upgrade edX Ginkgo or edX Hawthorn to Ironwood.1

The recommended approach to upgrade edX Ginkgo/Hawthorn to Ironwood.1 is to migrate the data to a fresh installation of the Bitnami edX Ironwood.1 stack, and copy data and settings to it.

In order to proceed, log in to your edX Ginkgo/Hawthorn machine and follow the steps below:

Create a directory where to store all the data:
```
$ mkdir edx-upgrade-data
```

Dump databases required by edX:

MySQL:

$ /opt/bitnami/mysql/bin/mysqldump -u root -p --databases bitnami_edxapp bitnami_edxapp_csmh bitnami_xqueue --add-drop-database > edx-upgrade-data/bitnami_edx_mysql.sql

MongoDB:

$ /opt/bitnami/mongodb/bin/mongodump --authenticationDatabase admin -h localhost -u root -d bitnami_edxapp --out edx-upgrade-data/bitnami_edx_mongodb
$ /opt/bitnami/mongodb/bin/mongodump --authenticationDatabase admin -h localhost -u root -d bitnami_cs_comments_service --out edx-upgrade-data/bitnami_edx_mongodb

Copy edX Platform configuration and data files (which contain uploads, courses, etc.). The commands below also clear the assets to reduce the size of the directory:
```
$ cp /opt/bitnami/apps/edx/conf/*.json edx-upgrade-data
$ cp -R /opt/bitnami/apps/edx/var edx-upgrade-data
$ rm -rf edx-upgrade-data/var/staticfiles/*
```

Copy edX XQueue configuration files:

$ cp /opt/bitnami/apps/edx/conf/*.json edx-upgrade-data

Copy edX Forum configuration files:

$ cp /opt/bitnami/apps/forum/conf/httpd-forum.conf edx-upgrade-data

Compress the generated files:

$ tar czf edx-upgrade-data.tar.gz edx-upgrade-data

Copy edx-upgrade-data.tar.gz over to the new edX Ironwood.1 machine.

Next, log in to your edX Ironwood.1 machine, and navigate to the directory where you copied edx-upgrade-data.tar.gz. Then, proceed with the following steps:

Extract the compressed file:
```
$ tar xzf edx-upgrade-data.tar.gz
```

Migrate databases:

MySQL:

$ /opt/bitnami/mysql/bin/mysql -u root -p < edx-upgrade-data/bitnami_edx_mysql.sql

MongoDB:

$ /opt/bitnami/mongodb/bin/mongorestore --authenticationDatabase admin -h localhost -u root --drop -d bitnami_edxapp edx-upgrade-data/bitnami_edx_mongodb/bitnami_edxapp
$ /opt/bitnami/mongodb/bin/mongorestore --authenticationDatabase admin -h localhost -u root --drop -d bitnami_cs_comments_service edx-upgrade-data/bitnami_edx_mongodb/bitnami_cs_comments_service

Migrate edX Platform data:
- Replace the edX Platform data directory with the one copied from your edX Ginkgo/Hawthorn installation:
```
$ sudo rm -rf /opt/bitnami/apps/edx/var
$ sudo cp -R edx-upgrade-data/var /opt/bitnami/apps/edx/var
$ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var
```
- (Optional) Review edX Platform configuration for custom changes in the previous installation (e.g. themes, SMTP configuration, etc.):
  - LMS configuration files: /opt/bitnami/apps/edx/conf/lms.auth.json and /opt/bitnami/apps/edx/conf/lms.env.json.
  - CMS configuration files: /opt/bitnami/apps/edx/conf/cms.auth.json and /opt/bitnami/apps/edx/conf/cms.env.json.
- Drop tables used by djcelery (which should already be empty in your Ginkgo data):
  
  NOTE: Only run the command below if you are upgrading from edX Ginkgo to Ironwood.1. It is not needed for edX Hawthorn to Ironwood.1 upgrades.
```
$ sudo /opt/bitnami/apps/edx/bin/python.edxapp manage.py lms drop_djcelery_tables --settings=bitnami
```
- Run edX Platform migrations, which will update your Ginkgo/Hawthorn data to be valid for Ironwood.1:
```
$ sudo /opt/bitnami/apps/edx/bin/edxapp-migrate-lms
$ sudo /opt/bitnami/apps/edx/bin/edxapp-migrate-cms
```
- Update assets for edX Platform:
```
$ sudo /opt/bitnami/apps/edx/bin/edxapp-update-assets-lms
$ sudo /opt/bitnami/apps/edx/bin/edxapp-update-assets-cms
```
Migrate edX XQueue data:
- (Optional) Review XQueue configuration for custom changes in your previous installation: /opt/bitnami/apps/xqueue/conf/xqueue.yml.
  
  NOTE: Since edX Hawthorn, a couple of major changes have happened: The XQueue configuration files xqueue.auth.json and xqueue.env.json have been merged into xqueue.yml, and it does not use RabbitMQ anymore.
- Run edX XQueue migrations, which will update your Ginkgo/Hawthorn data to be valid for Ironwood.1:
```
$ sudo /opt/bitnami/apps/xqueue/bin/django-admin.xqueue migrate --noinput
```

Restart Apache, edX Celery Workers and XQueue:

$ sudo /opt/bitnami/ctlscript.sh restart apache
$ sudo /opt/bitnami/ctlscript.sh restart edx
$ sudo /opt/bitnami/ctlscript.sh restart xqueue

Upgrade edX Ficus.4 to Gingko.2

IMPORTANT: In order to upgrade any edX Ficus version to edX Ginkgo, you must first upgrade to edX Ficus.4

To upgrade edX Ficus.4 to Gingko.2, follow these steps:

Load the Bitnami stack environment, if it is not already loaded:

$ sudo /opt/bitnami/use_edx
$ . /opt/bitnami/scripts/setenv.sh

Stop all edX services:
```
$ sudo /opt/bitnami/ctlscript.sh stop
```

Install the GCC compiler.

Debian/Ubuntu:

$ sudo apt-get update
$ sudo apt-get install gcc g++

RHEL/CentOS:

$ sudo yum group install "Development Tools"

Navigate to the application directory at /opt/bitnami/apps/edx/edx-platform and save the current state of the Git repository:
```
$ cd /opt/bitnami/apps/edx/edx-platform
$ git diff -- . ':!requirements' > ~/bitnami_edx-platform_changes.txt
```

Check out the Gingko.2 branch and apply the Bitnami changes to the Git repository:

$ git fetch
$ git checkout -f open-release/ginkgo.2
$ git apply ~/bitnami_edx-platform_changes.txt

Install the necessary modules:
```
$ npm install
```

Install edX’s Python dependencies in this order:

$ . ../venvs/edxapp/bin/activate
$ pip install --exists-action=w -r requirements/edx/setuptools.txt
$ pip install --exists-action=w -r requirements/edx/pre.txt
$ pip install --exists-action=w -r requirements/edx/github.txt
$ pip install --exists-action=w -r requirements/edx/local.txt
$ pip install --exists-action=w -r requirements/edx/base.txt
$ pip install --exists-action=w -r requirements/edx/paver.txt
$ pip install --exists-action=w -r requirements/edx/post.txt
$ deactivate

Install edX Sandbox’s Python dependencies and fix ownerships:

$ . ../venvs/edxapp-sandbox/bin/activate
$ sudo pip install -r requirements/edx-sandbox/setuptools.txt
$ sudo pip install -r requirements/edx-sandbox/base.txt --ignore-installed
$ sudo pip install -r requirements/edx-sandbox/local.txt
$ sudo pip install -r requirements/edx-sandbox/post.txt
$ deactivate
$ sudo chown -R sandbox:sandbox ../venvs/edxapp-sandbox/

Start databases:

$ sudo /opt/bitnami/ctlscript.sh start mysql
$ sudo /opt/bitnami/ctlscript.sh start mongodb
$ sudo /opt/bitnami/ctlscript.sh start memcached
$ sudo /opt/bitnami/ctlscript.sh start rabbitmq
$ sudo /opt/bitnami/ctlscript.sh start elasticsearch

Update edX assets. Note that this may take a significant amount of time for each step:
```
$ ../bin/edxapp-update-assets-cms
$ ../bin/edxapp-update-assets-lms
```

Drop the djcelery database tables:

$ sudo ../bin/python.edxapp manage.py lms drop_djcelery_tables --settings=aws

Run edX migrations (this may also take some time):

$ sudo ../bin/edxapp-migrate-cms
$ sudo ../bin/edxapp-migrate-lms

Fix log file ownership:

$ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var/log

Edit the /opt/bitnami/apps/edx/scripts/ctl-edx-celery-workers.sh script and replace this:

start_worker() {
    ...
}

with this:

start_worker() {
    EDX_HOME="/opt/bitnami/apps/edx"
    cd $EDX_HOME/edx-platform
    APP=$(echo $1 | cut -d. -f2)
    ADDITIONAL_PARAMS=""
    if [ "$APP" = "cms" ]; then
        ADDITIONAL_PARAMS="-I contentstore.tasks"
    fi
    if [ `id|sed -e s/uid=//g -e s/\(.*//g` -eq 0 ]; then
        su -s /bin/sh daemon -c "$EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &"
    else
        $EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &
    fi
    cd - > /dev/null
}

Edit the lms.env.json file and edit the CELERY_QUEUES section so that it looks like this:

"CELERY_QUEUES": [
    "edx.lms.core.low",
    "edx.lms.core.default",
    "edx.lms.core.high",
    "edx.lms.core.high_mem"
],
...

Upgrade xqueue:

$ cd /opt/bitnami/apps/xqueue/xqueue
$ git diff -- . ':!*requirements.txt' ':!xqueue/test_settings.py' > ~/bitnami_xqueue_changes.txt
$ git fetch
$ git checkout -f open-release/ginkgo.2
$ git apply ~/bitnami_xqueue_changes.txt

Migrate and update users:

$ sudo ../bin/django-admin.xqueue migrate --noinput
$ sudo ../bin/django-admin.xqueue update_users

Upgrade the comments service:

$ cd /opt/bitnami/apps/forum/cs_comments_service
$ git fetch
$ git checkout -f open-release/ginkgo.2
$ sed -i 's/^ruby ".*"$//' Gemfile
$ bundle install

Start the remaining services:

$ sudo /opt/bitnami/ctlscript.sh start apache
$ sudo /opt/bitnami/ctlscript.sh start edx
$ sudo /opt/bitnami/ctlscript.sh start xqueue

Your Bitnami edX stack should now be upgraded to Gingko.2. Confirm that the upgrade was successful by logging into edX and verifying the integrity of your data.

Upgrade edX Ficus to Ficus.4

To upgrade edX Ficus to Ficus.4, follow these steps:

Load the Bitnami stack environment, if it is not already loaded:

$ sudo /opt/bitnami/use_edx
$ . /opt/bitnami/scripts/setenv.sh

Stop all edX services:
```
$ sudo /opt/bitnami/ctlscript.sh stop
```

Install the GCC compiler.

Debian/Ubuntu:

$ sudo apt-get update
$ sudo apt-get install gcc g++

RHEL/CentOS:

$ sudo yum group install "Development Tools"

Navigate to the application directory at /opt/bitnami/apps/edx/edx-platform and save the current state of the Git repository:
```
$ cd /opt/bitnami/apps/edx/edx-platform
$ git diff -- . ':!requirements' > ~/bitnami_edx-platform_changes.txt
```

Check out the Ficus.4 branch and apply the Bitnami changes to the Git repository:

$ git fetch
$ git checkout -f open-release/ficus.4
$ git apply ~/bitnami_edx-platform_changes.txt

Install the necessary modules:
```
$ npm install
```

Install edX’s Python dependencies in this order:

$ . ../venvs/edxapp/bin/activate
$ pip install --exists-action=w -r requirements/edx/setuptools.txt
$ pip install --exists-action=w -r requirements/edx/pre.txt
$ pip install --exists-action=w -r requirements/edx/github.txt
$ pip install --exists-action=w -r requirements/edx/local.txt
$ pip install --exists-action=w -r requirements/edx/base.txt
$ pip install --exists-action=w -r requirements/edx/paver.txt
$ pip install --exists-action=w -r requirements/edx/post.txt
$ deactivate

Install edX Sandbox’s Python dependencies and fix ownerships:

$ . ../venvs/edxapp-sandbox/bin/activate
$ sudo pip install -r requirements/edx-sandbox/setuptools.txt
$ sudo pip install -r requirements/edx-sandbox/base.txt --ignore-installed
$ sudo pip install -r requirements/edx-sandbox/local.txt
$ sudo pip install -r requirements/edx-sandbox/post.txt
$ deactivate
$ sudo chown -R sandbox:sandbox ../venvs/edxapp-sandbox/

Start databases:

$ sudo /opt/bitnami/ctlscript.sh start mysql
$ sudo /opt/bitnami/ctlscript.sh start mongodb
$ sudo /opt/bitnami/ctlscript.sh start memcached
$ sudo /opt/bitnami/ctlscript.sh start rabbitmq
$ sudo /opt/bitnami/ctlscript.sh start elasticsearch

Update edX assets. Note that this may take a significant amount of time for each step:
```
$ ../bin/edxapp-update-assets-cms
$ ../bin/edxapp-update-assets-lms
```

Run edX migrations (this may also take some time):

$ sudo ../bin/edxapp-migrate-cms
$ sudo ../bin/edxapp-migrate-lms

Fix log file ownership:

$ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var/log

Edit the /opt/bitnami/apps/edx/scripts/ctl-edx-celery-workers.sh script and replace this:

start_worker() {
    ...
}

with this:

start_worker() {
    EDX_HOME="/opt/bitnami/apps/edx"
    cd $EDX_HOME/edx-platform
    APP=$(echo $1 | cut -d. -f2)
    ADDITIONAL_PARAMS=""
    if [ "$APP" = "cms" ]; then
        ADDITIONAL_PARAMS="-I contentstore.tasks"
    fi
    if [ `id|sed -e s/uid=//g -e s/\(.*//g` -eq 0 ]; then
        su -s /bin/sh daemon -c "$EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &"
    else
        $EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &
    fi
    cd - > /dev/null
}

Edit the lms.env.json file and edit the CELERY_QUEUES section so that it looks like this:

"CELERY_QUEUES": [
    "edx.lms.core.low",
    "edx.lms.core.default",
    "edx.lms.core.high",
    "edx.lms.core.high_mem"
],

Start remaining services:

$ sudo /opt/bitnami/ctlscript.sh start apache
$ sudo /opt/bitnami/ctlscript.sh start edx

Your Bitnami edX stack should now be upgraded to Ficus.4. Confirm that the upgrade was successful by logging into edX and verifying the integrity of your data.

Kubeapps

Kubernetes Tutorials

Documentation

Engineering Blog

Support

Training

Open Source

We're hiring

Blogs

Resources

Newsroom

About Us

Careers

Partners

Upgrade Open edX

Upgrade edX Ginkgo or edX Hawthorn to Ironwood.1

Upgrade edX Ficus.4 to Gingko.2

Upgrade edX Ficus to Ficus.4

Products

Projects

Solutions

Company

Legal

Support