Upgrade Open edX
NOTE: Before proceeding to upgrade the Bitnami edX stack, ensure that you have performed a complete backup.
Upgrade edX Ginkgo or edX Hawthorn to Ironwood.1
The recommended approach to upgrade edX Ginkgo/Hawthorn to Ironwood.1 is to migrate the data to a fresh installation of the Bitnami edX Ironwood.1 stack, and copy data and settings to it.
In order to proceed, log in to your edX Ginkgo/Hawthorn machine and follow the steps below:
Create a directory where to store all the data:
$ mkdir edx-upgrade-data
Dump databases required by edX:
MySQL:
$ /opt/bitnami/mysql/bin/mysqldump -u root -p --databases bitnami_edxapp bitnami_edxapp_csmh bitnami_xqueue --add-drop-database > edx-upgrade-data/bitnami_edx_mysql.sql
MongoDB:
$ /opt/bitnami/mongodb/bin/mongodump --authenticationDatabase admin -h localhost -u root -d bitnami_edxapp --out edx-upgrade-data/bitnami_edx_mongodb $ /opt/bitnami/mongodb/bin/mongodump --authenticationDatabase admin -h localhost -u root -d bitnami_cs_comments_service --out edx-upgrade-data/bitnami_edx_mongodb
Copy edX Platform configuration and data files (which contain uploads, courses, etc.). The commands below also clear the assets to reduce the size of the directory:
$ cp /opt/bitnami/apps/edx/conf/*.json edx-upgrade-data $ cp -R /opt/bitnami/apps/edx/var edx-upgrade-data $ rm -rf edx-upgrade-data/var/staticfiles/*
Copy edX XQueue configuration files:
$ cp /opt/bitnami/apps/edx/conf/*.json edx-upgrade-data
Copy edX Forum configuration files:
$ cp /opt/bitnami/apps/forum/conf/httpd-forum.conf edx-upgrade-data
Compress the generated files:
$ tar czf edx-upgrade-data.tar.gz edx-upgrade-data
Copy edx-upgrade-data.tar.gz over to the new edX Ironwood.1 machine.
Next, log in to your edX Ironwood.1 machine, and navigate to the directory where you copied edx-upgrade-data.tar.gz. Then, proceed with the following steps:
Extract the compressed file:
$ tar xzf edx-upgrade-data.tar.gz
Migrate databases:
MySQL:
$ /opt/bitnami/mysql/bin/mysql -u root -p < edx-upgrade-data/bitnami_edx_mysql.sql
MongoDB:
$ /opt/bitnami/mongodb/bin/mongorestore --authenticationDatabase admin -h localhost -u root --drop -d bitnami_edxapp edx-upgrade-data/bitnami_edx_mongodb/bitnami_edxapp $ /opt/bitnami/mongodb/bin/mongorestore --authenticationDatabase admin -h localhost -u root --drop -d bitnami_cs_comments_service edx-upgrade-data/bitnami_edx_mongodb/bitnami_cs_comments_service
Migrate edX Platform data:
Replace the edX Platform data directory with the one copied from your edX Ginkgo/Hawthorn installation:
$ sudo rm -rf /opt/bitnami/apps/edx/var $ sudo cp -R edx-upgrade-data/var /opt/bitnami/apps/edx/var $ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var
(Optional) Review edX Platform configuration for custom changes in the previous installation (e.g. themes, SMTP configuration, etc.):
- LMS configuration file: /opt/bitnami/apps/edx/conf/lms.yml.
- CMS configuration file: /opt/bitnami/apps/edx/conf/cms.yml.
NOTE: Since edX Koa, a couple of major changes have happened: The LMS and CMS configuration files lms.auth.json, lms.env.json, cms.auth.json and cms.env.json have been merged into lms.yml and cms.yml.
Drop tables used by djcelery (which should already be empty in your Ginkgo data):
NOTE: Only run the command below if you are upgrading from edX Ginkgo to Ironwood.1. It is not needed for edX Hawthorn to Ironwood.1 upgrades.
$ sudo /opt/bitnami/apps/edx/bin/python.edxapp manage.py lms drop_djcelery_tables --settings=bitnami
Run edX Platform migrations, which will update your Ginkgo/Hawthorn data to be valid for Ironwood.1:
$ sudo /opt/bitnami/apps/edx/bin/edxapp-migrate-lms $ sudo /opt/bitnami/apps/edx/bin/edxapp-migrate-cms
Update assets for edX Platform:
$ sudo /opt/bitnami/apps/edx/bin/edxapp-update-assets-lms $ sudo /opt/bitnami/apps/edx/bin/edxapp-update-assets-cms
Migrate edX XQueue data:
(Optional) Review XQueue configuration for custom changes in your previous installation: /opt/bitnami/apps/xqueue/conf/xqueue.yml.
NOTE: Since edX Hawthorn, a couple of major changes have happened: The XQueue configuration files xqueue.auth.json and xqueue.env.json have been merged into xqueue.yml, and it does not use RabbitMQ anymore.
Run edX XQueue migrations, which will update your Ginkgo/Hawthorn data to be valid for Ironwood.1:
$ sudo /opt/bitnami/apps/xqueue/bin/django-admin.xqueue migrate --noinput
Restart Apache, edX Celery Workers and XQueue:
$ sudo /opt/bitnami/ctlscript.sh restart apache $ sudo /opt/bitnami/ctlscript.sh restart edx $ sudo /opt/bitnami/ctlscript.sh restart xqueue
Upgrade edX Ficus.4 to Gingko.2
IMPORTANT: In order to upgrade any edX Ficus version to edX Ginkgo, you must first upgrade to edX Ficus.4
To upgrade edX Ficus.4 to Gingko.2, follow these steps:
Load the Bitnami stack environment, if it is not already loaded:
$ sudo /opt/bitnami/use_edx $ . /opt/bitnami/scripts/setenv.sh
Stop all edX services:
$ sudo /opt/bitnami/ctlscript.sh stop
Install the GCC compiler.
Debian/Ubuntu:
$ sudo apt-get update $ sudo apt-get install gcc g++
RHEL/CentOS:
$ sudo yum group install "Development Tools"
Navigate to the application directory at /opt/bitnami/apps/edx/edx-platform and save the current state of the Git repository:
$ cd /opt/bitnami/apps/edx/edx-platform $ git diff -- . ':!requirements' > ~/bitnami_edx-platform_changes.txt
Check out the Gingko.2 branch and apply the Bitnami changes to the Git repository:
$ git fetch $ git checkout -f open-release/ginkgo.2 $ git apply ~/bitnami_edx-platform_changes.txt
Install the necessary modules:
$ npm install
Install edX’s Python dependencies in this order:
$ . ../venvs/edxapp/bin/activate $ pip install --exists-action=w -r requirements/edx/setuptools.txt $ pip install --exists-action=w -r requirements/edx/pre.txt $ pip install --exists-action=w -r requirements/edx/github.txt $ pip install --exists-action=w -r requirements/edx/local.txt $ pip install --exists-action=w -r requirements/edx/base.txt $ pip install --exists-action=w -r requirements/edx/paver.txt $ pip install --exists-action=w -r requirements/edx/post.txt $ deactivate
Install edX Sandbox’s Python dependencies and fix ownerships:
$ . ../venvs/edxapp-sandbox/bin/activate $ sudo pip install -r requirements/edx-sandbox/setuptools.txt $ sudo pip install -r requirements/edx-sandbox/base.txt --ignore-installed $ sudo pip install -r requirements/edx-sandbox/local.txt $ sudo pip install -r requirements/edx-sandbox/post.txt $ deactivate $ sudo chown -R sandbox:sandbox ../venvs/edxapp-sandbox/
Start databases:
$ sudo /opt/bitnami/ctlscript.sh start mysql $ sudo /opt/bitnami/ctlscript.sh start mongodb $ sudo /opt/bitnami/ctlscript.sh start memcached $ sudo /opt/bitnami/ctlscript.sh start rabbitmq $ sudo /opt/bitnami/ctlscript.sh start elasticsearch
Update edX assets. Note that this may take a significant amount of time for each step:
$ ../bin/edxapp-update-assets-cms $ ../bin/edxapp-update-assets-lms
Drop the djcelery database tables:
$ sudo ../bin/python.edxapp manage.py lms drop_djcelery_tables --settings=aws
Run edX migrations (this may also take some time):
$ sudo ../bin/edxapp-migrate-cms $ sudo ../bin/edxapp-migrate-lms
Fix log file ownership:
$ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var/log
Edit the /opt/bitnami/apps/edx/scripts/ctl-edx-celery-workers.sh script and replace this:
start_worker() { ... }
with this:
start_worker() { EDX_HOME="/opt/bitnami/apps/edx" cd $EDX_HOME/edx-platform APP=$(echo $1 | cut -d. -f2) ADDITIONAL_PARAMS="" if [ "$APP" = "cms" ]; then ADDITIONAL_PARAMS="-I contentstore.tasks" fi if [ `id|sed -e s/uid=//g -e s/\(.*//g` -eq 0 ]; then su -s /bin/sh daemon -c "$EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &" else $EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 & fi cd - > /dev/null }
Edit the lms.env.json file and edit the CELERY_QUEUES section so that it looks like this:
"CELERY_QUEUES": [ "edx.lms.core.low", "edx.lms.core.default", "edx.lms.core.high", "edx.lms.core.high_mem" ], ...
Upgrade xqueue:
$ cd /opt/bitnami/apps/xqueue/xqueue $ git diff -- . ':!*requirements.txt' ':!xqueue/test_settings.py' > ~/bitnami_xqueue_changes.txt $ git fetch $ git checkout -f open-release/ginkgo.2 $ git apply ~/bitnami_xqueue_changes.txt
Migrate and update users:
$ sudo ../bin/django-admin.xqueue migrate --noinput $ sudo ../bin/django-admin.xqueue update_users
Upgrade the comments service:
$ cd /opt/bitnami/apps/forum/cs_comments_service $ git fetch $ git checkout -f open-release/ginkgo.2 $ sed -i 's/^ruby ".*"$//' Gemfile $ bundle install
Start the remaining services:
$ sudo /opt/bitnami/ctlscript.sh start apache $ sudo /opt/bitnami/ctlscript.sh start edx $ sudo /opt/bitnami/ctlscript.sh start xqueue
Your Bitnami edX stack should now be upgraded to Gingko.2. Confirm that the upgrade was successful by logging into edX and verifying the integrity of your data.
Upgrade edX Ficus to Ficus.4
To upgrade edX Ficus to Ficus.4, follow these steps:
Load the Bitnami stack environment, if it is not already loaded:
$ sudo /opt/bitnami/use_edx $ . /opt/bitnami/scripts/setenv.sh
Stop all edX services:
$ sudo /opt/bitnami/ctlscript.sh stop
Install the GCC compiler.
Debian/Ubuntu:
$ sudo apt-get update $ sudo apt-get install gcc g++
RHEL/CentOS:
$ sudo yum group install "Development Tools"
Navigate to the application directory at /opt/bitnami/apps/edx/edx-platform and save the current state of the Git repository:
$ cd /opt/bitnami/apps/edx/edx-platform $ git diff -- . ':!requirements' > ~/bitnami_edx-platform_changes.txt
Check out the Ficus.4 branch and apply the Bitnami changes to the Git repository:
$ git fetch $ git checkout -f open-release/ficus.4 $ git apply ~/bitnami_edx-platform_changes.txt
Install the necessary modules:
$ npm install
Install edX’s Python dependencies in this order:
$ . ../venvs/edxapp/bin/activate $ pip install --exists-action=w -r requirements/edx/setuptools.txt $ pip install --exists-action=w -r requirements/edx/pre.txt $ pip install --exists-action=w -r requirements/edx/github.txt $ pip install --exists-action=w -r requirements/edx/local.txt $ pip install --exists-action=w -r requirements/edx/base.txt $ pip install --exists-action=w -r requirements/edx/paver.txt $ pip install --exists-action=w -r requirements/edx/post.txt $ deactivate
Install edX Sandbox’s Python dependencies and fix ownerships:
$ . ../venvs/edxapp-sandbox/bin/activate $ sudo pip install -r requirements/edx-sandbox/setuptools.txt $ sudo pip install -r requirements/edx-sandbox/base.txt --ignore-installed $ sudo pip install -r requirements/edx-sandbox/local.txt $ sudo pip install -r requirements/edx-sandbox/post.txt $ deactivate $ sudo chown -R sandbox:sandbox ../venvs/edxapp-sandbox/
Start databases:
$ sudo /opt/bitnami/ctlscript.sh start mysql $ sudo /opt/bitnami/ctlscript.sh start mongodb $ sudo /opt/bitnami/ctlscript.sh start memcached $ sudo /opt/bitnami/ctlscript.sh start rabbitmq $ sudo /opt/bitnami/ctlscript.sh start elasticsearch
Update edX assets. Note that this may take a significant amount of time for each step:
$ ../bin/edxapp-update-assets-cms $ ../bin/edxapp-update-assets-lms
Run edX migrations (this may also take some time):
$ sudo ../bin/edxapp-migrate-cms $ sudo ../bin/edxapp-migrate-lms
Fix log file ownership:
$ sudo chown -R daemon:daemon /opt/bitnami/apps/edx/var/log
Edit the /opt/bitnami/apps/edx/scripts/ctl-edx-celery-workers.sh script and replace this:
start_worker() { ... }
with this:
start_worker() { EDX_HOME="/opt/bitnami/apps/edx" cd $EDX_HOME/edx-platform APP=$(echo $1 | cut -d. -f2) ADDITIONAL_PARAMS="" if [ "$APP" = "cms" ]; then ADDITIONAL_PARAMS="-I contentstore.tasks" fi if [ `id|sed -e s/uid=//g -e s/\(.*//g` -eq 0 ]; then su -s /bin/sh daemon -c "$EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 &" else $EDX_HOME/bin/python.edxapp ./manage.py $APP --settings=aws celery worker $ADDITIONAL_PARAMS --loglevel=info --queues=$1 --hostname=$1.%h --concurrency=$2 > $EDX_HOME/var/log/celery/$1.log 2>&1 & fi cd - > /dev/null }
Edit the lms.env.json file and edit the CELERY_QUEUES section so that it looks like this:
"CELERY_QUEUES": [ "edx.lms.core.low", "edx.lms.core.default", "edx.lms.core.high", "edx.lms.core.high_mem" ],
Start remaining services:
$ sudo /opt/bitnami/ctlscript.sh start apache $ sudo /opt/bitnami/ctlscript.sh start edx
Your Bitnami edX stack should now be upgraded to Ficus.4. Confirm that the upgrade was successful by logging into edX and verifying the integrity of your data.