How we restored our data in AWX after upgrading
June 16, 2021
This is what I did after upgrading from version 2 to version 17 and all the configuration was lost. Paths in this tutorial might be different for you, depending on your configuration. This might also be a bit sketchy, but hopefully it can give you some ideas on how to continue to fix your installation.
Based on what I have read: You need to have the same secrets configured in your old and new inventory when running the installation playbook, otherwise the encrypted credentials cannot be opened.
- old database A (with your data) in directory a
- new database B (empty) in directory b
- dump A to some path
- rename B to free its name for the next step
- restore A with the old name of B in location b
- awx-manage migrate in the awx_task docker container
Figuring out where the data is and verifying that it is not migrated
sudo docker container ls
lists your containers. There were two postgres containers:
sudo docker exec -it awx_postgres bash # entering the docker container psql -U awx \list
There is the "awx" database. You can for example find users via
SELECT * FROM auth_user;
which was empty for me except for the admin.
From an older version there is also a container called "postgres".
Doing the same thing in there yields all of my old data. But
that database is outdated and some columns are therefore missing, like
After looking around, I found a Migration to add the column of the example: https://github.com/ansible/awx/blob/devel/awx/main/migrations/0120_galaxy_credentials.py, so I wanted to now figure out a way to run this and all other migrations.
Going back to the host machine (out of the docker bash) via Ctrl + D, you should check the path in your inventory file (cloned from the repo) or vars.yml (if you use one) that leads to the postgres data:
cat awx/installer/inventory | grep postgres_data postgres_data_dir="~/.awx/pgdocker cat awx/installer/vars.yml | grep postgres_data postgres_data_dir: /var/lib/awx/pgdocker
Is it correct? If not you might have also screwed up your upgrade procedure. If you use vars.yml for the installation the setting there takes priority over the inventory file.
Turned out in /var/lib/awx/pgdocker there are two postgres data locations:
/var/lib/awx/pgdocker$ sudo find . -type f -name postgresql.conf ./pgdata/postgresql.conf ./12/data/postgresql.conf
You might just compare the timestamps of those files to figure out which directory contains your database with your data.
Migrating the old data
I created a dump of the old databse and moved it to the directory of the new container (so that it is accessible from a bash in there). Depending on your container and path setup paths might be different!
sudo docker exec -it postgres bash # old container, old db pg_dump -U awx -d awx > /var/lib/postgresql/data/12/data/dump # ctrl + d exit container
the dump should now be in the postgres_data_dir on the host.
but you can put it anywhere if that doesn't work for you
In the new container restore that dump (Also see https://www.postgresqltutorial.com/postgresql-rename-database/)
sudo docker exec -it awx_postgres bash # new container, new db # preparation and backup psql -U awx \connect postgres SELECT pg_terminate_backend (pid) FROM pg_stat_activity WHERE datname = 'awx'; # closes all db connections ALTER DATABASE awx RENAME TO awx_empty; CREATE DATABASE awx; # resteoring the dump # ctrl + d exit psql psql -U awx -d awx < /var/lib/postgresql/data/dump # takes a few minutes
The new container now contains the old data, but in an outdated database format. The migrations need to be executed.
# leave awx_postgres and enter awx_task: sudo docker exec -it awx_task bash awx-manage migrate # output: Operations to perform: Apply all migrations: auth, conf, contenttypes, main, oauth3_provider, sessions, sites, social_django, sso, taggit Running migrations: Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying conf.0006_v331_ldap_group_type... OK ...
Since the connections to the db have been closed manually previously, I restart the docker service which restarts AWX.
sudo systemctl restart docker
I then was able to log in into the new version with my old account.