Upgrade Learnings — Real Experience

Hard-won knowledge from real upgrades.

The Biggest Gotcha: Stock Inventory Must Be Updated

Every release can add new inventory groups. Using the old inventory causes fatal precheck errors.

2025.2 added: neutron-rpc-server, neutron-periodic-worker, neutron-ovn-maintenance-worker. Using the 2025.1 inventory caused: 'dict object' has no attribute 'neutron-rpc-server'.

Fix: Always pull the new stock inventory before upgrading and re-apply your host customizations:

source /opt/kolla-venv/bin/activate
KOLLA_SHARE="/opt/kolla-venv/share/kolla-ansible"

# Save your current host assignments
head -30 inventory/hosts > /tmp/my-hosts.txt

# Get new stock inventory
cp "$KOLLA_SHARE/ansible/inventory/all-in-one" inventory/hosts
# or: cp "$KOLLA_SHARE/ansible/inventory/multinode" inventory/hosts

# Re-apply your host assignments at the top
# (edit inventory/hosts, replace localhost with your hosts)

Breaking Changes by Release

2025.1 → 2025.2

• redis → valkey — inventory group renamed, enable_redis → enable_valkey in globals
• venus removed — monitoring service and inventory group gone
• VMware support removed — vmware_nsxv, vmware_nsxv3, vmware_nsxp, vmware_dvs neutron plugins removed, along with related glance/cinder VMware backends
• New inventory groups — neutron-rpc-server, neutron-periodic-worker, neutron-ovn-maintenance-worker

How to Discover Breaking Changes

# Diff old vs new stock globals
diff globals.yml.stock-old "$KOLLA_SHARE/etc_examples/kolla/globals.yml"

# Diff old vs new stock inventory (look for new groups)
diff <(grep '^\[' inventory/hosts.old | sort) \
     <(grep '^\[' "$KOLLA_SHARE/ansible/inventory/all-in-one" | sort)

MariaDB Backup Before Upgrade

kolla-ansible mariadb-backup requires enable_mariabackup: "yes" in globals.yml. Without it, the backup playbook is a no-op.

For manual backup without mariabackup:

docker exec mariadb mariadb-dump --all-databases --single-transaction > backup.sql
# or with podman:
podman exec mariadb mariadb-dump --all-databases --single-transaction > backup.sql

Password Merge is Safe

kolla-genpwd -p passwords.yml only fills empty fields — it preserves existing passwords. Safe to run on upgrade. New releases may add new password fields that need to be filled.

Upgrade vs Deploy

Always use kolla-ansible upgrade, never deploy for upgrades. Upgrade handles: - Rolling restarts (one service at a time) - Database migrations - RPC version pinning/unpinning - Dependency-ordered service upgrades

Deploy does NOT do these things and can leave services in an inconsistent state.

Upgrade Order is Automatic

kolla-ansible upgrades services in dependency order:

common → mariadb → memcached → rabbitmq → keystone → glance →
placement → OVS/OVN → nova → neutron → heat → horizon

Never use --limit with upgrade — known bugs with partial upgrades.

Release Availability

Not all releases are on PyPI. Check git branches:

git ls-remote --heads https://opendev.org/openstack/kolla-ansible | grep stable

Install from git:

pip install 'kolla-ansible @ git+https://opendev.org/openstack/kolla-ansible@stable/2025.2'

Proven Upgrade Checklist

1. Read release notes for target release
2. Backup database (enable_mariabackup or manual dump)
3. Record current state (health-check, service versions)
4. Git commit current configs

5. pip install new kolla-ansible version
6. kolla-ansible install-deps  (Galaxy collections update too)

7. Pull new stock inventory → re-apply host customizations
8. Pull new stock globals → diff for breaking changes
9. kolla-genpwd -p passwords.yml  (fills new fields, preserves existing)
10. Copy updated files to /etc/kolla/

11. kolla-ansible prechecks -i inventory/hosts
12. kolla-ansible pull -i inventory/hosts
13. kolla-ansible upgrade -i inventory/hosts
14. kolla-ansible post-deploy -i inventory/hosts

15. Verify: openstack compute service list, network agent list
16. Git commit updated configs