Learn OpenStack · reference/known-issues
Known Issues — 2025.1 (Epoxy)
Released: April 2025. SLURP release — extended support, eligible for direct upgrade from 2024.1.
Upgrade Blockers
Nova DB Migration Fails When shadow_instance_extra Table Has Orphaned Rows
Symptom: nova-manage db sync exits with IntegrityError: Duplicate entry or ForeignKeyViolation during upgrade from 2024.2 to 2025.1.
Cause: The 2025.1 migration 085_drop_legacy_migrate_fields attempts to clean up shadow tables. Orphaned rows in shadow_instance_extra that reference non-existent instances in shadow_instances cause the FK constraint check to fail.
Workaround:
# 1. Take a full DB backup first
mysqldump -u root -p nova > /tmp/nova-pre-upgrade-$(date +%Y%m%d).sql
# 2. Identify orphaned shadow rows
mysql -u root -p nova -e "
SELECT id FROM shadow_instance_extra sie
WHERE NOT EXISTS (
SELECT 1 FROM shadow_instances si WHERE si.id = sie.instance_id
) LIMIT 20;"
# 3. Delete the orphaned rows
mysql -u root -p nova -e "
DELETE sie FROM shadow_instance_extra sie
WHERE NOT EXISTS (
SELECT 1 FROM shadow_instances si WHERE si.id = sie.instance_id
);"
# 4. Retry the DB migration
nova-manage db sync
Affected versions: 2024.2 → 2025.1 upgrade path only.
Status: Fixed in 2025.1.1. If upgrading directly, apply the SQL cleanup before running db sync.
Neutron DB Migration Stalls on Large port_binding_levels Tables
Symptom: neutron-db-manage upgrade heads hangs or takes more than 30 minutes on clouds with more than 100,000 port binding records.
Cause: The expand_add_port_forwarding_description migration adds a nullable column to portforwardings but also re-indexes port_binding_levels. On large tables this index rebuild locks the table and blocks in-flight API calls.
Workaround: Run the migration during a maintenance window. If the migration is stuck:
# Check if the migration is running (from another terminal)
mysql -u root -p neutron -e "SHOW FULL PROCESSLIST;" | grep -i "alter\|index"
# Do not kill it mid-migration — wait for it to complete
# To estimate progress, check the table row count
mysql -u root -p neutron -e "SELECT COUNT(*) FROM port_binding_levels;"
# After completion, verify
neutron-db-manage current
# Should show the heads hash
Affected versions: 2024.1 → 2025.1 and 2024.2 → 2025.1 upgrades on clouds with > 50,000 port binding records.
Status: Open. The migration is correct but performance is poor on large datasets.
Configuration Changes
Nova: [compute] cpu_dedicated_set Replaces vcpu_pin_set
Symptom: Nova compute refuses to start with Unknown option(s) [DEFAULT] vcpu_pin_set. Instances that previously had CPU pinning may lose their pinning after upgrade.
Cause: vcpu_pin_set was deprecated in Train and is removed in 2025.1. The replacement [compute] cpu_dedicated_set was added in Victoria.
Workaround:
# Find usages of the old option
grep -r "vcpu_pin_set" /etc/nova/
# Replace in nova.conf
# OLD:
# [DEFAULT]
# vcpu_pin_set = 0-3,8-11
#
# NEW:
# [compute]
# cpu_dedicated_set = 0-3,8-11
# Verify the new option is accepted
nova-manage config validate | grep cpu
# Restart nova-compute after config change
systemctl restart nova-compute
Affected versions: Any deployment upgrading from a release prior to 2024.2 that still uses the old option.
Status: Expected removal — migrate the config option before upgrading.
Keystone: [token] expiration Minimum Enforced at 300 Seconds
Symptom: Keystone service fails to start with ConfigFileValueError: The minimum value for [token] expiration is 300.
Cause: 2025.1 enforces a 300-second (5-minute) minimum for token expiration to prevent excessive token churn. Deployments that set a very short token life (e.g., 60s for security testing) will fail to start.
Workaround:
grep "expiration" /etc/keystone/keystone.conf
# Increase to at least 300 seconds
# [token]
# expiration = 3600
Affected versions: 2025.1 only.
Status: By design. The minimum is documented in the 2025.1 release notes.
Cinder: [DEFAULT] backup_driver Path Renamed
Symptom: Cinder backup service logs DriverNotFound: Could not load backup driver after upgrade.
Cause: Several backup driver import paths were reorganized in 2025.1. The old path cinder.backup.drivers.swift is now cinder.backup.drivers.swift.SwiftBackupDriver.
Workaround:
grep "backup_driver" /etc/cinder/cinder.conf
# Update to the full class path
# OLD: backup_driver = cinder.backup.drivers.swift
# NEW: backup_driver = cinder.backup.drivers.swift.SwiftBackupDriver
# OLD: backup_driver = cinder.backup.drivers.ceph
# NEW: backup_driver = cinder.backup.drivers.ceph.CephBackupDriver
systemctl restart cinder-backup
Affected versions: 2025.1 initial release.
Status: Fixed in 2025.1.1 with backward compatibility shim. Recommended to update the config anyway.
Deprecations Enforced
Nova: Legacy nova-network References Removed from DB Schema
All nova-network related tables and columns (which have been no-ops since Nova dropped nova-network in Ocata) are removed from the schema by the 2025.1 DB migrations. This is normally transparent but can fail if manual DB modifications accidentally re-introduced these columns.
# Check if any legacy columns exist before migration
mysql -u root -p nova -e "SHOW COLUMNS FROM instances LIKE '%network%';"
# Should return nothing — legacy columns should not exist
Neutron: linuxbridge_agent.ini Standalone File No Longer Loaded
Symptom: Linux Bridge agent ignores configuration from /etc/neutron/plugins/linuxbridge/linuxbridge_agent.ini. Settings appear to reset to defaults.
Cause: The standalone config file loading was deprecated in 2023.2. In 2025.1, all ML2 Linux Bridge configuration must be in /etc/neutron/plugins/ml2/ml2_conf.ini or the main neutron.conf.
Workaround:
# Merge linuxbridge_agent.ini settings into ml2_conf.ini
cat /etc/neutron/plugins/linuxbridge/linuxbridge_agent.ini
# Move [linux_bridge], [vxlan], [securitygroup] sections to:
# /etc/neutron/plugins/ml2/ml2_conf.ini
# Restart the agent
systemctl restart neutron-linuxbridge-agent
Affected versions: 2025.1 and later.
Status: Expected — complete the migration before upgrading.
Known Bugs
Nova Live Migration Fails with "Timed out waiting for running domain" on Large Instances
Symptom: Live migrations of instances with > 128 GB RAM fail with MigrationError: Timed out waiting for running domain even when sufficient bandwidth is available.
Cause: The default libvirt migration completion timeout (libvirt_live_migration_completion_timeout) is calculated as a function of instance memory size. A bug in the 2025.1.0 formula applies an incorrect multiplier, yielding timeouts that are too short for instances > 128 GB.
Workaround:
# Increase the timeout explicitly in nova.conf on compute nodes
# [libvirt]
# live_migration_completion_timeout = 600 # seconds (default formula gives ~120 for 128 GB+)
# Or disable the timeout (use with caution on loaded hosts)
# [libvirt]
# live_migration_completion_timeout = 0 # 0 = no timeout
# Apply on the source compute node before retrying migration
systemctl restart nova-compute
openstack server migrate --live-migration --host <DEST_HOST> <SERVER_ID>
Affected versions: 2025.1.0.
Status: Fixed in 2025.1.1. Upgrade to 2025.1.1 or apply the workaround.
Neutron OVN: Router Gateway Port Binding Fails After Controller Restart
Symptom: After restarting ovn-northd or neutron-server, router gateway ports show binding:vif_type=binding_failed. North-south traffic through the affected routers drops.
Cause: A race condition in the OVN northd restart sequence causes some chassis-binding entries to be written before the chassis registers with the SB database. The gateway port then has no valid chassis assignment.
Workaround:
# Identify affected gateway ports
openstack port list --device-owner network:router_gateway -f json | \
jq '.[] | select(."Binding VIF Type" == "binding_failed") | .ID'
# Force re-bind by updating the port (triggers rebinding logic)
for PORT_ID in <AFFECTED_PORT_IDS>; do
openstack port set --host "" "$PORT_ID"
sleep 2
openstack port set --host <GATEWAY_CHASSIS_HOSTNAME> "$PORT_ID"
done
# Or trigger a full Neutron → OVN resync (more disruptive)
# Restart neutron-server on all controller nodes (rolling)
systemctl restart neutron-server
Affected versions: 2025.1.0.
Status: Fixed in 2025.1.1.
Glance: Image Download Corrupted When Store Backend Uses Chunked Encoding
Symptom: Downloading images from certain Glance store backends (specifically Swift with chunked-encoding enabled) produces corrupted image files. qemu-img check reports errors on downloaded images.
Cause: A regression in the Glance image download pipeline does not correctly handle the final chunk in a chunked-transfer response, truncating the last chunksize bytes of the image.
Workaround:
# Disable chunked encoding in the Swift store
# /etc/glance/glance-api.conf
# [glance_store]
# swift_enable_snet = false
# swift_store_large_object_size = 5120 # use standard PUT for small images
# Restart Glance
systemctl restart glance-api
# Verify image integrity after download
openstack image save <IMAGE_ID> --file /tmp/test-image.raw
qemu-img check /tmp/test-image.raw
Affected versions: 2025.1.0.
Status: Fixed in 2025.1.1.
Security Advisories
OSSA-2025-001: Nova Metadata API Allows Unauthenticated Instance Identity Spoofing
Symptom: An attacker with network access to the Nova metadata endpoint (169.254.169.254) could craft requests that retrieve metadata for a different instance in the same network.
Cause: The metadata proxy did not enforce strict IP-to-instance mapping when multiple instances shared a Neutron allowed_address_pair entry pointing to the same IP.
Workaround:
# Ensure metadata proxy uses per-instance isolation (enabled by default in OVN)
# For OVS deployments, verify:
grep "metadata_proxy_shared_secret" /etc/nova/nova.conf
grep "metadata_proxy_shared_secret" /etc/neutron/neutron.conf
# Both must match and be a strong random secret
# Generate a new shared secret if not set
python3 -c "import secrets; print(secrets.token_hex(32))"
# Set nova.conf [neutron] metadata_proxy_shared_secret = <value>
# Set neutron.conf [DEFAULT] metadata_proxy_shared_secret = <value>
# Restart nova-api-metadata and neutron-metadata-agent
# Verify metadata proxy is in router namespace (not global)
# /etc/neutron/metadata_agent.ini
# [DEFAULT]
# nova_metadata_host = <NOVA_API_HOST>
# metadata_proxy_shared_secret = <SAME_SECRET>
Affected versions: 2024.1 through 2025.1.0.
Status: Fixed in 2025.1.1 and 2024.2.2. Reference: OSSA-2025-001.
Performance Notes
Nova Scheduler Latency Increases with > 500 Compute Nodes
After upgrading to 2025.1, operators with more than 500 compute nodes report scheduler latency increases of 20-40% during peak scheduling periods.
Cause: The new placement candidate scoring algorithm in 2025.1 introduced an additional database query per resource provider during the weighting phase. With many providers, this serialized query pattern adds measurable latency.
Workaround:
# Increase nova-scheduler workers
# /etc/nova/nova.conf
# [DEFAULT]
# scheduler_workers = 8 # default is auto (CPU count)
# Cache placement results more aggressively
# [placement]
# region_name = RegionOne # ensure this is set correctly for caching
# Enable the scheduler cache
# [scheduler]
# discover_hosts_in_cells_interval = 300 # default 0 (disabled)
systemctl restart nova-scheduler
The full fix is in 2025.1.2. Increasing scheduler workers reduces the impact.