# Known Issues — 2025.1 (Epoxy)

Released: April 2025. SLURP release — extended support, eligible for direct upgrade from 2024.1.

---

## Upgrade Blockers

### Nova DB Migration Fails When shadow_instance_extra Table Has Orphaned Rows

**Symptom:** `nova-manage db sync` exits with `IntegrityError: Duplicate entry` or `ForeignKeyViolation` during upgrade from 2024.2 to 2025.1.

**Cause:** The 2025.1 migration `085_drop_legacy_migrate_fields` attempts to clean up shadow tables. Orphaned rows in `shadow_instance_extra` that reference non-existent instances in `shadow_instances` cause the FK constraint check to fail.

**Workaround:**
```bash
# 1. Take a full DB backup first
mysqldump -u root -p nova > /tmp/nova-pre-upgrade-$(date +%Y%m%d).sql

# 2. Identify orphaned shadow rows
mysql -u root -p nova -e "
  SELECT id FROM shadow_instance_extra sie
  WHERE NOT EXISTS (
    SELECT 1 FROM shadow_instances si WHERE si.id = sie.instance_id
  ) LIMIT 20;"

# 3. Delete the orphaned rows
mysql -u root -p nova -e "
  DELETE sie FROM shadow_instance_extra sie
  WHERE NOT EXISTS (
    SELECT 1 FROM shadow_instances si WHERE si.id = sie.instance_id
  );"

# 4. Retry the DB migration
nova-manage db sync
```

**Affected versions:** 2024.2 → 2025.1 upgrade path only.

**Status:** Fixed in 2025.1.1. If upgrading directly, apply the SQL cleanup before running db sync.

---

### Neutron DB Migration Stalls on Large port_binding_levels Tables

**Symptom:** `neutron-db-manage upgrade heads` hangs or takes more than 30 minutes on clouds with more than 100,000 port binding records.

**Cause:** The `expand_add_port_forwarding_description` migration adds a nullable column to `portforwardings` but also re-indexes `port_binding_levels`. On large tables this index rebuild locks the table and blocks in-flight API calls.

**Workaround:** Run the migration during a maintenance window. If the migration is stuck:

```bash
# Check if the migration is running (from another terminal)
mysql -u root -p neutron -e "SHOW FULL PROCESSLIST;" | grep -i "alter\|index"

# Do not kill it mid-migration — wait for it to complete
# To estimate progress, check the table row count
mysql -u root -p neutron -e "SELECT COUNT(*) FROM port_binding_levels;"

# After completion, verify
neutron-db-manage current
# Should show the heads hash
```

**Affected versions:** 2024.1 → 2025.1 and 2024.2 → 2025.1 upgrades on clouds with > 50,000 port binding records.

**Status:** Open. The migration is correct but performance is poor on large datasets.

---

## Configuration Changes

### Nova: `[compute] cpu_dedicated_set` Replaces `vcpu_pin_set`

**Symptom:** Nova compute refuses to start with `Unknown option(s) [DEFAULT] vcpu_pin_set`. Instances that previously had CPU pinning may lose their pinning after upgrade.

**Cause:** `vcpu_pin_set` was deprecated in Train and is removed in 2025.1. The replacement `[compute] cpu_dedicated_set` was added in Victoria.

**Workaround:**
```bash
# Find usages of the old option
grep -r "vcpu_pin_set" /etc/nova/

# Replace in nova.conf
# OLD:
#   [DEFAULT]
#   vcpu_pin_set = 0-3,8-11
#
# NEW:
#   [compute]
#   cpu_dedicated_set = 0-3,8-11

# Verify the new option is accepted
nova-manage config validate | grep cpu

# Restart nova-compute after config change
systemctl restart nova-compute
```

**Affected versions:** Any deployment upgrading from a release prior to 2024.2 that still uses the old option.

**Status:** Expected removal — migrate the config option before upgrading.

---

### Keystone: `[token] expiration` Minimum Enforced at 300 Seconds

**Symptom:** Keystone service fails to start with `ConfigFileValueError: The minimum value for [token] expiration is 300`.

**Cause:** 2025.1 enforces a 300-second (5-minute) minimum for token expiration to prevent excessive token churn. Deployments that set a very short token life (e.g., 60s for security testing) will fail to start.

**Workaround:**
```bash
grep "expiration" /etc/keystone/keystone.conf

# Increase to at least 300 seconds
# [token]
# expiration = 3600
```

**Affected versions:** 2025.1 only.

**Status:** By design. The minimum is documented in the 2025.1 release notes.

---

### Cinder: `[DEFAULT] backup_driver` Path Renamed

**Symptom:** Cinder backup service logs `DriverNotFound: Could not load backup driver` after upgrade.

**Cause:** Several backup driver import paths were reorganized in 2025.1. The old path `cinder.backup.drivers.swift` is now `cinder.backup.drivers.swift.SwiftBackupDriver`.

**Workaround:**
```bash
grep "backup_driver" /etc/cinder/cinder.conf

# Update to the full class path
# OLD: backup_driver = cinder.backup.drivers.swift
# NEW: backup_driver = cinder.backup.drivers.swift.SwiftBackupDriver

# OLD: backup_driver = cinder.backup.drivers.ceph
# NEW: backup_driver = cinder.backup.drivers.ceph.CephBackupDriver

systemctl restart cinder-backup
```

**Affected versions:** 2025.1 initial release.

**Status:** Fixed in 2025.1.1 with backward compatibility shim. Recommended to update the config anyway.

---

## Deprecations Enforced

### Nova: Legacy `nova-network` References Removed from DB Schema

All `nova-network` related tables and columns (which have been no-ops since Nova dropped nova-network in Ocata) are removed from the schema by the 2025.1 DB migrations. This is normally transparent but can fail if manual DB modifications accidentally re-introduced these columns.

```bash
# Check if any legacy columns exist before migration
mysql -u root -p nova -e "SHOW COLUMNS FROM instances LIKE '%network%';"
# Should return nothing — legacy columns should not exist
```

---

### Neutron: `linuxbridge_agent.ini` Standalone File No Longer Loaded

**Symptom:** Linux Bridge agent ignores configuration from `/etc/neutron/plugins/linuxbridge/linuxbridge_agent.ini`. Settings appear to reset to defaults.

**Cause:** The standalone config file loading was deprecated in 2023.2. In 2025.1, all ML2 Linux Bridge configuration must be in `/etc/neutron/plugins/ml2/ml2_conf.ini` or the main `neutron.conf`.

**Workaround:**
```bash
# Merge linuxbridge_agent.ini settings into ml2_conf.ini
cat /etc/neutron/plugins/linuxbridge/linuxbridge_agent.ini

# Move [linux_bridge], [vxlan], [securitygroup] sections to:
# /etc/neutron/plugins/ml2/ml2_conf.ini

# Restart the agent
systemctl restart neutron-linuxbridge-agent
```

**Affected versions:** 2025.1 and later.

**Status:** Expected — complete the migration before upgrading.

---

## Known Bugs

### Nova Live Migration Fails with "Timed out waiting for running domain" on Large Instances

**Symptom:** Live migrations of instances with > 128 GB RAM fail with `MigrationError: Timed out waiting for running domain` even when sufficient bandwidth is available.

**Cause:** The default libvirt migration completion timeout (`libvirt_live_migration_completion_timeout`) is calculated as a function of instance memory size. A bug in the 2025.1.0 formula applies an incorrect multiplier, yielding timeouts that are too short for instances > 128 GB.

**Workaround:**
```bash
# Increase the timeout explicitly in nova.conf on compute nodes
# [libvirt]
# live_migration_completion_timeout = 600  # seconds (default formula gives ~120 for 128 GB+)

# Or disable the timeout (use with caution on loaded hosts)
# [libvirt]
# live_migration_completion_timeout = 0   # 0 = no timeout

# Apply on the source compute node before retrying migration
systemctl restart nova-compute
openstack server migrate --live-migration --host <DEST_HOST> <SERVER_ID>
```

**Affected versions:** 2025.1.0.

**Status:** Fixed in 2025.1.1. Upgrade to 2025.1.1 or apply the workaround.

---

### Neutron OVN: Router Gateway Port Binding Fails After Controller Restart

**Symptom:** After restarting ovn-northd or neutron-server, router gateway ports show `binding:vif_type=binding_failed`. North-south traffic through the affected routers drops.

**Cause:** A race condition in the OVN northd restart sequence causes some chassis-binding entries to be written before the chassis registers with the SB database. The gateway port then has no valid chassis assignment.

**Workaround:**
```bash
# Identify affected gateway ports
openstack port list --device-owner network:router_gateway -f json | \
  jq '.[] | select(."Binding VIF Type" == "binding_failed") | .ID'

# Force re-bind by updating the port (triggers rebinding logic)
for PORT_ID in <AFFECTED_PORT_IDS>; do
  openstack port set --host "" "$PORT_ID"
  sleep 2
  openstack port set --host <GATEWAY_CHASSIS_HOSTNAME> "$PORT_ID"
done

# Or trigger a full Neutron → OVN resync (more disruptive)
# Restart neutron-server on all controller nodes (rolling)
systemctl restart neutron-server
```

**Affected versions:** 2025.1.0.

**Status:** Fixed in 2025.1.1.

---

### Glance: Image Download Corrupted When Store Backend Uses Chunked Encoding

**Symptom:** Downloading images from certain Glance store backends (specifically Swift with chunked-encoding enabled) produces corrupted image files. `qemu-img check` reports errors on downloaded images.

**Cause:** A regression in the Glance image download pipeline does not correctly handle the final chunk in a chunked-transfer response, truncating the last `chunksize` bytes of the image.

**Workaround:**
```bash
# Disable chunked encoding in the Swift store
# /etc/glance/glance-api.conf
# [glance_store]
# swift_enable_snet = false
# swift_store_large_object_size = 5120   # use standard PUT for small images

# Restart Glance
systemctl restart glance-api

# Verify image integrity after download
openstack image save <IMAGE_ID> --file /tmp/test-image.raw
qemu-img check /tmp/test-image.raw
```

**Affected versions:** 2025.1.0.

**Status:** Fixed in 2025.1.1.

---

## Security Advisories

### OSSA-2025-001: Nova Metadata API Allows Unauthenticated Instance Identity Spoofing

**Symptom:** An attacker with network access to the Nova metadata endpoint (169.254.169.254) could craft requests that retrieve metadata for a different instance in the same network.

**Cause:** The metadata proxy did not enforce strict IP-to-instance mapping when multiple instances shared a Neutron allowed_address_pair entry pointing to the same IP.

**Workaround:**
```bash
# Ensure metadata proxy uses per-instance isolation (enabled by default in OVN)
# For OVS deployments, verify:
grep "metadata_proxy_shared_secret" /etc/nova/nova.conf
grep "metadata_proxy_shared_secret" /etc/neutron/neutron.conf
# Both must match and be a strong random secret

# Generate a new shared secret if not set
python3 -c "import secrets; print(secrets.token_hex(32))"
# Set nova.conf [neutron] metadata_proxy_shared_secret = <value>
# Set neutron.conf [DEFAULT] metadata_proxy_shared_secret = <value>
# Restart nova-api-metadata and neutron-metadata-agent

# Verify metadata proxy is in router namespace (not global)
# /etc/neutron/metadata_agent.ini
# [DEFAULT]
# nova_metadata_host = <NOVA_API_HOST>
# metadata_proxy_shared_secret = <SAME_SECRET>
```

**Affected versions:** 2024.1 through 2025.1.0.

**Status:** Fixed in 2025.1.1 and 2024.2.2. Reference: OSSA-2025-001.

---

## Performance Notes

### Nova Scheduler Latency Increases with > 500 Compute Nodes

After upgrading to 2025.1, operators with more than 500 compute nodes report scheduler latency increases of 20-40% during peak scheduling periods.

**Cause:** The new placement candidate scoring algorithm in 2025.1 introduced an additional database query per resource provider during the weighting phase. With many providers, this serialized query pattern adds measurable latency.

**Workaround:**
```bash
# Increase nova-scheduler workers
# /etc/nova/nova.conf
# [DEFAULT]
# scheduler_workers = 8  # default is auto (CPU count)

# Cache placement results more aggressively
# [placement]
# region_name = RegionOne  # ensure this is set correctly for caching

# Enable the scheduler cache
# [scheduler]
# discover_hosts_in_cells_interval = 300  # default 0 (disabled)

systemctl restart nova-scheduler
```

The full fix is in 2025.1.2. Increasing scheduler workers reduces the impact.