▤ Stackers Network

Learn OpenStack · deploy/diagnose

Diagnose

Systematic troubleshooting for OpenStack issues. Given a symptom, investigate to find the root cause.

Diagnostic Process

1. Gather Context

# What was the last action taken?
git log --oneline -5

# What's the current state?
source /etc/kolla/admin-openrc.sh
openstack compute service list
openstack network agent list

2. Check Service Health

# Are all services reporting in?
openstack endpoint list --interface internal
openstack compute service list
openstack network agent list
openstack volume service list

# Any containers down?
ssh <host> docker ps --filter "label=kolla_service" --format "{{.Names}}: {{.Status}}"
ssh <host> docker ps --filter "label=kolla_service" --filter "status=restarting"

3. Read Logs

Most service logs are in /var/log/kolla/<service>/:

# Find the log volume
ssh <host> readlink -f /var/log/kolla
# Usually: /var/lib/docker/volumes/kolla_logs/_data

# Read logs for a specific service
ssh <host> tail -100 /var/log/kolla/<service>/<service>.log | grep -i error

# Container stdout (most containers don't log here, but some do)
ssh <host> docker logs --tail 100 kolla_<service>

4. Check Configuration

# What config is the service actually running with?
ssh <host> docker exec kolla_<service> cat /etc/<service>/<config_file>

# Generate configs without deploying (useful for comparing)
kolla-ansible genconfig -i inventory/hosts.yml

5. Check Infrastructure

# MariaDB cluster
ssh <control> docker exec mariadb mysql -u root -e "SHOW STATUS LIKE 'wsrep_cluster_size'"
ssh <control> docker exec mariadb mysql -u root -e "SHOW STATUS LIKE 'wsrep_ready'"

# RabbitMQ cluster
ssh <control> docker exec rabbitmq rabbitmqctl cluster_status

# OVN (if using OVN)
ssh <control> docker exec ovn_northd ovn-nbctl show

Common Issues and Solutions

Service Won't Start

  1. Check container logs: docker logs kolla_<service>
  2. Check config for invalid options: docker exec kolla_<service> cat /etc/<service>/<config>.conf
  3. Check dependencies: is the database reachable? is RabbitMQ up?
  4. Check ports: ss -tlnp | grep <port> — is another process binding?
  5. Try restarting just that container: docker restart kolla_<service>

API Returns 500

  1. Check the specific service logs (not the proxy/haproxy)
  2. Look for database connection errors — MariaDB might be down
  3. Check Keystone token validity — token may be expired or Keystone may be down
  4. Check RabbitMQ connectivity — service may not be able to reach the message queue

VMs Can't Reach Network

  1. Check OVN/OVS agent status: openstack network agent list
  2. Check bridge configuration: ovs-vsctl show on the host
  3. Check security groups: default group may block traffic
  4. Check floating IP association and external network setup
  5. Check that neutron_external_interface is UP and has no IP

Deploy or Reconfigure Failed

  1. Read the Ansible error — the failed task name tells you which service and which step
  2. Check host reachability: ansible -i inventory/hosts.yml all -m ping
  3. Check disk space: df -h on failed host
  4. Check Docker status: systemctl status docker on failed host
  5. Try running just the failed service: kolla-ansible deploy -i inventory/hosts.yml --tags <service>
  6. Do NOT run destroy — fix the issue and re-run deploy

MariaDB / Galera Issues

  1. Check cluster status: SHOW STATUS LIKE 'wsrep%'
  2. wsrep_cluster_size should equal number of control nodes
  3. wsrep_ready should be ON
  4. If split-brain: identify the node with most recent data, bootstrap from it
  5. Never force bootstrap unless you understand the cluster state
  6. Recovery: kolla-ansible mariadb-recovery -i inventory/hosts.yml

RabbitMQ Issues

  1. Check cluster status: rabbitmqctl cluster_status
  2. Check for network partitions in the output
  3. Check queue depths: rabbitmqctl list_queues name messages consumers
  4. Growing queues with 0 consumers = service consuming the queue is down
  5. If cluster is broken: may need to reset one node and rejoin

HAProxy / VIP Issues

  1. Check keepalived: docker logs kolla_keepalived
  2. Check VIP is assigned: ip addr show on control nodes — one should have the VIP
  3. Check HAProxy backends: curl -s http://<vip>:1984/stats (HAProxy stats if enabled)
  4. If VIP is floating between nodes: check for network issues between control nodes

Certificate / TLS Issues

  1. Check certificate expiry: openssl s_client -connect <vip>:5000 -servername <vip> </dev/null 2>/dev/null | openssl x509 -noout -dates
  2. Check CA trust: curl -v https://<vip>:5000/v3 — look for certificate errors
  3. Regenerate certificates: kolla-ansible certificates -i inventory/hosts.yml

Safety Rules

  • Never take remediation action without operator approval — present findings and recommended fix, let them decide
  • Log all diagnostic steps — if the fix doesn't work, you need to know what you already tried
  • If uncertain about root cause, say so — don't guess and don't apply random fixes
  • Don't restart services during a deploy or upgrade — let the operation complete or fail first