▤ Stackers Network

Learn OpenStack · core/storage/block

Cinder Architecture

Service Components

Cinder is composed of four daemon processes that communicate over an oslo.messaging RPC bus (typically RabbitMQ). Each process can be scaled and placed independently.

cinder-api

The WSGI API server. Validates and authenticates requests (via keystonemiddleware), enforces policy (policy.yaml / oslo.policy), and forwards work to cinder-scheduler or cinder-volume via RPC. The API is versioned; the current stable version is v3 (microversions). API v2 was removed in the Xena release.

The API process is stateless and horizontally scalable behind a load balancer. All state lives in the database.

Key responsibilities: - Accept and validate volume/snapshot/backup/type CRUD requests - Enforce project quotas before dispatching (quota checks use the DB) - Route requests to the appropriate scheduler or volume RPC target - Return job IDs for asynchronous operations (volume creation, backup, etc.)

Configuration file: /etc/cinder/cinder.conf WSGI entry point: cinder.wsgi.osapi_volume

cinder-scheduler

Selects which backend (cinder-volume service instance and pool) should fulfill a create_volume or retype request. It runs a configurable filter and weigher pipeline, similar to nova-scheduler.

The scheduler maintains an in-memory capability cache populated by periodic update_service_capabilities messages sent by each cinder-volume worker. This cache holds free capacity, total capacity, thin-provisioning info, driver capabilities, and backend-specific extra specs support.

Default scheduler driver: cinder.scheduler.filter_scheduler.FilterScheduler

The scheduler is stateless between requests and can be scaled horizontally. Multiple scheduler instances process requests concurrently from the same RabbitMQ queue.

cinder-volume

The data-plane orchestrator. One cinder-volume process manages one or more storage backends. It receives RPC calls from the scheduler (for creates) or from cinder-api (for operations on existing volumes) and calls the appropriate storage driver.

Each cinder-volume process registers itself in the services table with a host value of the form <hostname>@<backend_name>. For multi-backend deployments the host value includes the backend stanza name, e.g., storage01@lvm or storage01@ceph.

Key responsibilities: - Call storage driver to create/delete/extend/clone volumes and snapshots - Export volumes for attachment (iSCSI target creation, RBD map, NFS export) - Call Nova's os-initialize_connection and os-terminate_connection APIs - Manage volume migration between backends - Perform periodic capacity reporting to the scheduler

cinder-volume is the only component that requires direct network connectivity to the storage backend. It does not need to be co-located with compute nodes.

cinder-backup

Manages volume backup and restore operations. Supports multiple backup backends:

Backend Driver Notes
Ceph (RBD) cinder.backup.drivers.ceph.CephBackupDriver Efficient incremental backups via RBD diff
Swift cinder.backup.drivers.swift.SwiftBackupDriver Default; stores backup chunks in Swift containers
NFS cinder.backup.drivers.nfs.NFSBackupDriver Mounts NFS share, stores backup files
POSIX cinder.backup.drivers.posix.PosixBackupDriver Local filesystem; rarely used in production
Google Cloud Storage cinder.backup.drivers.google.GoogleBackupDriver For hybrid clouds

Backup operations are asynchronous. cinder-backup reads from the volume (via the volume driver or via a snapshot) and writes chunked, optionally compressed and encrypted, data to the backup store.

cinder-backup can be scaled horizontally. Multiple instances share the work queue. Backup metadata is stored in the Cinder database.

Volume Lifecycle

User request                   cinder-api
   │                               │
   │  POST /volumes                │  validate + quota check
   │ ────────────────────────────► │
   │                               │  cast to scheduler
   │                               ▼
                           cinder-scheduler
                               │  run filter/weigher pipeline
                               │  select backend host+pool
                               │  cast to cinder-volume@host
                               ▼
                       cinder-volume (on chosen host)
                               │  call driver.create_volume()
                               │  update DB: status=available
                               ▼
                         Storage Backend
                           (LVM / Ceph / etc.)

Attach flow (Nova-initiated):

nova-compute
   │  os-attach (reserve)          cinder-api
   │ ────────────────────────────► │
   │                               │  RPC → cinder-volume
   │                               ▼
                       cinder-volume
                           │  driver.initialize_connection()
                           │  returns connection_info (iqn, portal, rbd pool, etc.)
                           ▼
                       cinder-api → returns connection_info to nova-compute

nova-compute
   │  attach device (libvirt/os-brick connects iSCSI/RBD)
   │  os-attach (finalize)
   │ ────────────────────────────► cinder-api → volume status: in-use

Volume Types

A volume type is a named set of extra specifications that maps user intent to backend capabilities. Every volume create request either specifies a type or uses the configured default (default_volume_type in [DEFAULT]).

Volume types contain: - Extra specs: key-value pairs matched against driver-reported capabilities and scheduler filters. Common keys: - volume_backend_name — direct-target a specific backend - replication_enabled — <is> True for replicated backends - IOPS_sec — passed to QoS specs if linked - multiattach — <is> True to allow multi-attach volumes - encrypted — set by encryption providers (see below) - QoS specs: linked QoS policy (see below) - Encryption spec: linked encryption provider (see below)

# Create a volume type
openstack volume type create ssd-replicated \
  --property volume_backend_name=ceph-ssd \
  --property replication_enabled='<is> True'

# Show type and its extra specs
openstack volume type show ssd-replicated

Default Volume Type

Set in cinder.conf:

[DEFAULT]
default_volume_type = standard

If no default is configured and a request omits --type, Cinder uses any type named __DEFAULT__ (created automatically during DB migration in recent releases).

QoS Specs

QoS specs define I/O limits that are enforced either front-end (by the hypervisor/libvirt) or back-end (by the storage array). They are created independently and then associated with one or more volume types.

Front-end QoS (enforced by libvirt via cgroup blkio):

Key Description
total_bytes_sec Total throughput limit (bytes/sec)
read_bytes_sec Read throughput limit
write_bytes_sec Write throughput limit
total_iops_sec Total IOPS limit
read_iops_sec Read IOPS limit
write_iops_sec Write IOPS limit
total_bytes_sec_max Burst total throughput
total_iops_sec_max Burst total IOPS

Back-end QoS keys are driver-specific (e.g., Pure Storage uses maxIOPS, maxBWS).

openstack volume qos create high-iops \
  --consumer front-end \
  --property total_iops_sec=5000 \
  --property total_bytes_sec=524288000

openstack volume qos associate high-iops ssd-replicated

Volume Encryption

Cinder supports per-type volume encryption using the LUKS provider backed by Barbican for key management.

When a volume of an encrypted type is created: 1. cinder-volume calls Barbican to generate a new symmetric key (AES-256 by default). 2. The key reference (Barbican secret UUID) is stored in the encryption_key_id column of the volumes table. 3. nova-compute retrieves the key from Barbican at attach time and sets up dm-crypt/LUKS on the compute node. 4. The encrypted device is presented to the instance.

# Create an encryption provider for a volume type
openstack volume encryption type create \
  --provider nova.volume.encryptors.luks.LuksEncryptor \
  --cipher aes-xts-plain64 \
  --key-size 256 \
  --control-location front-end \
  encrypted-ssd

Requirements: - Barbican must be deployed and registered in the Keystone catalog - [keymgr] backend = barbican in cinder.conf - Nova compute nodes need cryptsetup and dm-crypt kernel module

Integration with Nova

Nova interacts with Cinder for:

Operation Nova action Cinder action
Boot from volume os-createBackup or regular create Cinder creates volume from image (clones or copies)
Attach volume os-attach (reserve → initialize_connection → attach) Driver exports volume, returns connector info
Detach volume os-detach (begin_detach → terminate_connection → detach) Driver unexports volume
Resize instance (root volume) os-resize triggers volume extend extend_volume on the root Cinder volume
Evacuate Nova calls detach + attach on new host Cinder volume stays in-use until re-attached
Snapshot instance Nova calls os-createBackup Cinder creates a snapshot of each attached volume

The Nova compute node uses os-brick (the OpenStack connector library) to handle the actual host-level connection: logging into iSCSI targets, mapping RBD devices, mounting NFS, etc.

Multiattach volumes (requires multiattach=True extra spec and a capable driver): - Nova must use the libvirt shareable disk mode - Only supported for volumes where the driver reports multiattach capability

Integration with Glance

Cinder can use Glance images as volume sources and can export volumes back to Glance:

Flow Description
Volume from image openstack volume create --image <img> — Cinder downloads image from Glance, writes to new volume
Upload volume to image openstack image create --volume <vol> — Cinder reads volume data, uploads to Glance
Glance image store (Cinder) Glance can store images directly in a Cinder volume (via cinder store in glance-api.conf)

For the "volume from image" path, certain drivers (Ceph, NetApp) support native cloning from Glance images stored in the same backend, bypassing full data copy.

Active-Active HA Architecture

Traditional Active-Passive

The simplest HA model: two cinder-volume nodes share a Pacemaker/Corosync cluster. One node is active; the other takes over on failure. Volume host names are stable (virtual IP or fixed hostname). No DLM required.

Limitation: failover time is 30–60 seconds; backend I/O is interrupted during that window.

Active-Active HA (2025.x)

Active-active allows multiple cinder-volume workers for the same backend to process requests simultaneously. This requires:

  1. A distributed lock manager (DLM) — Tooz-backed, typically using etcd3 or ZooKeeper as the coordination backend. Prevents concurrent conflicting operations on the same volume.

  2. Stable per-worker host names — Each worker registers with a UUID-based cluster name rather than hostname, so the scheduler can address the cluster rather than individual nodes.

  3. [coordination] section in cinder.conf:

[coordination]
backend_url = etcd3+http://etcd01:2379,etcd3+http://etcd02:2379,etcd3+http://etcd03:2379
  1. [DEFAULT] cluster = cinder-cluster-1 — All workers in the same AA group share this cluster name.

Active-active cinder-backup is also supported using the same Tooz coordination mechanism.

The scheduler routes requests to the cluster name. Individual workers compete for lock ownership before executing a driver call. If a worker dies mid-operation, another worker detects the stale lock (via TTL expiry) and can attempt recovery.

Requirements for active-active: - All workers must use a shared-access backend (Ceph, a SAN with multipath, etc.). LVM is not suitable for active-active unless the volume group is on shared storage. - etcd 3.x or ZooKeeper 3.6+ - oslo.concurrency and tooz Python packages

Database HA

Cinder requires a highly-available database. Recommended: MariaDB Galera Cluster (3+ nodes) fronted by ProxySQL or HAProxy. The API and scheduler processes are stateless and tolerate brief DB unavailability; in-flight driver operations may fail if the DB becomes unavailable mid-operation.

Service Registration and Health

All Cinder services register in the services table. Check service health:

openstack volume service list

Expected output for a healthy three-backend deployment:

+------------------+----------------------------+------+---------+-------+
| Binary           | Host                       | Zone | Status  | State |
+------------------+----------------------------+------+---------+-------+
| cinder-scheduler | controller01               | nova | enabled | up    |
| cinder-volume    | storage01@lvm              | nova | enabled | up    |
| cinder-volume    | storage01@ceph             | nova | enabled | up    |
| cinder-backup    | storage01                  | nova | enabled | up    |
+------------------+----------------------------+------+---------+-------+

A service with State: down indicates missed heartbeats. Default heartbeat interval: 10 seconds. Service is considered down after service_down_time seconds (default: 60).