Cinder Internals

Driver Interface

Every Cinder volume driver inherits from cinder.volume.driver.BaseVD (and optionally from mixin classes for snapshots, cloning, backup, etc.). The driver is loaded by the cinder-volume service at startup based on the volume_driver config option in the backend stanza.

Required Driver Methods

Method	Signature	Description
`create_volume`	`(volume)`	Allocate storage for a new volume. Sets `provider_location` if needed.
`delete_volume`	`(volume)`	Release storage and clean up. Must be idempotent.
`create_snapshot`	`(snapshot)`	Create a point-in-time snapshot of a volume.
`delete_snapshot`	`(snapshot)`	Remove a snapshot.
`initialize_connection`	`(volume, connector)`	Export volume to a host. Returns `connection_info` dict consumed by os-brick on the compute node.
`terminate_connection`	`(volume, connector, **kwargs)`	Unexport volume from host.
`get_volume_stats`	`(refresh=False)`	Return dict of backend stats (free_capacity_gb, total_capacity_gb, driver_version, capabilities). Called periodically by the volume manager to update the scheduler cache.
`do_setup`	`(context)`	One-time initialization on service start. Connect to storage backend, validate credentials.
`check_for_setup_error`	`()`	Validate configuration and connectivity after do_setup. Raises exception if backend is not usable.

Optional but Common Driver Methods

Method	Description
`create_volume_from_snapshot`	Create a new volume from an existing snapshot (faster than full copy)
`create_cloned_volume`	Clone an existing volume to a new volume
`extend_volume`	Grow a volume to a larger size
`copy_image_to_volume`	Write a Glance image to a volume (used during `volume create --image`)
`copy_volume_to_image`	Read a volume and upload to Glance
`migrate_volume`	Driver-assisted migration to another backend
`retype`	Change volume type in-place without data movement (if driver supports it)
`update_migrated_volume`	Finalize migration — update provider_id, rename backend objects
`manage_existing`	Adopt a pre-existing backend volume
`manage_existing_get_size`	Return size of a pre-existing volume
`unmanage`	Release Cinder management without deleting data
`failover_host`	Switch to replication target (replication v2.1)
`enable_replication`	Enable replication for a volume
`disable_replication`	Disable replication for a volume
`create_consistencygroup`	Create a consistency group (deprecated; use generic groups)
`create_group`	Create a generic group
`create_group_snapshot`	Create a group snapshot

`connection_info` Structure

The dict returned by initialize_connection is passed to os-brick on the compute node. Structure varies by protocol:

iSCSI:

{
    "driver_volume_type": "iscsi",
    "data": {
        "target_iqn": "iqn.2010-10.org.openstack:volume-abc123",
        "target_portal": "10.0.0.30:3260",
        "target_lun": 1,
        "volume_id": "abc123...",
        "auth_method": "CHAP",
        "auth_username": "cinder-volume-abc123",
        "auth_password": "chappassword",
        "encrypted": False,
        "qos_specs": None,
        "access_mode": "rw"
    }
}

Ceph RBD:

{
    "driver_volume_type": "rbd",
    "data": {
        "name": "volumes/volume-abc123",
        "hosts": ["ceph-mon01", "ceph-mon02", "ceph-mon03"],
        "ports": ["6789", "6789", "6789"],
        "cluster_name": "ceph",
        "auth_enabled": True,
        "auth_username": "cinder",
        "secret_type": "ceph",
        "secret_uuid": "457eb676-33da-42ec-9a8c-9293d545c337",
        "volume_id": "abc123...",
        "access_mode": "rw",
        "encrypted": False
    }
}

NFS:

{
    "driver_volume_type": "nfs",
    "data": {
        "export": "10.0.0.40:/exports/cinder",
        "name": "volume-abc123",
        "options": "vers=4,sec=sys",
        "access_mode": "rw"
    }
}

Multi-Backend Configuration

Multi-backend allows a single cinder-volume process to manage multiple storage backends simultaneously.

Enable Multi-Backend

[DEFAULT]
enabled_backends = lvm-ssd,ceph-hdd,pure-nvme

[lvm-ssd]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_backend_name = lvm-ssd
volume_group = cinder-ssd
target_protocol = iscsi
target_helper = lioadm

[ceph-hdd]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph-hdd
rbd_pool = volumes-hdd
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337

[pure-nvme]
volume_driver = cinder.volume.drivers.pure.PureISCSIDriver
volume_backend_name = pure-nvme
san_ip = 192.168.10.20
pure_api_token = T-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
use_chap_auth = true

When enabled_backends is set, the volume_driver in [DEFAULT] is ignored. Each stanza becomes an independent backend. The cinder-volume process registers one service entry per backend, e.g.: - storage01@lvm-ssd - storage01@ceph-hdd - storage01@pure-nvme

The scheduler's volume_backend_name filter matches the type's extra spec volume_backend_name against the value in each backend stanza.

Pool-Aware Scheduling

Many drivers report multiple pools within a single backend (e.g., Ceph reports pools, NetApp reports aggregates). Pool information is included in get_volume_stats:

{
    "volume_backend_name": "ceph",
    "driver_version": "2.0.0",
    "storage_protocol": "ceph",
    "pools": [
        {
            "pool_name": "volumes-ssd",
            "total_capacity_gb": 10240,
            "free_capacity_gb": 7168,
            "reserved_percentage": 0,
            "thin_provisioning_support": True,
            "thick_provisioning_support": False,
        },
        {
            "pool_name": "volumes-hdd",
            "total_capacity_gb": 51200,
            "free_capacity_gb": 40960,
            "reserved_percentage": 5,
        }
    ]
}

The full host string including pool: storage01@ceph#volumes-ssd

Scheduler Internals

The scheduler uses a two-phase pipeline: filters (eliminate ineligible backends) then weighers (rank remaining backends).

Built-In Scheduler Filters

Filter	Description
`AvailabilityZoneFilter`	Backend must be in the requested AZ (or AZ must be `nova` which matches any)
`CapacityFilter`	Backend must have enough free capacity. Respects `reserved_percentage` and `max_over_subscription_ratio`
`CapabilitiesFilter`	Backend capabilities must match volume type extra specs. Parses `<is>`, `<in>`, `<all-in>`, `<or>`, `==`, `!=`, `>=`, `<=` operators
`DriverFilter`	Evaluates a Python expression from the backend's `filter_function` capability
`JsonFilter`	Parses complex JSON-encoded filter expressions
`RetryFilter`	Excludes backends that were already tried and failed for this request
`InstanceLocalityFilter`	Prefer backends local to the requesting Nova compute host (for performance)
`DifferentBackendFilter`	Places volumes of a consistency group on different backends
`SameBackendFilter`	Places volumes of a consistency group on the same backend
`AffinityFilter`	Soft affinity: schedule near another volume

Default filter list in cinder.conf:

[DEFAULT]
scheduler_default_filters = AvailabilityZoneFilter,CapacityFilter,CapabilitiesFilter,DriverFilter

Built-In Scheduler Weighers

Weigher	Description
`CapacityWeigher`	Prefer backends with more free capacity (default weight multiplier: 1.0)
`AllocatedCapacityWeigher`	Prefer backends with less allocated capacity (inverse of CapacityWeigher)
`ChanceWeigher`	Random weight — distributes load randomly
`GoodnessWeigher`	Uses the backend's `goodness_function` expression (allows custom weighting logic)
`VolumeNumberWeigher`	Prefer backends with fewer volumes

Configure weighers:

[DEFAULT]
scheduler_default_weighers = CapacityWeigher,GoodnessWeigher
capacity_weight_multiplier = 1.0    # positive = prefer more free space
goodness_weight_multiplier = 1.0

DriverFilter and GoodnessWeigher Expressions

These allow backends to provide custom scheduling logic using Python expressions evaluated against backend stats:

[lvm-ssd]
filter_function = "stats.total_capacity_gb < 500"
goodness_function = "100 - stats.allocated_capacity_gb / stats.total_capacity_gb * 100"

Available variables: stats.* (any key from get_volume_stats), volume.* (properties of the volume being scheduled), context.*.

Scheduler Capabilities Cache

The scheduler receives update_service_capabilities RPC messages from each cinder-volume service at scheduler_driver_init_wait_time intervals (default: 60 seconds). These are stored in HostManager._pool_state_cache. This cache is in-memory and is lost on scheduler restart — the scheduler waits for at least one capability update from each backend before processing requests.

Volume Replication v2.1

Cinder replication v2.1 (introduced in Mitaka, current in 2025.x) provides disaster-recovery failover between a primary and one or more replica backends. It is a per-backend operation, not per-volume (all volumes on the backend are replicated).

Concepts

Term	Description
Primary backend	Normal active backend serving I/O
Secondary/replica backend	Backup backend; data is replicated here asynchronously or synchronously
Replication status	Per-volume: `enabled`, `disabled`, `not-capable`, `error`, `failing-over`, `failed-over`
`failover_host`	Admin RPC command that promotes the replica to primary
`freeze_host`	Marks a backend as not accepting new operations (pre-failover step)
`thaw_host`	Re-enables a frozen backend

Driver Requirements for Replication v2.1

Drivers must implement: - enable_replication(context, group, volumes) — enable replication on a group of volumes - disable_replication(context, group, volumes) — disable replication - failover_replication(context, group, volumes, secondary_backend_id) — perform failover - list_replication_targets(context, group) — return available replication targets - Report replication_enabled: True and replication_targets in get_volume_stats

Configuring Replication

Example for a driver that supports replication:

[ceph-primary]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph-primary
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = cinder
rbd_secret_uuid = 457eb676-33da-42ec-9a8c-9293d545c337
replication_device = backend_id:ceph-dr,conf:/etc/ceph/ceph-dr.conf,user:cinder,pool:volumes-dr,secret_uuid:88b4c97c-aaaa-bbbb-cccc-1234567890ab

Failover Procedure

# 1. Freeze the primary backend (stops new operations)
cinder --os-volume-api-version 3.0 failover-host storage01@ceph-primary --backend-id ceph-dr

# Alternatively using the admin API:
openstack volume backend failover storage01@ceph-primary --backend-id ceph-dr

# 2. After failover, cinder-volume reports host as 'storage01@ceph-primary' but
#    operating on ceph-dr. Update Nova to use new attachment paths.

# 3. To fail back (when primary is restored):
openstack volume backend failover storage01@ceph-primary --backend-id default

Database Models

Cinder uses SQLAlchemy ORM with Alembic-managed migrations. Key tables:

`volumes`

Column	Type	Description
`id`	UUID	Primary key
`display_name`	VARCHAR(255)	User-visible name
`display_description`	VARCHAR(255)	User-visible description
`user_id`	VARCHAR(255)	Owner user UUID
`project_id`	VARCHAR(255)	Owner project UUID
`status`	VARCHAR(255)	`creating`, `available`, `in-use`, `deleting`, `error`, `migrating`, `maintenance`, etc.
`size`	INTEGER	Size in GiB
`volume_type_id`	UUID	FK → `volume_types.id`
`availability_zone`	VARCHAR(255)	AZ name
`host`	VARCHAR(255)	Backend host: `hostname@backend#pool`
`provider_location`	VARCHAR(255)	Driver-specific location (e.g., LV path, RBD image name)
`provider_id`	VARCHAR(255)	Driver-specific identifier
`encryption_key_id`	UUID	Barbican secret UUID for encrypted volumes
`replication_status`	VARCHAR(255)	Replication state
`migration_status`	VARCHAR(255)	Migration state
`multiattach`	BOOLEAN	Multi-attach enabled
`bootable`	BOOLEAN	Marked as bootable
`deleted`	BOOLEAN	Soft-delete flag
`deleted_at`	DATETIME	Soft-delete timestamp
`created_at`	DATETIME	Creation timestamp
`updated_at`	DATETIME	Last update timestamp

`snapshots`

Column	Type	Description
`id`	UUID	Primary key
`volume_id`	UUID	FK → `volumes.id`
`status`	VARCHAR(255)	`creating`, `available`, `deleting`, `error`, etc.
`volume_size`	INTEGER	Size of parent volume at snapshot time
`provider_location`	VARCHAR(255)	Driver-specific snapshot location
`encryption_key_id`	UUID	Inherited from parent volume
`cgsnapshot_id`	UUID	FK → `cgsnapshots.id` if part of a CG snapshot
`group_snapshot_id`	UUID	FK → `group_snapshots.id`

`backups`

Column	Type	Description
`id`	UUID	Primary key
`volume_id`	UUID	UUID of the backed-up volume
`status`	VARCHAR(255)	`creating`, `available`, `restoring`, `deleting`, `error`
`size`	INTEGER	Volume size at backup time (GiB)
`object_count`	INTEGER	Number of backup objects written
`container`	VARCHAR(255)	Swift container or S3 bucket name
`parent_id`	UUID	Parent backup UUID for incremental backups
`is_incremental`	BOOLEAN	True if this is an incremental backup
`has_dependent_backups`	BOOLEAN	True if there are incrementals dependent on this backup
`driver_info`	TEXT	JSON blob of driver-specific backup metadata
`encryption_key_id`	UUID	Key UUID for encrypted volume backups

`volume_types`

Column	Type	Description
`id`	UUID	Primary key
`name`	VARCHAR(255)	User-visible type name
`description`	VARCHAR(255)	Optional description
`qos_specs_id`	UUID	FK → `quality_of_service_specs.id`
`is_public`	BOOLEAN	Visible to all projects
`deleted`	BOOLEAN	Soft-delete flag

`volume_type_extra_specs`

Column	Type	Description
`id`	INTEGER	Primary key
`volume_type_id`	UUID	FK → `volume_types.id`
`key`	VARCHAR(255)	Extra spec key
`value`	VARCHAR(255)	Extra spec value

`quality_of_service_specs`

Column	Type	Description
`id`	UUID	Primary key
`name`	VARCHAR(255)	QoS spec name
`consumer`	VARCHAR(255)	`front-end`, `back-end`, or `both`
`specs`	JSON	Key-value pairs of QoS parameters

`volume_attachment`

Column	Type	Description
`id`	UUID	Primary key
`volume_id`	UUID	FK → `volumes.id`
`instance_uuid`	UUID	Nova instance UUID
`attached_host`	VARCHAR(255)	Compute host where attached
`mountpoint`	VARCHAR(255)	Device path on compute node
`attach_mode`	VARCHAR(36)	`rw` or `ro`
`attach_status`	VARCHAR(255)	`attached`, `detached`
`connection_info`	TEXT	JSON blob from `initialize_connection`

RPC Flow

All inter-service communication uses oslo.messaging. Cinder uses both call (synchronous, waits for reply) and cast (fire-and-forget).

Create Volume RPC Flow

cinder-api (HTTP POST /volumes)
  │
  ├─ DB: create volumes row (status=creating)
  ├─ DB: reserve quota
  │
  ├─ scheduler_rpcapi.create_volume()   [cast to scheduler queue]
  │
cinder-scheduler (consumes from cinder-scheduler queue)
  │
  ├─ run filters → eliminate backends
  ├─ run weighers → rank backends
  ├─ select winner: storage01@ceph#volumes-ssd
  │
  ├─ volume_rpcapi.create_volume()   [cast to storage01@ceph queue]
  │
cinder-volume (storage01@ceph)
  │
  ├─ driver.create_volume(volume)
  ├─ DB: update volumes row (status=available, host=storage01@ceph#volumes-ssd)
  ├─ update quota usage

Attach Volume RPC Flow (Nova-initiated)

nova-compute
  │
  ├─ cinder_client.volumes.reserve(volume_id)          [HTTP → cinder-api]
  │   └─ DB: volume status → attaching
  │
  ├─ cinder_client.volumes.initialize_connection(       [HTTP → cinder-api]
  │       volume_id, connector)
  │   └─ volume_rpcapi.initialize_connection() [call → cinder-volume]
  │       └─ driver.initialize_connection() → returns connection_info
  │
  ├─ os-brick.connect_volume(connection_info)          [local on compute node]
  │   └─ iscsiadm / rbd map / NFS mount
  │
  ├─ libvirt: hot-attach device to instance
  │
  ├─ cinder_client.volumes.attach(volume_id,            [HTTP → cinder-api]
  │       instance_uuid, mountpoint)
  │   └─ DB: volume status → in-use, create volume_attachment row

Backup Create RPC Flow

cinder-api (POST /backups)
  │
  ├─ DB: create backups row (status=creating)
  │
  ├─ backup_rpcapi.create_backup()   [cast to cinder-backup queue]
  │
cinder-backup
  │
  ├─ Create a temporary snapshot of the volume
  ├─ Read snapshot via volume driver (attach locally or use driver export)
  ├─ Write chunks to backup store (Swift, Ceph, etc.)
  ├─ Delete temporary snapshot
  ├─ DB: update backups row (status=available, object_count, size)

Volume Manager Periodic Tasks

The cinder-volume manager runs several periodic tasks:

Task	Default Interval	Description
`_report_driver_status`	60 seconds	Call `get_volume_stats`, publish capabilities to scheduler
`_clean_temporary_snapshots`	300 seconds	Remove leaked temporary snapshots from failed backups
`_update_volume_pg_fields`	60 seconds	Sync volume group membership metadata
`_notify_about_service_capabilities`	60 seconds	oslo.notifications: publish backend stats to Ceilometer/Gnocchi
`_do_cleanup`	At startup	Recover volumes stuck in transitional states after crash

The startup cleanup (_do_cleanup) handles volumes left in creating, deleting, or attaching states after a cinder-volume crash. It resets them to error so the operator can investigate.

Thin Provisioning

Thin provisioning allows the sum of allocated volume sizes to exceed actual backend capacity. The scheduler enforces limits via:

[DEFAULT]
max_over_subscription_ratio = 20.0   # Allow 20x overcommit
reserved_percentage = 10             # Reserve 10% of capacity for internal use

A backend reports thin_provisioning_support = True and provisioned_capacity_gb in its stats. The CapacityFilter computes:

virtual_free = total_capacity * max_over_subscription_ratio
             - provisioned_capacity
             - total_capacity * reserved_percentage / 100

If virtual_free < requested_size, the backend is filtered out.

Per-backend override:

[ceph-ssd]
max_over_subscription_ratio = 10.0
reserved_percentage = 5