▤ Stackers Network

Learn OpenStack · core/bare-metal

Ironic Internals

Driver Composition Model

Every Ironic node is configured as a combination of a hardware type and a set of interface implementations. The hardware type declares which interfaces are supported; the operator selects one implementation per interface when creating or updating a node.

Node
 ├── hardware_type: "ipmi"
 ├── power_interface: "ipmitool"
 ├── management_interface: "ipmitool"
 ├── deploy_interface: "direct"
 ├── boot_interface: "ipxe"
 ├── inspect_interface: "inspector"
 ├── bios_interface: "no-bios"
 ├── raid_interface: "agent"
 ├── console_interface: "ipmitool-socat"
 └── vendor_interface: "ipmitool"

At runtime, ironic-conductor resolves the actual Python class for each interface and calls its methods. For example, when a power-on is requested:

# ironic/conductor/manager.py
node = objects.Node.get_by_uuid(context, node_uuid)
driver = self._get_driver(node.driver)
power_iface = driver.power  # resolves to IPMIPower() for "ipmitool"
power_iface.validate(task)
power_iface.set_power_state(task, states.POWER_ON)

Interface Resolution

When a node is created with --driver ipmi (hardware type) and --boot-interface ipxe, Ironic:

  1. Loads the IPMIHardware class from the ipmi entrypoint
  2. Verifies that ipxe is in IPMIHardware.supported_boot_interfaces
  3. Stores "ipxe" in node.boot_interface
  4. At runtime, loads iPXEBoot from the ipxe boot interface entrypoint

Interface implementations are registered as Python entry points in setup.cfg:

[entry_points]
ironic.hardware.types =
    ipmi = ironic.drivers.ipmi:IPMIHardware
    redfish = ironic.drivers.redfish:RedfishHardware

ironic.hardware.interfaces.boot =
    ipxe = ironic.drivers.modules.ipxe:iPXEBoot
    pxe = ironic.drivers.modules.pxe:PXEBoot
    redfish-virtual-media = ironic.drivers.modules.redfish.boot:RedfishVirtualMediaBoot

Conductor Hash Ring

When multiple ironic-conductor processes run, node operations are distributed among them using a consistent hash ring. This prevents two conductors from simultaneously operating on the same node.

conductors registered in DB:
  conductor-01  conductor-02  conductor-03

Hash ring (tokens distributed across 360-degree ring):
  [conductor-01][conductor-02][conductor-03][conductor-01][conductor-02]...

For node UUID "abc123":
  hash("abc123") → position 147 → assigned to conductor-02

Hash Ring Mechanics

  • Each conductor registers itself in the conductors database table on startup with a heartbeat timestamp
  • ironic-conductor updates its heartbeat every heartbeat_interval seconds (default 10 s)
  • If a conductor's heartbeat is not updated within heartbeat_timeout seconds, it is considered dead and removed from the ring
  • When the ring changes (conductor added or removed), node ownership shifts: some nodes are reassigned to a different conductor on the next periodic task run
  • The conductor that "owns" a node is the only one that will operate on it during periodic tasks; API-triggered operations use acquire_lock() which also respects ring ownership

Node takeover when a conductor fails:

1. conductor-02 stops sending heartbeats
2. Other conductors detect conductor-02 is gone via DB check
3. Hash ring is recalculated; conductor-02's nodes are redistributed
4. conductor-01 and conductor-03 pick up the orphaned nodes
5. For any node stuck in a transitional state (deploying, cleaning),
   the new owner conductor runs a "takeover" procedure:
   - If IPA is running: re-attach to the IPA heartbeat
   - If no IPA response: fail the node and set provision_state=ERROR

Deploy Steps and Clean Steps

Operations in Ironic are decomposed into ordered lists of steps. Each step is a method on an interface class with metadata:

@METRICS.timer('AgentDeployMixin.write_image')
@base.deploy_step(priority=80, argsinfo={
    'image_url': {'description': 'URL of image to deploy'},
    'image_checksum': {'description': 'SHA512 checksum'},
})
def write_image(self, task, image_url=None, image_checksum=None):
    """Write an OS image to the target disk via IPA."""
    ...

Step metadata:

Field Meaning
priority Execution order; higher priority runs first; 0 = disabled
interface Which interface this step belongs to
step Method name
requires_ramdisk Whether IPA must be running on the node
argsinfo Named arguments that can be passed via the API

Deploy Step Execution Order (default direct deploy)

priority 100:  deploy.deploy           (kick off the full deploy workflow)
priority 80:   deploy.write_image      (IPA writes image to disk)
priority 60:   deploy.inject_files     (inject authorized_keys, cloud-init data)
priority 40:   deploy.prepare_image    (convert qcow2 to raw if needed)

Clean Step Execution Order (default automated clean)

priority 90:   management.reset_bios_to_default   (optional, if BIOS interface supports it)
priority 10:   deploy.erase_devices               (shred/blkdiscard all disks)

Operators can add custom clean steps by shipping a plugin that registers an interface with @base.clean_step(priority=N) methods.

Manual Clean Step Syntax

The API accepts a list of step dicts:

[
  {"interface": "deploy",     "step": "erase_devices",       "args": {}},
  {"interface": "raid",       "step": "delete_configuration", "args": {}},
  {"interface": "raid",       "step": "create_configuration", "args": {}},
  {"interface": "management", "step": "inject_nmi",          "args": {}}
]

Steps are executed in the order provided for manual cleaning. For automated cleaning, steps are sorted by priority descending across all active interfaces.

IPA — Ironic Python Agent

IPA is a Python application that runs inside a lightweight ramdisk image on the bare metal node during deploy and clean operations. It provides a REST HTTP API that ironic-conductor calls to execute operations directly on the node's hardware.

IPA Lifecycle During Deployment

1. Node is powered on
2. DHCP assigns an IP; PXE/iPXE boots the IPA ramdisk
3. IPA starts and performs "lookup":
   GET https://<ironic-api>:6385/v1/lookup?addresses=<mac1>,<mac2>
   → Returns the node UUID and conductor API endpoint

4. IPA sends periodic heartbeats:
   POST https://<ironic-api>:6385/v1/heartbeat/<node-uuid>
   {"callback_url": "https://<ipa-ip>:9999/v1", "agent_token": "..."}

5. ironic-conductor receives first heartbeat → "continue" signal
6. conductor calls IPA to start deploy steps:
   POST https://<ipa-ip>:9999/v1/commands
   {"name": "standby.cache_image", "params": {"image_info": {...}}}

7. IPA downloads the image, writes it to disk:
   POST https://<ipa-ip>:9999/v1/commands
   {"name": "deploy.write_image", ...}

8. conductor calls IPA to finalize (write config drive, set bootloader):
   POST https://<ipa-ip>:9999/v1/commands
   {"name": "deploy.install_bootloader", ...}

9. conductor tells IPA to reboot:
   POST https://<ipa-ip>:9999/v1/commands
   {"name": "system_passthrough.reboot", ...}

10. Node reboots into the deployed OS
11. ironic-conductor sets node provision_state=ACTIVE

IPA Lookup

IPA discovers which Ironic node it is by looking up its MAC addresses:

GET /v1/lookup?addresses=aa:bb:cc:dd:ee:ff&addresses=aa:bb:cc:dd:ee:00

Ironic matches the MAC addresses against the ports table and returns the node UUID and configuration. If no match is found, the lookup returns 404 and IPA retries.

IPA Heartbeat

IPA heartbeats every 10 seconds (configurable). The heartbeat carries:

  • callback_url: the HTTPS URL where ironic-conductor can reach IPA
  • agent_token: a one-time token issued by Ironic to authenticate the agent
  • version: IPA version

The agent_token mechanism prevents rogue agents from hijacking node deployments.

IPA Commands

IPA exposes a command execution API. Commands are asynchronous — the conductor polls for results:

POST /v1/commands
{"name": "deploy.write_image", "params": {...}}
→ {"command_name": "write_image", "command_status": "RUNNING", "command_result": null}

GET /v1/commands?wait=10
→ {"commands": [{"command_status": "SUCCEEDED", "command_result": {...}}]}

Key IPA command namespaces:

Namespace Commands
deploy write_image, prepare_image, install_bootloader, get_clean_steps, execute_clean_step
standby cache_image, sync
system_passthrough reboot, power_off, get_cpu_count, get_memory_info
raid create_configuration, delete_configuration, get_logical_disks
image get_os_install_command (Anaconda deploy)

Provisioning State Machine

The full state machine with all valid transitions:

                            +----------+
                            |  enroll  | ◄── initial state after node create
                            +----------+
                                 │
                         manage  │  (verify driver credentials)
                                 ▼
                          +------------+
                     ┌───►│ verifying  │
                     │    +------------+
                     │         │ success
                     │         ▼
                     │   +------------+
                     │   │ manageable │ ◄──────────────────────────────────┐
                     │   +------------+                                    │
                     │        │ │                                          │
                     │provide │ │ inspect                                  │
                     │        │ ▼                                          │
                     │        │  +------------------+                      │
                     │        │  │    inspecting    │                      │
                     │        │  +------------------+                      │
                     │        │     │ success / fail                       │
                     │        │     ▼                                      │
                     │        │  +------------------+                      │
                     │        │  │  inspect failed  │ ─── manage ────────►│
                     │        │  +------------------+                      │
                     │        │ success                                    │
                     │        ▼                                            │
                     │   +-----------+                                     │
                     │   │ available │ ◄── (spare pool / Nova resource)   │
                     │   +-----------+                                     │
                     │        │ deploy (Nova or standalone)               │
                     │        ▼                                            │
                     │   +-----------+                                     │
                     │   │ deploying │                                     │
                     │   +-----------+                                     │
                     │        │ IPA starts / PXE boot                     │
                     │        ▼                                            │
                     │   +--------------+                                  │
                     │   │ wait call-   │ ◄── IPA heartbeat received      │
                     │   │ back         │                                  │
                     │   +--------------+                                  │
                     │        │ deploy complete / fail                    │
                     │    ┌───┴───────┐                                   │
                     │    │           │                                    │
                     │    ▼           ▼                                   │
                     │ +--------+  +---------------+                      │
                     │ │ active │  │ deploy failed │ ─── undeploy ───────►│
                     │ +--------+  +---------------+     or manage        │
                     │     │                                               │
                     │undeploy│  (Nova delete or standalone)               │
                     │        ▼                                            │
                     │  +----------+                                       │
                     │  │ deleting │  (tear-down, unplug network)          │
                     │  +----------+                                       │
                     │        │ automated clean starts                    │
                     │        ▼                                            │
                     │  +-----------+                                      │
                     │  │ cleaning  │                                      │
                     │  +-----------+                                      │
                     │        │ success                                    │
                     │        └──────────────────────────────────────────►│
                     │                                          (available)│
                     │                                                     │
                     │  +------------------+                               │
                     └──│  clean failed    │ ─── manage ─────────────────►│
                        +------------------+

Additional states:

  rescue ──► rescuing ──► rescue failed ──► manage ──► manageable
                │
                └──► rescued (node in rescue OS for operator access)

  manageable ──► servicing ──► service failed ──► manage ──► manageable
                     │
                     └──► available / active (post-service)

Transition Table

From State Trigger To State Notes
enroll manage verifying Conductor verifies BMC credentials
verifying (success) manageable Credentials valid
verifying (fail) enroll Credentials invalid; check driver-info
manageable inspect inspecting Boot inspector ramdisk
manageable provide cleaning / available Runs automated clean first
manageable clean cleaning Manual clean
inspecting (success) manageable Properties and ports populated
inspecting (fail) inspect failed Check last_error
inspect failed manage manageable Fix and retry
available deploy deploying Boot node into IPA ramdisk
deploying (IPA boot) wait call-back IPA sends heartbeat
wait call-back (success) active Deploy steps completed
wait call-back (fail) deploy failed Check last_error
active undeploy deleting Tear down instance
deleting (complete) cleaning / available Run automated clean
deploy failed undeploy deleting Clean up partial deploy
deploy failed manage manageable Skip clean, go to manageable
cleaning (success) available (auto) or manageable (manual) Clean steps done
cleaning (fail) clean failed Check last_error
clean failed manage manageable Fix clean config and retry
active rescue rescuing Boot rescue ramdisk for operator
rescuing (success) rescued Node in rescue OS
rescued unrescue active Return to deployed state

Periodic Tasks

ironic-conductor runs several periodic tasks:

Task Interval Purpose
_sync_power_states 60 s Query BMC for actual power state; reconcile with DB; sync if mismatch
_check_deploy_timeouts 60 s Fail deployments that have exceeded deploy_callback_timeout
_check_clean_timeouts 60 s Fail cleanings that have exceeded clean_callback_timeout
_sync_local_state 180 s Recalculate hash ring; take over nodes from failed conductors
_do_takeover 180 s Attempt to re-attach to IPA agents on nodes in wait call-back
_check_orphan_nodes 60 s Find nodes whose conductor no longer exists; reassign
handle_sync_power_status 60 s Update updated_at for all active nodes

Node Locking

ironic-conductor uses database-level locking to serialize operations on a node. Before starting any operation, the conductor acquires an exclusive lock:

with task_manager.acquire(context, node_uuid, shared=False) as task:
    # Only one conductor can hold this lock at a time
    task.driver.deploy.deploy(task)

shared=True is used for read-only operations (power state sync). shared=False is required for state changes. If a conductor crashes while holding a lock, the lock is automatically released after node_locked_retry_attempt retries.

Config Drive

Ironic can inject a Nova-compatible config drive into deployed instances. The config drive is a small FAT32 filesystem image mounted at /dev/disk/by-label/config-2 and contains:

  • /openstack/latest/meta_data.json (instance metadata, SSH keys, hostname)
  • /openstack/latest/user_data (cloud-init user-data)
  • /openstack/latest/network_data.json (Neutron network configuration)

The config drive is generated by Nova (for Nova-managed deployments) or can be passed directly to the Ironic API (for standalone deployments):

# Generate and base64-encode a config drive
CONFIG_DRIVE=$(genisoimage -o - -input-charset utf-8 \
  -volid config-2 -joliet -rock \
  -graft-points openstack/latest/meta_data.json=meta_data.json | base64 -w0)

openstack baremetal node set compute-01 \
  --instance-info configdrive="$CONFIG_DRIVE"

Fast-Track Deployment

Fast-track deployment (introduced in Stein, enabled by default in recent releases) avoids rebooting the node into IPA when it is already running IPA from a previous inspection or deployment attempt. This reduces deployment time from ~5 minutes to ~30 seconds for subsequent deployments:

[deploy]
fast_track = True

With fast-track enabled, if a node's IPA is already running and heartbeating, the conductor skips the PXE boot phase and proceeds directly to deploy steps.