[How To] Automate OCI Out of Host Capacity Retries with Ansible

Launching an instance on Oracle Cloud Infrastructure (OCI) often fails with the Out of host capacity error, especially on the Always Free tier and other free-tier compute shapes. Oracle does not guarantee immediate capacity for free shapes, so the same apply job can fail for minutes or hours before it finally succeeds. Because this failure is transient, the correct response is to wait and retry until capacity becomes available. In this guide, you will learn how to automate OCI out of host capacity retries with a self-healing Ansible playbook orchestrated by Semaphore UI running inside Docker on an Ubuntu 24.04 LTS host. By the end, a single click resubmits your stack apply job on a fixed schedule until OCI provisions your instance successfully.

Table of Contents

• Understanding the OCI Out of Host Capacity Error
• Why Automate OCI Out of Host Capacity Retries?
• Prerequisites
• Step 1: Prepare the OCI Config File on Ubuntu
• Step 2: Deploy Semaphore UI with Docker
• Step 3: Configure Semaphore Projects
• Step 4: Create the Retry Ansible Playbook
• Step 5: Run and Monitor the Task
• Troubleshooting Common Errors
• Best Practices
• Conclusion

Understanding the OCI Out of Host Capacity Error

The Out of host capacity message appears when the OCI Resource Manager submits an APPLY job and the compute service cannot place your virtual machine onto a physical host in the targeted region or availability domain. From Oracle’s perspective this is a service-side capacity availability problem, not a configuration problem with your Terraform stack or your API credentials.

This situation is far more common on the free tiers. Free shapes such as the VM.Standard.E2.1.Micro AMD instance and the free ARM VM.Standard.A1.Flex shapes receive lower scheduling priority than paid shapes, so their availability fluctuates heavily. A typical job log reports the failure like this:

[2026/08/28 16:39:38] ERROR - Error: 500-InternalError, Out of host capacity.
[2026/08/28 16:39:38] INFO - Suggestion: The service for this resource encountered an error. Please contact support for help with service: Core Instance

When you receive this exact output, your stack definition is usually fine. The OCI out of host capacity error simply means the instance cannot be scheduled right now. There is no guarantee that a single retry will work either, so the practical strategy is to keep resubmitting the apply operation on a strict interval until a host frees up.

Why Automate OCI Out of Host Capacity Retries?

You have two ways to handle capacity failures: you can retry manually, or you can automate the retry. Retrying manually means refreshing the job log, waiting, clicking Apply again, and repeating the cycle whenever the schedule fails. This approach has several drawbacks.

• It requires you to sit in front of the console and watch the log, sometimes for hours.
• You could easily miss the brief window when capacity opens up, wasting productive time.
• There is no fixed discipline: a tired operator might give up and leave the instance unprovisioned.

Automating the retry removes all of these problems. A scheduled task re-submits the apply job every fixed interval, keeps working through the night, and stops the moment OCI returns a success. It is also deterministic: the retry interval and the maximum number of attempts are written down in code, so the behavior is repeatable and auditable across your whole team.

Automation is a recurring theme in Linux administration. For a lighter-weight example of scripting terminal workflows without Ansible, see our guide on how to automate Linux with ShellGPT.

Prerequisites

Before you start, make sure the following items are in place on your Ubuntu host and in your Oracle account:

• An Ubuntu 24.04 LTS server or workstation with outbound internet access.
• Docker Engine installed and running on the Ubuntu host.
• An Oracle Cloud Infrastructure account with an active Resource Manager stack that provisions the failing instance.
• An OCI API key pair generated and stored locally as a .pem file.
• Basic familiarity with the OCI Console, shell commands, YAML, and Ansible concepts.
• A text editor such as nano or vim for writing the playbook.

Step 1: Prepare the OCI Config File on Ubuntu

The OCI SDK and Ansible modules authenticate by reading a plain-text config file that points to your API key. On your Ubuntu host, create a dedicated directory, place both the private key and the configuration inside it, and mount that directory into the Semaphore container. This example uses /opt/oci as the host folder.

1.1 Create the Working Directory

Create the folder and secure the private key that you will download from the OCI Console:

lc-root@ubuntu:~$ sudo mkdir -p /opt/oci
lc-root@ubuntu:~$ sudo chown $USER:$USER /opt/oci

1.2 Gather Your Credentials from the OCI Console

Open your profile menu in the upper-right corner of the OCI Console and collect the following values:

• User OCID: Open User Settings and copy the OCID from the User Details page.
• Tenancy OCID: Open Tenancy from the same profile menu and copy the OCID.
• Region: Note the API region such as us-chicago-1 or eu-frankfurt-1.
• API key: Under API keys, choose Generate API Key Pair, download the private key, and copy the fingerprint from the generated preview.

Upload the downloaded private key file to your Ubuntu host and place it in the working directory as oci_api_key.pem, then restrict its permissions:

lc-root@ubuntu:~$ cp /path/to/your_private_key.pem /opt/oci/oci_api_key.pem
lc-root@ubuntu:~$ chmod 600 /opt/oci/oci_api_key.pem

1.3 Write the config File

Create a file named config (without any .txt extension) in /opt/oci. Because Docker mounts this folder into the container, point the key_file at the Linux path that Ansible will see inside the container:

[DEFAULT]
user=ocid1.user.oc1..aaaaaaaaexample
fingerprint=12:34:56:78:9a:bc:de:f0:12:34:56:78:9a:bc:de:f0
tenancy=ocid1.tenancy.oc1..aaaaaaaaexample
region=us-chicago-1
key_file=/etc/oci/oci_api_key.pem

Keep the key_file value as /etc/oci/oci_api_key.pem. This path exists only inside the container and maps back to /opt/oci/oci_api_key.pem on the host. Never use a path such as /opt/oci/oci_api_key.pem here, because the Ansible modules run inside the container and cannot see the host filesystem directly.

Step 2: Deploy Semaphore UI with Docker

Semaphore UI provides a friendly web interface on top of Ansible, including task scheduling, live logs, inventories, and variable groups. This container-based approach follows the same portable pattern you would use to deploy applications with Docker on Linux. Run the container from your Ubuntu shell and mount /opt/oci into the container as /etc/oci.

docker run -d --name semaphore -p 3000:3000 \
  -e SEMAPHORE_DB_DIALECT=sqlite \
  -e SEMAPHORE_DB_PATH=/etc/oci/semaphore.db \
  -e SEMAPHORE_ADMIN=admin \
  -e SEMAPHORE_ADMIN_PASSWORD=adminpass \
  -e SEMAPHORE_ADMIN_NAME=Admin \
  -e SEMAPHORE_ADMIN_EMAIL=admin@localhost \
  -v /opt/oci:/etc/oci \
  semaphoreui/semaphore:latest

Note that several generations of Semaphore renamed its embedded database dialect between bolt, boltdb, and finally sqlite. Current releases only accept sqlite. If the container exits immediately with an Unknown database dialect error, you used one of the older names; re-create it with SEMAPHORE_DB_DIALECT=sqlite.

2.1 Persist the Semaphore Database on the Volume

Semaphore stores every project, task template, environment variable, and run history in a single database file. The docker run command above points that database at SEMAPHORE_DB_PATH=/etc/oci/semaphore.db. Because the entire /opt/oci folder is mounted into the container at /etc/oci, this file physically lives on your Ubuntu host rather than inside the container’s writable layer. A docker restart, docker stop, and even docker rm never touch files inside a mounted volume, so your whole Semaphore configuration survives any container lifecycle event and is yours for as long as the host folder remains on disk.

Confirm the database really lives on the host:

lc-root@ubuntu:~$ ls -la /opt/oci
lc-root@ubuntu:~$ docker exec -it semaphore ls -la /etc/oci

You should see semaphore.db in both listings because the two paths are mirrors of the same folder. If it only appears in the second listing, the host mount is not configured correctly.

Back up your projects and history at any time by copying the database file:

lc-root@ubuntu:~$ cp /opt/oci/semaphore.db /opt/oci/semaphore_backup.db

2.2 Verify the Container Started

Confirm that the container is running and listening on port 3000:

lc-root@ubuntu:~$ docker ps
lc-root@ubuntu:~$ docker logs semaphore

If the log shows a healthy startup and no errors, move on to the configuration step.

2.3 Create the First Project

Open http://localhost:3000 in a browser and sign in with admin / adminpass. On first launch Semaphore asks you to create a project. Give it a meaningful name such as OCI Automation and create it. This project holds all of the credentials, inventories, variables, repositories, and task templates that you configure next.

Step 3: Configure Semaphore Projects

Inside the project, set up the building blocks that your playbook references: a key, an inventory, a variable group, and a repository.

3.1 Add a Key

Open Key Store and create a new key named OCI Credentials with the type None. Even though local execution uses no SSH, Semaphore requires an associated credential for each environment resource.

3.2 Add an Inventory

Open Inventory and create a new static inventory named Localhost. Attach the OCI Credentials key and set the static inventory content to the following single line so Ansible runs entirely inside the container without any SSH connection:

localhost ansible_connection=local

3.3 Add a Variable Group

Open Variable Groups (or Environment) and create a new environment named OCI Stack Env. In the Extra Variables field, store your stack identifier. Copy the stack OCID from the Resource Manager > Stacks detail page in the OCI Console.

{
  "stack_ocid": "ocid1.ormstack.oc1.us-chicago-1.aaaaaaaaexample"
}

3.4 Add a Repository

Semaphore insists that every playbook live inside a repository. Open Repositories and create one named Local OCI Repo that points at the mounted folder path /etc/oci, set the branch to main, and attach the OCI Credentials key. This way Semaphore reads retry_stack.yml straight from your Ubuntu host folder through the Docker mount.

Step 4: Create the Retry Ansible Playbook

Create a file named retry_stack.yml inside /opt/oci. A well-designed playbook keeps all of the logic in one place: it installs the OCI Python SDK if needed, submits the APPLY job, and automatically retries on failure at a fixed interval.

lc-root@ubuntu:~$ nano /opt/oci/retry_stack.yml

Paste the following content into the file and save it:

---
- name: Auto-retry OCI Stack Apply
  hosts: localhost
  gather_facts: no

  tasks:
    - name: Ensure OCI Python SDK is installed
      ansible.builtin.pip:
        name: oci

    - name: Run Apply Job (Retry on Capacity Error)
      oracle.oci.oci_resource_manager_job:
        config_file_location: "/etc/oci/config"
        stack_id: "{{ stack_ocid }}"
        job_operation_details:
          operation: "APPLY"
          execution_plan_strategy: "AUTO_APPROVED"
      register: apply_result
      until: apply_result is not failed
      retries: 100
      delay: 600

4.1 What Each Task Does

• hosts: localhost and gather_facts: no tell Ansible to run inside the container and skip system fact collection for speed.
• ansible.builtin.pip installs the oci Python package. This task is idempotent: if the SDK is already present, it finishes within a second and moves on. Because it runs on every execution, it survives container restarts without any manual install.
• oracle.oci.oci_resource_manager_job sends the APPLY operation to the stack referenced by stack_ocid using the credentials at /etc/oci/config. The AUTO_APPROVED execution strategy removes the need for a manual approval step.
• register: apply_result captures the module result in a variable so the retry condition can inspect it.
• until: apply_result is not failed is the retry core. When OCI reports Out of host capacity, the job lifecycle state becomes FAILED, the condition stays unsatisfied, and Ansible waits and retries.
• delay: 600 waits ten minutes between attempts, while retries: 100 allows up to one hundred attempts, giving roughly sixteen hours of automated retrying.

4.2 Match the Retry Timing to the Free Tier

Free-tier capacity can free up at any time of day. A ten-minute interval balances aggressiveness against the risk of hammering the API. If your region frees capacity rarely, raise delay to 900 or 1800; if it frees capacity quickly, lower it to 60. Adjust retries to fit how long you are willing to keep trying.

Step 5: Run and Monitor the Task

Open Task Templates in Semaphore and create a new template. Name it Retry OCI Stack Apply, select the Local OCI Repo repository, enter retry_stack.yml as the playbook path, and bind the Localhost inventory along with the OCI Stack Env variable group.

# Optional: verify the mounted files are visible inside the container
docker exec -it semaphore ls -la /etc/oci

Hit the green Run button next to the template. Semaphore executes the playbook inside the container, and the task log streams live output. When capacity is missing, you will see the apply job fail, followed by a quiet ten-minute pause and a fresh apply attempt, repeating until OCI succeeds. Leave the browser tab open or use the Dashboard to watch the progress in real time.

Troubleshooting Common Errors

The following failures are the most frequent stumbling blocks when building this exact setup, together with their fixes.

Unknown database dialect: bolt

Old Semaphore guides mention bolt as the built-in database. Current images reject it. Re-create the container using SEMAPHORE_DB_DIALECT=sqlite and, for persistence, point SEMAPHORE_DB_PATH at the mounted volume such as /etc/oci/semaphore.db.

YAML parsing failed: Tabs are usually invalid in YAML

YAML forbids tab characters. If you paste a playbook into a rich-text editor, hidden tabs can sneak in. Rewrite the file with a shell tool such as nano that preserves spaces, and verify that indentation uses spaces only.

couldn’t resolve module/action ‘oracle.oci.oci_resource_manager_job’

This means the Oracle collection is missing. Semaphore installs it automatically when it finds a requirements.yml file in the repository root. Place the following file in /opt/oci:

collections:
  - name: oracle.oci

Module failed: oci python sdk required for this module

The collection is present, but the oci Python SDK is not installed in the Ansible virtual environment. Installing it manually with docker exec works only until the container restarts. The robust fix is the ansible.builtin.pip task at the top of the playbook, which reinstalls the SDK automatically on every run and survives restarts.

Could not find config file at /etc/oci/config

The config file is missing, misnamed, or the mount is misconfigured. Confirm that /opt/oci/config exists without an extension and that the key_file line points to /etc/oci/oci_api_key.pem rather than a host path.

Operation failed as resource entered into a failure state: FAILED

The apply module treats a failed job as a hard error and stops the playbook. Instead of trying to inspect a failed job ID, move the retry loop onto the apply task itself using register: apply_result together with until: apply_result is not failed. This approach neither depends on a job ID nor blocks on the synchronous wait.

object of type ‘dict’ has no attribute ‘job’

After a failed apply, the returned JSON contains no job.id field, so a second facts task cannot reference it. The single-task retry pattern above avoids this entirely, because it never needs the job identifier between tasks.

Best Practices

Follow these guidelines to keep your OCI auto-retry automation reliable and maintainable.

• Keep dependencies declarative: Use requirements.yml for the Oracle collection and a pip task for the Python SDK so nothing depends on manual container edits.
• Persist Semaphore on a volume: Point SEMAPHORE_DB_PATH at the mounted folder (for example /etc/oci/semaphore.db) and set SEMAPHORE_DB_DIALECT=sqlite. Projects, templates, variables, and run history then live on the host and survive any container restart or deletion.
• Tune the retry window: Adjust delay and retries to match how quickly your free-tier region typically frees capacity; ten minutes and one hundred attempts is a sensible starting point.
• Respect YAML rules: Use spaces for indentation and avoid copy-pasting playbooks from rich-text editors that can insert invisible tabs.
• Separate secrets from code: Keep the API key and stack OCID in variable groups and the config file, never hard-coded into the playbook itself.
• Restrict key permissions: Set chmod 600 on the private key so only the owning user can read it.

Conclusion

Treating the OCI out of host capacity error as a transient condition and letting an automated playbook handle the retries turns a frustrating manual chore into a one-click operation. This is especially valuable on the Always Free tier, where capacity is rarely guaranteed and free compute shapes can fail repeatedly before a host finally opens up. By combining a mounted OCI config file, a Dockerized Semaphore UI, and a single self-healing Ansible playbook on a Ubuntu 24.04 host, you can keep provisioning your stack until OCI grants the capacity you need. Finally, remember to place the retry logic directly on the apply task and to declare your dependencies so the solution survives restarts without any manual intervention.