Podman task driver plugin

Name: podman

The Podman task driver plugin for Nomad uses the Podman daemonless container runtime for executing Nomad tasks. Podman supports OCI containers and its command line tool serves as a drop-in replacement for Docker.

You can find the Podman task driver source in the nomad-driver-podman GitHub repository. HashiCorp maintains the plugin.

Releases

Installation

Usage

The example job created by nomad init -short can be adapted to use Podman instead:

job "redis" {
  group "cache" {
    network {
      port "redis" { to = 6379 }
    }

    task "redis" {
      driver = "podman"
      config {
        image = "docker.io/library/redis:7"
        ports = ["redis"]
      }
    }
  }
}

Refer to the project's homepage for details.

Client requirements

The Podman task driver is not built into Nomad. You must download the driver onto the client host in the configured plugin directory.

• Linux host with podman installed
• For rootless containers, you need a system supporting cgroups v2 and a few other things. Refer to the Podman rootless tutorial for instructions.

You need a v3.x or higher Podman binary and a system socket activation unit. Refer to the Podman socket activation tutorial for more information.

We recommend installing Podman using your system's package manager, which configures systemd for you.

Ensure that Nomad can find the plugin. Refer to plugin_dir for details.

Capabilities

The podman driver implements the following capabilities.

Task configuration

• apparmor_profile - (Optional) Name of an AppArmor profile to use instead of the default profile. The special value unconfined disables AppArmor for this container.

  config {
    apparmor_profile = "your-profile"
  }
  

• args - (Optional) A list of arguments to the optional command. If no command is specified, the arguments are passed directly to the container.

  config {
    args = [
      "arg1",
      "arg2",
    ]
  }
  

• auth - (Optional) Authenticate to the image registry using a static credential. Setting tls_verify to false lets the driver use self-signed certificates or plain HTTP connections to the registry.

  config {
    image = "your.registry.tld/some/image"
    auth {
      username = "someuser"
      password = "sup3rs3creT"
      tls_verify = false
    }
  }
  

• auth_soft_fail - (Optional) Ignore errors returned from auth backend so Podman can fall back to a different auth method.

  config {
    auth_soft_fail = true
  }
  

• cap_add - (Optional) A list of Linux capabilities as strings to pass to --cap-add.

  config {
    cap_add = [
      "SYS_TIME"
    ]
  }
  

• cap_drop - (Optional) A list of Linux capabilities as strings to pass to --cap-drop.

  config {
    cap_drop = [
      "MKNOD"
    ]
  }
  

• command - (Optional) The command to run when starting the container.

  config {
    command = "some-command"
  }
  

• cpu_hard_limit - (Optional) true or false. Use hard CPU limiting instead of soft limiting. By default this is false, which means Podman uses soft limiting so that containers are able to burst above their CPU limit when there is idle capacity.

• cpu_cfs_period - (Optional) Set the CPU period for the Completely Fair Scheduler (CFS), which is a duration in microseconds. Refer to podman run --cpu-period for details.

• devices - (Optional) A list of host-device[:container-device][:permissions] definitions. Each entry adds a host device to the container. Optional permissions specify device permissions as a combination of r for read, w for write, and m for mknod(2). Refer to Podman's documentation for more details.

  config {
    devices = [
      "/dev/net/tun"
    ]
  }
  

• entrypoint - (Optional) The entrypoint for the container. Defaults to the entrypoint set in the image.

  config {
    entrypoint = "/entrypoint.sh"
  }
  

• extra_hosts - (Optional) Set additional hosts in the container.

  config {
    extra_hosts = ["test4.localhost:127.0.0.2", "test6.localhost:[::1]"]
  }
  

• force_pull - (Optional) true or false (default). Always pull the latest image on container start.

  config {
    force_pull = true
  }
  

• hostname - (Optional) The hostname to assign to the container. When launching more than one of a task (using count) with this option set, every container the task starts has the same hostname.

• image - The image to run. Accepted transports are docker (default if missing), oci-archive, and docker-archive. Archive references can point to local files or http:// and https:// URLs. Images referenced as short-names are treated according to user-configured preferences.

  config {
    image = "docker://redis"
  }
  

  config {
    image = "oci-archive:https://artifacts.example.com/images/myapp.tar"
  }
  

• os - (Optional) Override the OS used when pulling the image. This mirrors Podman's --os flag.

• arch - (Optional) Override the architecture used when pulling the image. This mirrors Podman's --arch flag.

• variant - (Optional) Override the variant used when pulling the image. This mirrors Podman's --variant flag.

  config {
    image   = "docker.io/library/alpine:latest"
    os      = "linux"
    arch    = "amd64"
    variant = "v8"
  }
  

  When you set os, arch, or variant, the driver always pulls the image so it fetches the correct platform variant. Podman skips the download when the matching image is already present. Nomad servers cannot validate these overrides at scheduling time, so use constraint blocks to pin the workload to compatible nodes.

• image_pull_timeout - (Optional) Time duration for your pull timeout (default to "5m"). Cannot be longer than the client_http_timeout.

  config {
    image_pull_timeout = "5m"
  }
  

• init - (Optional) Run an init inside the container that forwards signals and reaps processes.

  config {
    init = true
  }
  

• init_path - (Optional) Path to the container-init binary.

  config {
    init = true
    init_path = "/usr/libexec/podman/catatonit"
  }
  

• ipc_mode - (Optional) Set the IPC mode for the container. The default is Podman's private namespace.

  • host - Use the host IPC namespace.
  • private - Use a private IPC namespace.
  • shareable - Use a private IPC namespace that other containers may join.
  • none - Use a private IPC namespace without /dev/shm mounted.
  • container:<id> - Join another container's IPC namespace.
  • ns:<path> - Join the IPC namespace at a path.
  • task:<name> - Join another task's IPC namespace in the same allocation.
  config {
    ipc_mode = "task:mps-daemon"
  }
  

  If you set shm_size, the IPC mode must create a private or shareable namespace.

• labels - (Optional) Set labels on the container.

  config {
    labels = {
      "nomad" = "job"
    }
  }
  

• logging - (Optional) Configure logging. Also refer to the plugin option disable_log_collection.

  • driver = "nomad" - (Default) Podman redirects its combined stdout/stderr logstream directly to a Nomad fifo. This mode has zero overhead and requires no log rotation at the system or Podman level. The trade-off is that you cannot ship the logstream to a log aggregator and stdout/stderr is multiplexed into a single stream.

    config {
      logging = {
        driver = "nomad"
      }
    }
    

  • driver = "journald" - The container log is forwarded from Podman to the journald on your host. The Podman API then pulls the log back from the journal into the Nomad fifo (controllable by disable_log_collection). All containers can log to the host journal, and you can ship a structured stream including metadata to your log aggregator. No log rotation at the Podman level. You can add additional tags to the journal. The trade-offs are additional overhead, a dependency on Journal (does not work on WSL2), and the need to configure a rotation policy for your journal. Requires Podman 3.1.0 or higher.

    config {
      logging = {
        driver = "journald"
        options = [
          {
            "tag" = "redis"
          }
        ]
      }
    }
    

• memory_reservation - (Optional) Memory soft limit (units = b (bytes), k (kilobytes), m (megabytes), or g (gigabytes)).

  After setting memory reservation, when the system detects memory contention or low memory, the system forces containers to restrict their consumption to their reservation. Always set the value below --memory; otherwise, the hard limit takes precedence. By default, memory reservation is the same as memory limit.

  config {
    memory_reservation = "100m"
  }
  

• memory_swap - (Optional) A limit value equal to memory plus swap. The swap limit should always be larger than the memory value. Unit can be b (bytes), k (kilobytes), m (megabytes), or g (gigabytes). If you don't specify a unit, b is used. Set LIMIT to -1 to enable unlimited swap.

  config {
    memory_swap = "180m"
  }
  

• memory_swappiness - Tune a container's memory swappiness behavior. Accepts an integer between 0 and 100.

  config {
    memory_swappiness = 60
  }
  

• network_mode - (Optional) Set the network mode for the container. By default, the task uses the network stack defined in the task group network block. If the group's network behavior is also undefined, it falls back to bridge in rootful mode, slirp4netns for rootless containers on Podman version less than 5.0.0, or pasta for rootless containers on Podman v5.0.0 or greater.

  You can also set network_mode to the name of an existing Podman network created with podman network create. The network must already exist, otherwise the task fails with a clear error. This requires Podman 4.0 or later.

  • bridge - (Default for rootful) Create a network stack on the default Podman bridge.
  • container:id - Reuse another container's network stack.
  • host - Use the Podman host network stack. Note: the host mode gives the container full access to local system services such as D-bus and is therefore insecure.
  • none - No networking.
  • pasta - Use pasta to create a user network stack. This is the default for rootless containers on Podman v5.0.0 or greater.
  • slirp4netns - Use slirp4netns to create a user network stack. This is the default for rootless containers on Podman versions less than 5.0.0. Podman does not support slirp4netns for root containers. Refer to Podman GitHub issue "podman-pod-create --network slirp4netns is unsupported for root containers" for an explanation.
  • task:name-of-other-task - Join the network of another task in the same allocation.
  config {
    network_mode = "bridge"
  }
  

  config {
    network_mode = "mynet"
  }
  

• pids_limit - (Optional) An integer value that specifies the PID limit for the container.

  config {
    pids_limit = 64
  }
  

• ports - (Optional) Forward and expose ports. Refer to Docker driver configuration for details.

• privileged - (Optional) true or false (default). A privileged container turns off the security features that isolate the container from the host. Dropped Capabilities, limited devices, read-only mount points, AppArmor/SELinux separation, and Seccomp filters are all disabled.

• readonly_rootfs - (Optional) true or false (default). Mount the rootfs as read-only.

  config {
    readonly_rootfs = true
  }
  

• security_opt - (Optional) A list of security-related options that Podman sets in the container.

  config {
    security_opt = [
      "no-new-privileges"
    ]
  
  }
  

• selinux_opts - (Optional) A list of process labels that the container uses.

  config {
    selinux_opts = [
      "type:my_container.process"
    ]
  }
  

• shm_size - (Optional) Set the size of /dev/shm. Refer to podman run --shm-size for details.

• socket - (Optional) The name of the socket as defined in the socket block in the client agent's plugin configuration. Defaults to the socket named "default".

  config {
    socket = "app1"
  }
  

• sysctl - (Optional) A key-value map of sysctl configurations to set to the containers on start.

  config {
    sysctl = {
      "net.core.somaxconn" = "16384"
    }
  }
  

• tmpfs - (Optional) A list of /container_path strings for tmpfs mount points. Refer to podman run --tmpfs options for details.

  config {
    tmpfs = [
      "/var"
    ]
  }
  

• tty - (Optional) true or false (default). Allocate a pseudo-TTY for the container.

• volumes - (Optional) A list of host_path:container_path:options strings to bind host paths to container paths. Named volumes are not supported.

  config {
    volumes = [
      "/some/host/data:/container/data:ro,noexec"
    ]
  }
  

• working_dir - (Optional) The working directory for the container. Defaults to the default set in the image.

  config {
    working_dir = "/data"
  }
  

• ulimit - (Optional) A key-value map of ulimit configurations to set to the containers to start.

  config {
    ulimit {
      nproc  = "4242"
      nofile = "2048:4096"
    }
  }
  

• userns - (Optional) Set the user namespace mode for the container.

  config {
    userns = "keep-id:uid=200,gid=210"
  }
  

Additionally, the Podman driver supports customization of the container's user through the task's user option.

Network configuration

Nomad lifecycle hooks combined with the driver's network_mode allow very flexible network namespace definitions. This feature does not build upon the native Podman pod structure but reuses the networking namespace of one container for other tasks in the same group.

A typical example is a network server and a metric exporter or log shipping sidecar. The metric exporter needs access to a private monitoring port which should not be exposed to the network and thus is usually bound to localhost.

The nomad-driver-podman repository includes three different example jobs for such a setup. All of them start a nats server and a prometheus-nats-exporter using different approaches.

You can use curl to prove that the job is working correctly and that you can get Prometheus metrics:

curl http://your-machine:7777/metrics

Task setup - server defines the network

Reference examples/jobs/nats_simple_pod.nomad.

Here, the server task is started as main workload and the exporter runs as a poststart sidecar. Because of that, Nomad guarantees that the server is started first and thus the exporter can join the server's network namespace using network_mode = "task:server".

Task setup - a pause container defines the network

Reference examples/jobs/nats_pod.nomad.

This job demonstrates a slightly different setup. It more closely resembles the idea of a pod by starting a pause task, named pod using a prestart sidecar hook.

Next, the main workload, server is started and joins the network namespace by using the network_mode = "task:pod" block. Finally, Nomad starts the poststart sidecar exporter which also joins the network.

Task setup - shared Nomad network namespace

Reference examples/jobs/nats_group.nomad.

This example is very different. Both server and exporter join a network namespace which is created and managed by Nomad itself. Refer to Nomad's network block to get started with this generic approach.

Plugin options

The Podman plugin has options which may be customized in the agent's configuration file.

• gc block:

  • container - Defaults to true. Set this option to false to prevent Nomad from removing a container when the task exits.
  plugin "nomad-driver-podman" {
    config {
      gc {
        container = false
      }
    }
  }
  

• recover_stopped - (Deprecated) Defaults to false. Allows the driver to start and reuse a previously stopped container after a Nomad client restart. Consider a single-node system and a complete reboot. All previously managed containers are reused instead of disposed and recreated.

  Warning

  This option may cause the Nomad client to hang on startup. It now defaults to disabled and may be removed in a future release.

  plugin "nomad-driver-podman" {
    config {
      recover_stopped = false
    }
  }
  

• socket_path (string) - Defaults to unix://run/podman/io.podman when running as root or a cgroup V1 system, and unix://run/user/<USER_ID>/podman/io.podman for rootless cgroup V2 systems.

• disable_log_collection (bool: false) - Setting this to true disables Nomad log collection for Podman tasks. If you rely exclusively on host-based log aggregation, use this option to disable Nomad log collection overhead. You also lose automatic log rotation.

  plugin "nomad-driver-podman" {
    config {
      disable_log_collection = false
    }
  }
  

• client_http_timeout (string: "60s") - Default timeout used by http.Client requests.

  plugin "nomad-driver-podman" {
    config {
      client_http_timeout = "60s"
    }
  }
  

• volumes block:

  • enabled - Defaults to true. Allows tasks to bind host paths (volumes) inside their container.
  • selinuxlabel - Allows the operator to set an SELinux label to the allocation and task local bind-mounts to containers. If used with volumes.enabled set to false, the labels still apply to the standard binds in the container.
  plugin "nomad-driver-podman" {
    config {
      volumes {
        enabled      = true
        selinuxlabel = "z"
      }
    }
  }
  

If you need to configure authorization to an external registry, you can write a file containing the details in a location such as /etc/nomad.d/registry/auth.json:

{
  "auths": {
    "yourregistry": {
      "auth": "token"
    }
  }
}

To use this information, add the following to the /etc/nomad.d/nomad.env file:

REGISTRY_AUTH_FILE=/etc/nomad.d/registry/auth.json