nomad operator raft migrate-backend command reference

The nomad operator raft migrate-backend command migrates the Raft log store from the BoltDB backend to the WAL (Write-Ahead Log) backend. You must run this command on a stopped Nomad server with direct access to the server's data directory.

The WAL backend can provide better performance characteristics for write-heavy workloads compared to BoltDB. However, once migrated to WAL, reverting back to BoltDB requires restoring from a snapshot taken before the migration.

Usage

nomad operator raft migrate-backend [options] <path to data dir>

Arguments

• <path to data dir>: The path to the Nomad server's data directory. This is typically the path specified in the server's data_dir configuration parameter.

Options

• -wal-segment-size: (int: 64) - The size in megabytes for WAL segment files. This should match the raft_logstore.wal.segment_size_mb configuration that is used when the server is restarted. Default is 64MB.

• -force: Skip the confirmation prompt and proceed with the migration immediately.

Migration process

The migration process performs the following actions:

1. Validates that the source data directory exists and contains a BoltDB-based Raft log store.
2. Creates a backup of the current BoltDB log store.
3. Reads all log entries from the BoltDB store.
4. Writes all log entries to the new WAL-based log store.
5. Preserves stable store metadata, such as current term and voted for.

During migration, the command displays progress information including the number of log entries migrated.

Examples

Basic migration

Migrate from BoltDB to WAL with default settings:

$ sudo nomad operator raft migrate-backend /var/nomad/data

Migration with custom WAL segment size

Migrate with a custom WAL segment size of 128MB.

$ sudo nomad operator raft migrate-backend -wal-segment-size=128 /var/nomad/data

Non-Interactive Migration

Skip the confirmation prompt.

$ sudo nomad operator raft migrate-backend -force /var/nomad/data

Post-Migration steps

Perform these steps after successfully migrating the Raft log store backend:

1. Update the server configuration file to use the WAL backend.

   server {
     raft_logstore {
       backend = "wal"
       wal {
         segment_size_mb = 64
       }
     }
   }
   

2. Start the Nomad server.

3. Verify the server starts successfully and joins the cluster.

4. Monitor the server logs for any issues related to the Raft log store.

Troubleshooting

Migration fails

If the migration fails partway through:

• Check the command output for specific error messages.
• Ensure sufficient disk space is available. The WAL backend initially requires approximately the same space as the BoltDB store.
• Verify file system permissions on the data directory.
• The original BoltDB files are preserved if migration fails, so you can retry the migration after addressing the issue.

Server won't start after migration

If the server fails to start after migration:

• Verify the raft_logstore configuration in the server config file matches the migration parameters.
• Check the server logs for specific error messages.
• If necessary, restore from the backup created before migration.

Best practices

• Create a Snapshot: Before migrating, create a Nomad snapshot using nomad operator snapshot save as an additional backup.
• Test in Non-Production: Test the migration process in a non-production environment first to understand the process and timing.
• Plan for Downtime: The server must be stopped during migration. Plan for this downtime, especially in production environments.
• Monitor Disk Space: Ensure adequate disk space is available for both the source BoltDB store and the new WAL store during migration.
• Rolling Migration: In a multi-server cluster, migrate servers one at a time to maintain cluster availability. Ensure each server successfully joins and stabilizes before migrating the next.