Vault
SSH secrets engine: One-time SSH password
In a distributed cloud environment, tenant, and system is increasingly important part of online security. If an attacker gains access to your virtual machines, they can get control of most running applications, local data as well as its connected machines and systems.
The Vault SSH secrets engine provides secure authentication and authorization for access to machines via the SSH protocol. It supports signed SSH certificate and one-time SSH password modes. This tutorial demonstrates the one-time SSH password mode.
Personas
The end-to-end scenario described in this tutorial involves two personas:
operations
with privileged permissions to setup SSH secrets engineclient
trusted entity to request SSH OTP from Vault
Challenge
SSH servers provide a range of authentication methods, but most use password based authentication by default. If any user on the system has a fairly weak password, an attacker can more readily determine the password and create an unauthorized SSH connection.
Solution
Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.
![SSH OTP Workflow](/_next/image?url=https%3A%2F%2Fcontent.hashicorp.com%2Fapi%2Fassets%3Fproduct%3Dtutorials%26version%3Dmain%26asset%3Dpublic%252Fimg%252Fvault%252Fdiagram-vault-ssh-otp.png%26width%3D964%26height%3D412&w=2048&q=75&dpl=dpl_3YmAyyr6aAD9giE8afyxHF2M53Fo)
An authenticated client requests an OTP from the Vault server. If the client is authorized, Vault issues and returns an OTP. The client uses this OTP during the SSH authentication to connect to the desired target host.
When the client establishes an SSH connection, the OTP is received by the Vault helper which validates the OTP with the Vault server. The Vault server then deletes this OTP, ensuring that it is only used once.
Since the Vault server is contacted during SSH connection establishment, every login attempt and the correlating Vault lease information can be logged to an audit device.
Note
Vault SSH Helper version 0.1.6 and higher supports the Vault Enterprise namespaces feature.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.
Prerequisites
This lab was tested on macOS using an x86_64 based processor. If you are running macOS on an Apple silicon-based processor, use a x86_64 based Linux virtual machine in your preferred cloud provider.
To perform the tasks described in this tutorial, you need to have:
- HCP or Vault Community Edition environment
- Vagrant installed with a configured provider such as VirtualBox
- jq installed
Policy requirements
Note
For the purpose of this tutorial, you can use root
token to work
with Vault. However, it is recommended that root tokens are only used for
initial setup or in emergencies. As a best practice, use tokens with
appropriate set of policies based on your role in the organization.
To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:
# To enable secrets engines
path "sys/mounts/*" {
capabilities = [ "create", "read", "update", "delete" ]
}
# To configure the SSH secrets engine
path "ssh/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
# To enable the userpass auth method
path "sys/auth/userpass" {
capabilities = [ "update" ]
}
# To create the policy for the test user
path "sys/policies/acl/test" {
capabilities = [ "read", "update" ]
}
# To create the test user
path "auth/userpass/users/ubuntu" {
capabilities = [ "create", "update" ]
}
If you are not familiar with policies, complete the policies tutorial.
Lab setup
Start Vault
Open a terminal session and determine your local workstations IP address.
$ ifconfig | grep inet
Depending on your operating system and workstations configuration, you may have multiple addresses. Copy the address that matches your network. For example, if your network uses a subnet of
192.168.100.0/24
copy the192.168.100.101
address.Example output:
inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet6 fe80::aede:48ff:fe00:1122%en5 prefixlen 64 scopeid 0x4 inet6 fe80::1c44:73de:cb90:5f84%en0 prefixlen 64 secured scopeid 0x6 inet 192.168.100.101 netmask 0xffffff00 broadcast 192.168.100.255 inet6 fe80::7ceb:9fff:fe93:555c%awdl0 prefixlen 64 scopeid 0x7 inet6 fe80::7ceb:9fff:fe93:555c%llw0 prefixlen 64 scopeid 0x8
Start a Vault dev server with
root
as the root token and IP address copied from the previous step.$ vault server -dev -dev-root-token-id root -dev-listen-address <copied-address>:8200
The Vault dev server defaults to running at the provided IP address on port 8200. The server is initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.
Open another terminal session, and export an environment variable for the
vault
CLI to address the Vault server.$ export VAULT_ADDR=http://<ip-address-copied-from-previous-step>:8200
Export an environment variable for the
vault
CLI to authenticate with the Vault server.$ export VAULT_TOKEN=root
Note
For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.
The Vault server is ready.
Start Ubuntu
Open another terminal session and initialize a new Ubuntu Vagrant box.
$ vagrant init generic/ubuntu2204
To successfully complete this tutorial, your Ubuntu box needs to be on the same network as your workstation. Edit the
Vagrantfile
to connect to your local network.$ sed -ibak "s/# config.vm.network \"public_network\"/config.vm.network \"public_network\"/g" Vagrantfile
Start the Ubuntu box.
$ vagrant up
When prompted, select the number that corresponds to your active network interface. For example, if you are connected to your local network via your Wi-Fi (Wireless) adapter, enter the number 1 and press enter.
Example output:
==> default: Available bridged network interfaces: 1) en0: Wi-Fi (Wireless) 2) en5: USB Ethernet(?) 3) awdl0 4) llw0 5) en2: Thunderbolt 2 6) en1: Thunderbolt 1 7) en3: Thunderbolt 3 8) en4: Thunderbolt 4 9) bridge0 ==> default: When choosing an interface, it is usually the one that is ==> default: being used to connect to the internet. ==> default: default: Which interface should the network bridge to?
You are ready to proceed with the tutorial.
Setup the SSH secrets engine
(Persona: operations)
On the Vault server, you must enable the SSH secrets engine and create a role.
Return to the terminal session where you created the
VAULT_ADDR
environment variable.Enable the SSH secrets engine.
$ vault secrets enable ssh Success! Enabled the ssh secrets engine at: ssh/
Create a role named
otp_key_role
withkey_type
set tootp
.$ vault write ssh/roles/otp_key_role \ key_type=otp \ default_user=vagrant \ cidr_list=0.0.0.0/0
This role defaults to creating credentials for the
vagrant
user and will allow all remote hosts whose IP addresses fit within this CIDR range.
Security: The tutorial uses a very permissive cidr_list
value of
0.0.0.0/0
. In production, we recommend that roles are defined with a range
that is granular to the range of remote hosts. We also recommend that you create
one role for each username to ensure isolation between usernames.
Setup the client authentication
(Persona: operations)
On the Vault server, you must create a policy to allow access to the SSH OTP role and then attach the policy to an authentication method.
Create a policy file named,
test.hcl
, that provides access to thessh/creds/otp_key_role
path.$ tee test.hcl <<EOF # To list SSH secrets paths path "ssh/*" { capabilities = [ "list" ] } # To use the configured SSH secrets engine otp_key_role role path "ssh/creds/otp_key_role" { capabilities = ["create", "read", "update"] } EOF
The path
ssh/creds/otp_key_role
is the path to the role created in the Setup the SSH secrets engine section.Create a policy named
test
with the policy defined intest.hcl
.$ vault policy write test ./test.hcl
Enable the
userpass
auth method.$ vault auth enable userpass
Create a user named
ubuntu
with the password "training" assigned thetest
policy.$ vault write auth/userpass/users/ubuntu password="training" policies="test"
Install vault-ssh-helper
(Persona: operations)
SELinux users
If you use SELinux in enforcing mode, you need to define and enable a module that allows vault-ssh-helper
to open and write log files, and communicate with the Vault server. Creating and enabling the SELinux module is beyond the scope of this tutorial. If you do not require SELinux enforcement, then operating SELinux in permissive mode also allows vault-ssh-helper
to function.
On each Remote host, you must:
- Install vault-ssh-helper
- Create a configuration file for the vault-ssh-helper
- Modify both the Pluggable Authentication Module (PAM) sshd configuration file and the sshd configuration file
- Restart the sshd service
Note
The example scenario in this tutorial uses the Ubuntu Linux distribution. If you use a different distribution, you will need to update paths and file locations to match your distribution.
Return to the terminal where you performed
vagrant up
and connect to Ubuntu.$ vagrant ssh
Download and install version
0.2.1
ofvault-ssh-helper
from releases.hashicorp.com.$ wget https://releases.hashicorp.com/vault-ssh-helper/0.2.1/vault-ssh-helper_0.2.1_linux_amd64.zip
Unzip the binary from the archive into
/usr/local/bin
.$ sudo unzip -q vault-ssh-helper_0.2.1_linux_amd64.zip -d /usr/local/bin
Set the
vault-ssh-helper
binary's permissions to executable.$ sudo chmod 0755 /usr/local/bin/vault-ssh-helper
Set the
vault-ssh-helper
binary's user and group toroot
.$ sudo chown root:root /usr/local/bin/vault-ssh-helper
Create a directory to store the configuration file.
$ sudo mkdir /etc/vault-ssh-helper.d/
The Vault SSH Helper reads a configuration at the path
/etc/vault-ssh-helper.d/config.hcl
with this format:vault_addr = "<VAULT_EXTERNAL_ADDR>" tls_skip_verify = false ca_cert = "<PEM_ENCODED_CA_CERT>" ssh_mount_point = "ssh" namespace = "my_namespace" allowed_roles = "*"
The
vault_addr
is the network address of the Vault server configured to generate the OTP.tls_skip_verify
enables or disables TLS verification.ca_cert
is the path to the PEM-encoded CA certificate files used to verify the Vault server's TLS certificate. Whenvault-ssh-helper
is run with the-dev
flag this is ignored.ssh_mount_point
is the Vault server path where the SSH secrets engine is enabled.namespace
is the namespace of the SSH mount point (Vault Enterprise and Vault Dedicated)allowed_roles
defines all*
or a comma-separated list of allowed roles defined in the SSH secrets engines.Refer to the documentation for the entire list of configuration properties.
Return to the terminal session where you created the
VAULT_ADDR
environment variable.Retrieve and copy the value for
VAULT_ADDR
.$ echo $VAULT_ADDR
Return to the terminal connected to the Ubuntu box.
Create a variable named
VAULT_EXTERNAL_ADDR
that contains the Vault server's external network address copied in the previous step.$ VAULT_EXTERNAL_ADDR=<VAULT_EXTERNAL_ADDR>
Example:
$ VAULT_EXTERNAL_ADDR=https://vault-cluster-public-vault-abcdefg.123456.z1.hashicorp.cloud:8200
Create a Vault SSH Helper configuration file
/etc/vault-ssh-helper.d/config.hcl
.$ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOF vault_addr = "$VAULT_EXTERNAL_ADDR" tls_skip_verify = false ssh_mount_point = "ssh" allowed_roles = "*" EOF
Backup the original PAM sshd configuration configuration file.
$ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig
Open the file in your preferred text editor.
$ sudo nano /etc/pam.d/sshd
The
common-auth
must be commented out or removed to disable the standard Unix authentication and replaced with authentication throughvault-ssh-helper
. Finally, a workaround for a bug that exists with some versions ofpam_exec.so
must also be included.Refer to the documentation for details about these parameter settings.
Example:
# PAM configuration for the Secure Shell service # Standard Un*x authentication. #@include common-auth auth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -dev -config=/etc/vault-ssh-helper.d/config.hcl auth optional pam_unix.so not_set_pass use_first_pass nodelay ...
The
vault-ssh-helper
is configured to run in development using the-dev
parameter. The configuration fileconfig.hcl
is specified with-config
parameter and while running it logs output to/var/log/vault-ssh.log
using the-log
parameter.Backup the original
sshd_config
configuration file.$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig
Modify the sshd configuration file.
$ sudo nano /etc/ssh/sshd_config
Add or set the following:
KbdInteractiveAuthentication yes UsePAM yes PasswordAuthentication no
This enables the keyboard-interactive authentication and PAM authentication modules. The password authentication is disabled.
Note
Older version of Ubuntu use
ChallengeResponseAuthentication
instead ofKbdInteractiveAuthentication
.Caution: You should ensure that there is another means of access to the server (such as a previously established SSH key for the root user) when disabling password authentication so that you are not prevented from logging in at all.
Restart the
sshd
service.$ sudo systemctl restart sshd
Verify the configuration.
$ vault-ssh-helper -verify-only -dev -config /etc/vault-ssh-helper.d/config.hcl ==> WARNING: Dev mode is enabled! [INFO] using SSH mount point: ssh [INFO] vault-ssh-helper verification successful!
Note
These steps must be performed on all remote hosts.
Determine the IP address of the Ubuntu box.
$ ip -4 address show eth1 | grep inet | awk '{print $2}' | cut -d'/' -f1 192.168.100.102
Note
By following this tutorial, you should have two
eth
adapters. Theeth0
adapter represents the internal VirtualBox network and theeth1
adapter represents the bridged connection to your local network.Copy the IP address for
eth1
.Disconnect from Ubuntu.
$ exit
Generate an OTP
(Persona: client)
A client configured to target a Vault server may now authenticate with the Vault server, generate an OTP and then use that OTP to establish an SSH connection.
Return to the terminal configured with the
VAULT_ADDR
environment variable.Create a variable named
REMOTE_HOST_IP
that stores the Ubuntu box's IP address.$ REMOTE_HOST_IP=<REMOTE_HOST_IP>
Example:
$ REMOTE_HOST_IP=192.168.100.102
Authenticate with Vault using the
userpass
auth method with the usernameubuntu
and passwordtraining
. Store the token in an environment variable.$ UBUNTU_TOKEN=$(vault login -method=userpass username=ubuntu password=training -format=json | jq -r '.auth | .client_token')
Generate an OTP, through the
otp_key_role
, for remote host given its IP address.$ VAULT_TOKEN=$UBUNTU_TOKEN vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP
Example Output:
$ vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP Key Value --- ----- lease_id ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0a lease_duration 768h lease_renewable false ip 192.168.100.102 key f1cb47ad-6255-0be8-6bd8-5c4b3b01c8df key_type otp port 22 username vagrant
The output displays a
key
. Its value is the OTP to use during SSH authentication.
Initiate an SSH session
Connect to the Ubunut box as the
vagrant
user and the one time password provided by Vault.$ ssh -o PubkeyAuthentication=no vagrant@$REMOTE_HOST_IP
Note
To demonstrate one time passwords, include the
PubkeyAuthentication=no
parameter to ensure your native SSH client does include any cached SSH keys.When prompted enter
yes
to accept the fingerprint and for aPassword:
enter the OTP key from the previous step.Example:
$ ssh -o PubkeyAuthentication=no vagrant@$REMOTE_HOST_IP The authenticity of host '192.168.100.102 (192.168.100.102)' can't be established. ECDSA key fingerprint is SHA256:N8CjNy6ahilQ3xvOO6FZKA3NkvpIOPi1LxpLvNx7xbQ. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Password: <Enter OTP> Last login: Tue Aug 16 13:20:22 2022 from 10.0.2.2 vagrant@ubuntu2204:~$
You are now connected to the Ubuntu box using the one time password from Vault.
Cleanup
Disconnect from the Ubuntu box.
$ exit
Switch to the terminal where you performed
vagrant up
and destroy the Ubuntu box.$ vagrant destroy --force
Stop the Vault dev mode server or delete your Vault Dedicated server.