Nomad
Integrate service mesh and API gateway
Consul service mesh is a feature that provides secure service-to-service communication with Transport Layer Security (TLS). In a service mesh, sidecar proxies run alongside each service instance, allowing the service to send requests to localhost instead of a defined upstream service address. All traffic sent to each application service travels through an outbound and inbound sidecar proxy. As a result, each service has encrypted communication and Consul enforces application routing rules with service intentions.
The next step of this monolith migration integrates Consul service mesh to allow secure TLS connections between each service. It also sets up an API gateway for external public access.
In this tutorial, you deploy a version of HashiCups that uses Consul service mesh while running on private client nodes. Then you deploy an API gateway that allows external public access the services running on the private client node.
Infrastructure overview
At the beginning of the tutorial you have a Nomad and Consul cluster with three server nodes, three private client nodes, and one publicly accessible client node. Each node runs a Consul agent and a Nomad agent.
 

This infrastructure matches the end state of the previous tutorial.
Prerequisites
This tutorial uses the infrastructure set up in a previous tutorial in this collection, Set up the cluster. Complete that tutorial to set up the infrastructure if you have not done so.
Review configuration files
In your terminal, navigate to the directory that contains the code from the learn-consul-nomad-vm code repository.
Navigate to the jobs directory.
$ cd shared/jobs
Additional files for configuring the API gateway and service intentions are located in the shared/jobs directory. These files include 04.api-gateway.config.sh, 04.api-gateway.nomad.hcl, and 04.intentions.consul.sh.
Review scripts for API gateway configuration
The Consul API gateway allows external traffic to the internal nginx service.
The repository provides a script, 04.api-gateway.config.sh, that automates the initial configuration required for Nomad and Consul.
The set up script starts by cleaning up previous Consul configurations, binding rules, auth methods, and the ingress namespace in Nomad.
/shared/jobs/04.api-gateway.config.sh
echo -e "${_COL}Clean previous configurations.${_NC}"
# Remove route for NGINX
consul config delete -kind http-route -name hashicups-http-route
# Remove Inline certificate
consul config delete -kind inline-certificate -name api-gw-certicate
# Remove API Gateway Listener
consul config delete -kind api-gateway -name api-gateway
# Remove all existing binding rules
# WARNING: if you have existing binding rules you want to maintain, modify this behavior
for i in `consul acl binding-rule list -format json | jq -r .[].ID`; do
  consul acl binding-rule delete -id=$i
done
# Delete Nomad namespace
nomad namespace delete ingress
# Delete Consul auth-method
consul acl auth-method delete -name nomad-workloads
if [ "$1 " == "-clean " ]; then
  echo -e "${_ERR}Only cleaning selected...Exiting.${_NC}"
  exit 0
  
fi
# ...
Then the script configures the Consul and Nomad ACL integration, which includes setting up the auth method, creating a binding rule, and creating the ingress namespace in Nomad.
/shared/jobs/04.api-gateway.config.sh
# ...
tee ${_JWT_FILE} > /dev/null << EOF
{
  # ...
}
EOF
# This auth method creates an endpoint for generating Consul ACL tokens 
#  from Nomad workload identities.
consul acl auth-method create \
            -name 'nomad-workloads' \
            -type 'jwt' \
            -description 'JWT auth-method for Nomad services and workloads' \
            -config "@${_JWT_FILE}"
# ...
# The 'ingress' namespace will be used to deploy the API Gateway and to identify
#   Nomad Jobs that require a token to be generated automatically.
nomad namespace apply \
    -description "namespace for Consul API Gateways" \
    ingress
# The binding-rule identifies all Nomad Jobs in the 'ingress' namespaces and 
#  uses the 'builtin/api-gateway' policy to generate a Consul ACL token for them.
# ...
tee ${_BR_FILE} > /dev/null << EOF
{
  # ...
}
EOF
curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --connect-to ${CONSUL_TLS_SERVER_NAME}:8443:${_consul_addr} \
  --cacert ${CONSUL_CACERT} \
  --data @${_BR_FILE} \
  --request PUT \
  https://${CONSUL_TLS_SERVER_NAME}:8443/v1/acl/binding-rule | jq
Finally, the script generates TLS certificates, creates a listener and route for the API gateway, and then writes them to Consul configurations.
/shared/jobs/04.api-gateway.config.sh
# ...
# Generate TLS certificates
tee ${_ssl_conf_FILE} > /dev/null << EOF
# ...
EOF
openssl genrsa -out ${_ssl_key_file}  4096 2>/dev/null
openssl req -new \
  -key ${_ssl_key_file} \
  -out ${_ssl_csr_file} \
  -config ${_ssl_conf_FILE} 2>/dev/null
openssl x509 -req -days 3650 \
  -in ${_ssl_csr_file} \
  -signkey ${_ssl_key_file} \
  -out ${_ssl_crt_file} 2>/dev/null
export API_GW_KEY=`cat ${_ssl_key_file}`
export API_GW_CERT=`cat ${_ssl_crt_file}`
# Create 'api-gateway-certificate' inline-certificate"
tee ${_GW_certificate_FILE} > /dev/null << EOF
# ...
EOF
consul config write ${_GW_certificate_FILE}
# Create 'api-gateway' HTTP listener on port 8443
tee ${_GW_config_FILE} > /dev/null << EOF
# ...
EOF
consul config write ${_GW_config_FILE}
# Create 'hashicups-http-route' HTTP route "/" > "nginx"
tee ${_GW_route_FILE} > /dev/null << EOF
# ...
EOF
consul config write ${_GW_route_FILE}
Review the API gateway jobspec
The API gateway job is constrained to a node with the ingress meta role that also runs in the ingress namespace. It has four tasks: two are for job set up and clean up, one registers the gateway with Consul, and one starts the gateway. The set up, clean up, and service registration tasks each define a lifecycle block that enforces the order in which they run.
/shared/jobs/04.api-gateway.nomad.hcl
# ...
job "api-gateway" {
  namespace = var.namespace
  constraint {
    attribute = "${meta.nodeRole}"
    operator  = "="
    value     = "ingress"
  }
  group "api-gateway" {
  # ...
  task "setup" {
    driver = "docker"
    config {
      image = var.consul_image # image containing Consul
      command = "/bin/sh"
      args = [
        "-c",
        "consul connect envoy -gateway api -register -deregister-after-critical 10s -service ${NOMAD_JOB_NAME} -admin-bind 0.0.0.0:19000 -ignore-envoy-compatibility -bootstrap > ${NOMAD_ALLOC_DIR}/envoy_bootstrap.json"
      ]
    }
    lifecycle {
      hook = "prestart"
      sidecar = false
    }
    # ...
  }
  task "api-gw" {
    # ...
    config {
      image = var.envoy_image # image containing Envoy
      args = [
        "--config-path",
        "${NOMAD_ALLOC_DIR}/envoy_bootstrap.json",
        "--log-level",
        "${meta.connect.log_level}",
        "--concurrency",
        "${meta.connect.proxy_concurrency}",
        "--disable-hot-restart"
      ]
    }
  }
  task "service_change" {
    # ...
    lifecycle {
      hook = "poststart"
    }
    # ...
    config {
      command = "consul"
      args    = ["services", "register", "${NOMAD_TASK_DIR}/svc-api-gateway.hcl"]
    }
    template {
      data = <<EOF
        service {
          id      = "api-gateway"
          name    = "api-gateway"
          kind    = "api-gateway"
          port    = 8443
          address = ""
          meta = {
            public_address = "https://{{ env "attr.unique.platform.aws.public-ipv4" }}:8443"
          }
        }
      EOF
      destination = "${NOMAD_TASK_DIR}/svc-api-gateway.hcl"
    }
  }
  task "cleanup" {
      # ...
      lifecycle {
        hook = "poststop"
      }
      # ...
      config {
        command = "consul"
        args    = ["services", "deregister", "-id=api-gateway"]
      }
    }
  }
Review the Consul intentions configuration
In a secure configuration, Consul uses intentions to specify allowed communications across services.
The repository provides a script, 04.intentions.consul.sh, that automates the creation of intentions.
The Consul intentions script cleans up any previous configurations and then creates intentions for the HashiCups services. Intentions are how Consul enforces routes of communication between services.
/shared/jobs/04.intentions.consul.sh
## ...
##-------------------------------------------------------------------------------
## Clean previous configurations
##-------------------------------------------------------------------------------
echo -e "${_COL}Clean previous configurations.${_NC}"
consul config delete -kind service-intentions -name database
consul config delete -kind service-intentions -name product-api
consul config delete -kind service-intentions -name payments-api
consul config delete -kind service-intentions -name public-api
consul config delete -kind service-intentions -name frontend
consul config delete -kind service-intentions -name nginx
## ...
### ----------------------------------------------------------------------------
### Configure Consul Intentions
### ----------------------------------------------------------------------------
## References:
## - https://developer.hashicorp.com/consul/docs/connect/config-entries/service-intentions
echo -e "${_COL}Create Consul intentions for Hashicups and API Gateway${_NC}"
tee ${_int_DB_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "database"
Sources = [
  {
    Name   = "product-api"
    Action = "allow"
  }
]
EOF
tee ${_int_PROD_API_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "product-api"
Sources = [
  {
    Name   = "public-api"
    Action = "allow"
  }
]
EOF
tee ${_int_PAY_API_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "payments-api"
Sources = [
  {
    Name   = "public-api"
    Action = "allow"
  }
]
EOF
tee ${_int_PUB_API_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "public-api"
Sources = [
  {
    Name   = "nginx"
    Action = "allow"
  }
]
EOF
tee ${_int_FE_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "frontend"
Sources = [
  {
    Name   = "nginx"
    Action = "allow"
  }
]
EOF
tee ${_int_NGINX_FILE} > /dev/null << EOF
Kind = "service-intentions"
Name = "nginx"
Sources = [
  {
    Name   = "api-gateway"
    Action = "allow"
  }
]
EOF
consul config write ${_int_DB_FILE}
consul config write ${_int_PROD_API_FILE}
consul config write ${_int_PAY_API_FILE}
consul config write ${_int_PUB_API_FILE}
consul config write ${_int_FE_FILE}
consul config write ${_int_NGINX_FILE}
Review the HashiCups jobspec
Open the 04.hashicups.nomad.hcl jobspec file and review the contents.
This version uses the same constraint as previous versions to run all of the HashiCups services on private client nodes. It integrates Consul service mesh with upstream configurations and configures services to use mTLS through the Envoy sidecar proxy. These configuration attributes are located in the connect block.
/shared/jobs/04.hashicups.nomad.hcl
job "hashicups" {
  # ...
  group "db" {
    # ...
    service {
      # ...
      connect {
        sidecar_service {}
      }
      # ...
    }
    # ...
  }
  # ...
  group "public-api" {
    # ...
    service {
      # ...
      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "product-api"
              local_bind_port = 9090
            }
            upstreams {
              destination_name = "payments-api"
              local_bind_port = 8080
            }
          }
        }
      }
      # ...
    }
    # ...
  }
  # ...
}
Deploy Consul API gateway
Set up and deploy the Consul API gateway before you submit the updated jobspec to Nomad. Set up and deploy the Consul API gateway before you submit the updated jobspec to Nomad.
Run the API setup script and jobspec
Set up the API gateway configurations in Consul.
$ ./04.api-gateway.config.sh
Configure environment.
Clean previous configurations.
Config entry deleted: http-route/hashicups-http-route
Config entry deleted: inline-certificate/api-gw-certicate
Config entry deleted: api-gateway/api-gateway
Error deleting namespace: Unexpected response code: 500 (rpc error: namespace not found)
Error deleting auth method "nomad-workloads": Unexpected response code: 404 (Cannot find auth method to delete)
## ...
Create Nomad namespace 'ingress'
Successfully applied namespace "ingress"!
Create Consul binding-rule 'Nomad API gateway'
## ...
Generate TLS certificate for 'hashicups.hashicorp.com'
Create 'api-gateway-certificate' inline-certificate
Config entry written: inline-certificate/api-gw-certificate
Create 'api-gateway' HTTP listener on port 8443
Config entry written: api-gateway/api-gateway
Create 'hashicups-http-route' HTTP route '/' > 'nginx'
Config entry written: http-route/hashicups-http-route
Submit the API gateway job to Nomad.
$ nomad job run 04.api-gateway.nomad.hcl
==> 2024-11-13T15:21:40+01:00: Monitoring evaluation "f25cae62"
    2024-11-13T15:21:40+01:00: Evaluation triggered by job "api-gateway"
    2024-11-13T15:21:40+01:00: Allocation "c1f36918" created: node "30b5f033", group "api-gateway"
    2024-11-13T15:21:42+01:00: Evaluation within deployment: "796699a7"
    2024-11-13T15:21:42+01:00: Evaluation status changed: "pending" -> "complete"
==> 2024-11-13T15:21:42+01:00: Evaluation "f25cae62" finished with status "complete"
==> 2024-11-13T15:21:42+01:00: Monitoring deployment "796699a7"
  ✓ Deployment "796699a7" successful
    2024-11-13T15:22:03+01:00
    ID          = 796699a7
    Job ID      = api-gateway
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully
    Deployed
    Task Group   Desired  Placed  Healthy  Unhealthy  Progress Deadline
    api-gateway  1        1       1        0          2024-11-13T14:32:00Z
Run the Consul intentions setup script
Set up the service intentions in Consul.
$ ./04.intentions.consul.sh
Configure environment.
Clean previous configurations.
Config entry deleted: service-intentions/database
Config entry deleted: service-intentions/product-api
Config entry deleted: service-intentions/payments-api
Config entry deleted: service-intentions/public-api
Config entry deleted: service-intentions/frontend
Config entry deleted: service-intentions/nginx
Create Consul intentions for Hashicups and API Gateway
Config entry written: service-intentions/database
Config entry written: service-intentions/product-api
Config entry written: service-intentions/payments-api
Config entry written: service-intentions/public-api
Config entry written: service-intentions/frontend
Config entry written: service-intentions/nginx
Verify API gateway deployment
Verify that the API gateway deployed successfully.
Use the nomad job command to retrieve information about the hashicups job.
$ nomad job allocs --namespace=ingress api-gateway
ID        Node ID   Task Group   Version  Desired  Status   Created    Modified
c1f36918  30b5f033  api-gateway  0        run      running  58m7s ago  57m46s ago
Use the consul catalog command to verify that the services are correctly registered inside Consul's catalog.
$ consul catalog services
api-gateway
consul
nomad
nomad-client
Deploy HashiCups with service mesh
After you deploy the Consul API gateway, deploy the new version of the HashiCups job.
Run the HashiCups jobspec
Submit the HashiCups job to Nomad.
$ nomad job run 04.hashicups.nomad.hcl
==> 2024-11-13T16:35:14+01:00: Monitoring evaluation "746b95fe"
    2024-11-13T16:35:14+01:00: Evaluation triggered by job "hashicups"
    2024-11-13T16:35:14+01:00: Allocation "04fffc73" created: node "b50f2376", group "product-api"
    2024-11-13T16:35:14+01:00: Allocation "23d2a286" created: node "b50f2376", group "public-api"
    2024-11-13T16:35:14+01:00: Allocation "444528dc" created: node "b50f2376", group "db"
    2024-11-13T16:35:14+01:00: Allocation "5af579d5" created: node "7fb20437", group "payments"
    2024-11-13T16:35:14+01:00: Allocation "8d4c039a" created: node "7fb20437", group "nginx"
    2024-11-13T16:35:14+01:00: Allocation "ffddd50e" created: node "b50f2376", group "frontend"
    2024-11-13T16:35:15+01:00: Evaluation within deployment: "5178e7b3"
    2024-11-13T16:35:15+01:00: Evaluation status changed: "pending" -> "complete"
==> 2024-11-13T16:35:15+01:00: Evaluation "746b95fe" finished with status "complete"
==> 2024-11-13T16:35:15+01:00: Monitoring deployment "5178e7b3"
  ✓ Deployment "5178e7b3" successful
    2024-11-13T16:36:03+01:00
    ID          = 5178e7b3
    Job ID      = hashicups
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully
    Deployed
    Task Group   Desired  Placed  Healthy  Unhealthy  Progress Deadline
    db           1        1       1        0          2024-11-13T15:45:49Z
    frontend     1        1       1        0          2024-11-13T15:45:54Z
    nginx        1        1       1        0          2024-11-13T15:45:34Z
    payments     1        1       1        0          2024-11-13T15:45:53Z
    product-api  1        1       1        0          2024-11-13T15:45:53Z
    public-api   1        1       1        0          2024-11-13T15:46:01Z
Verify the HashiCups deployment
Use the nomad job command to retrieve information about the hashicups job.
$ nomad job allocs hashicups
ID        Node ID   Task Group   Version  Desired  Status   Created    Modified
04fffc73  b50f2376  product-api  0        run      running  5m25s ago  4m45s ago
23d2a286  b50f2376  public-api   0        run      running  5m25s ago  4m37s ago
444528dc  b50f2376  db           0        run      running  5m25s ago  4m49s ago
5af579d5  7fb20437  payments     0        run      running  5m25s ago  4m44s ago
8d4c039a  7fb20437  nginx        0        run      running  5m25s ago  5m3s ago
ffddd50e  b50f2376  frontend     0        run      running  5m25s ago  4m44s ago
Use the consul catalog command to verify that the services are correctly registered inside Consul's catalog.
$ consul catalog services
api-gateway
consul
database
database-sidecar-proxy
frontend
frontend-sidecar-proxy
nginx
nginx-sidecar-proxy
nomad
nomad-client
payments-api
payments-api-sidecar-proxy
product-api
product-api-sidecar-proxy
public-api
public-api-sidecar-proxy
To view the Hashicups application, navigate to the public IP address of the API gateway.
In the following command, note the --namespace flag. The API gateway runs in a namespace that is separate from the application. You will need to trust the certificate in your browser.
$ nomad node status -verbose \
    $(nomad job allocs --namespace=ingress api-gateway | grep -i running | awk '{print $2}') | \
    grep -i public-ipv4 | awk -F "=" '{print $2}' | xargs | \
    awk '{print "https://"$1":8443"}'
Output from the above command.
https://3.15.17.40:8443
Visit the address in your web browser to view the application.
 

Before you proceed, interact with the HashiCups application and then observe the changes in the Nomad and Consul UIs.
Cleanup
Stop the HashiCups deployment when you are ready to move on.
$ nomad job stop -purge hashicups
==> 2024-11-04T17:26:49-05:00: Monitoring evaluation "8ff7d2ad"
    2024-11-04T17:26:49-05:00: Evaluation triggered by job "hashicups"
    2024-11-04T17:26:49-05:00: Evaluation status changed: "pending" -> "complete"
==> 2024-11-04T17:26:49-05:00: Evaluation "8ff7d2ad" finished with status "complete"
Warning
Do not stop the API gateway job. You will use it in the next tutorial.
Next steps
In this tutorial, you deployed the Consul API gateway to secure access to your application. Then you deployed a version of HashiCups that uses Consul service mesh to fully secure services running on private client nodes.
For more information about to Consul API gateway integration with Nomad, refer to Consul ACL with Nomad Workload Identities and Deploy a Consul API Gateway on Nomad.
In the next tutorial, you will deploy the Nomad Autoscaler and use it to scale up the frontend service of HashiCups.















