Installing a Nomad Cluster with Console on Cloud Servers

Introduction

In this tutorial, we will set up a HashiCorp Nomad cluster using the console to discover services and nodes. Using 3 server nodes and an arbitrary number of client nodes, this tutorial can be a good base for growing projects. We will also create a Snapshot on Hetzner Cloud for our clients, which will allow us to add more clients without having to manually configure them. This cluster will run on a private network between the servers and will support all of the default Nomad and console features, such as service discovery and volumes.

This tutorial partially follows the steps recommended in the official Consul and Nomad installation guides.

Prerequisites

• A Hetzner Cloud account
• Introduction to Linux and Terminal Commands
• Ability to connect to the server via ssh

This tutorial has been tested on Ubuntu 24.04 servers from Hetzner Cloud with Nomad 1.9.3 and Consul 1.20.1 versions.

Terms and symbols

Commands:

local$ <دستور> # این دستور باید در ماشین محلی شما اجرا شود

server$ <دستور> # این دستور باید در سرور به‌عنوان کاربر root اجرا شود

Step 1 – Create a base image

The following resources will be used in this step:

• 1 Hetzner Cloud Server Model CX22

We will start by setting up a Consul/Nomad server on a new CX22 server in Hetzner Cloud. The resulting snapshot will be used as a base image for all cluster servers and clients in the following steps.

Step 1.1 – Installing Consul

To install Consul, run the following commands:

server$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
server$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list
server$ apt update && apt install consul

To add autocomplete functionality in Consul (optional):

server$ consul -autocomplete-install

Step 1.2 – Install the Nomad binary

To install Nomad, run the following commands:

server$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
server$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list
server$ apt update && apt install nomad

To add autocomplete functionality in Nomad (optional):

server$ nomad -autocomplete-install

Step 1.3 – Preparing systemd services

Consul and Nomad should start automatically upon boot. To enable this feature, create a systemd service for each of them.

server$ chown consul:consul dc1-server-consul*
server$ chown consul:consul dc1-client-consul*
server$ chown -R consul:consul /opt/consul
server$ chown -R nomad:nomad /opt/nomad
server$ mkdir -p /opt/alloc_mounts && chown -R nomad:nomad /opt/alloc_mounts

Then create the configuration file /etc/systemd/system/consul.service with the following content:

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl
[Service]
EnvironmentFile=-/etc/consul.d/consul.env
User=consul
Group=consul
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

And create the configuration file /etc/systemd/system/nomad.service with the following content:

[Unit]
Description=Nomad
Documentation=https://www.nomadproject.io/docs/
Wants=network-online.target
After=network-online.target
[Service]
User=nomad
Group=nomad
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/bin/nomad agent -config /etc/nomad.d
KillMode=process
KillSignal=SIGINT
LimitNOFILE=65536
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
OOMScoreAdjust=-1000
TasksMax=infinity
[Install]
WantedBy=multi-user.target

Do not enable these services at this stage, as the installation is not yet complete.

Step 1.5 – Create a Base Snapshot

Finally, stop the server in the Hetzner Cloud console and create a Snapshot. This Snapshot will be used as the basis for setting up the cluster servers and clients.

Step 2 – Setting up cluster servers

In this step, you will create 3 cluster servers from the base image. These servers will be the bases of your cluster and will automatically elect a cluster leader.

We will use the following resources:

• 1 Hetzner Cloud Network
• 3 Hetzner Cloud Servers Model CX22

In the Hetzner Cloud Console, create 3 CX22 servers from the Snapshot created in Step 1 and connect them to a shared cloud network. This tutorial will use the 10.0.0.0/8 network, but smaller networks will work as well.

Step 2.1 – Generate a symmetric encryption key

First, create a symmetric encryption key that will be shared between all servers. Store this key in a safe place, as we will need it in the next steps.

server$ consul keygen

Step 2.2 – Distribute certificates

Now we can copy the correct certificates from step 1 to the Consul configuration directory. Run the following command on all servers:

server$ cp consul-agent-ca.pem /etc/consul.d/

The 3 server certificates created in step 1 need to be distributed so that each server gets its own unique certificate with the corresponding key. This tutorial provides the following commands for the different servers:

[10.0.0.2] server$ cp -a dc1-server-consul-0* /etc/consul.d/
[10.0.0.3] server$ cp -a dc1-server-consul-1* /etc/consul.d/
[10.0.0.4] server$ cp -a dc1-server-consul-2* /etc/consul.d/

Step 2.3 – Edit Consul Configuration

On all servers, edit the configuration file /etc/consul.d/consul.hcl and add the following contents:

datacenter = "dc1"
data_dir = "/opt/consul"
encrypt = "your-symmetric-encryption-key"
tls {
defaults {
ca_file = "/etc/consul.d/consul-agent-ca.pem"
cert_file = "/etc/consul.d/dc1-server-consul-0.pem"
key_file = "/etc/consul.d/dc1-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
},
internal_rpc {
verify_server_hostname = true
}
}
retry_join = ["10.0.0.2"]
bind_addr = "{{ GetPrivateInterfaces | include \"network\" \"10.0.0.0/8\" | attr \"address\" }}"
acl = {
enabled = true
default_policy = "allow"
enable_token_persistence = true
}
performance {
raft_multiplier = 1
}

Step 2.4 – Setting up Consul and Nomad

Now that the configurations are done correctly, you can start the Consul and Nomad services.

server$ systemctl enable consul
server$ systemctl start consul
server$ systemctl enable nomad
server$ systemctl start nomad

To check the status of services, you can use the following commands:

server$ systemctl status consul
server$ systemctl status nomad

If everything works correctly, the Consul and Nomad services should be running and the cluster should be properly set up.

Step 2.5 – Connecting other nodes to the cluster

Now that the main cluster servers are up and running, it's time to connect the other nodes. To connect client nodes to the cluster, simply apply the same settings to the Consul and Nomad configurations for the new nodes.

On each of the client nodes, create the same configuration files as the main server and then start the services:

client$ cp consul-agent-ca.pem /etc/consul.d
/client$ cp dc1-client-consul-* /etc/consul.d
/client$ systemctl enable consul client$ systemctl start consul
client$ systemctl enable nomad client$ systemctl start nomad

Step 3 – Verify Cluster and Configuration

After starting the servers and clients, you can check the status of the cluster using the following commands:

client$ consul members

This command should display a list of nodes in the cluster. You can also check the status and settings of Nomad using the following command:

client$ nomad status

Step 4 – Cluster Management

Now that your Consul and Nomad cluster is properly set up, you can manage it and take advantage of its various features.

To view the cluster status and run management requests, use the following commands:

• consul info # to get general information about the Consul cluster
• nomad job status # to view Nomad job status
• nomad alloc status # to view the status of allocations in the Nomad cluster

Using Nomad Cluster

Nomad allows you to perform various tasks such as managing jobs, deploying pods, and coordinating with Consul to discover services.

Using Consul to discover services

Consul provides automatic service discovery. Using the Consul web interface (available by default on port 8500), you can view available services and configure them if necessary.

Result

In this tutorial, you created a cluster of HashiCorp Nomad and Consul servers in a private cloud network using Hetzner Cloud. This cluster can be used to manage various large-scale workloads. This tutorial also covered how to launch new nodes via Snapshot and configure the servers in detail.