Skip to content

etcd distributed cluster deployment

etcd is a distributed, reliable key-value store used for the most critical data in a distributed system^[400-devops-06-kubernetes-k8s-paas-02-k8s.md]. It functions as a high-availability service discovery and storage directory, based on the Raft consensus algorithm^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

Architecture Overview

In the context of the deployment architecture described, the etcd cluster is deployed on nodes hdss7-12, hdss7-21, and hdss7-22^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

Prerequisites

User Creation

A dedicated system user should be created on all etcd nodes to run the service securely without a login shell^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

useradd -s /sbin/nologin -M etcd

Certificate Generation

etcd requires PKI infrastructure for secure communication. The cfssl toolset is used to generate a Certificate Authority (CA) and specific peer certificates^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

  1. CA Configuration: Create ca-config.json defining profiles for server, client, and peer usage^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].
  2. Peer Certificate CSR: Create etcd-peer-csr.json including the Common Name (CN) "k8s-etcd" and the IP addresses of all cluster nodes in the hosts field^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].
  3. Generate Certificates: Use cfssl gencert with the peer profile to create etcd-peer.pem and etcd-peer-key.pem^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

Installation

Software Deployment

Download and extract the etcd binaries (e.g., etcd-v3.1.20-linux-amd64.tar.gz) to /opt^[400-devops-06-kubernetes-k8s-paas-02-k8s.md]. Create a symbolic link for easier management and upgrades^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

tar xf etcd-v3.1.20-linux-amd64.tar.gz -C /opt
ln -s /opt/etcd-v3.1.20/ /opt/etcd

Directory Structure

Create necessary directories for data, logs, and certificates on all nodes^[400-devops-06-kubernetes-k8s-paas-02-k8s.md]:

mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server

Distribute the generated ca.pem, etcd-peer.pem, and etcd-peer-key.pem files to /opt/etcd/certs/ on each node^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

Set proper ownership for the etcd user^[400-devops-06-kubernetes-k8s-paas-02-k8s.md]:

chown -R etcd.etcd /opt/etcd-v3.1.20/
chown -R etcd.etcd /data/etcd/
chown -R etcd.etcd /data/logs/etcd-server/

Configuration

Startup Script

Create a startup script /opt/etcd/etcd-server-startup.sh on each node. The script must be modified per node to match specific IP addresses and hostnames^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

Key parameters include: * --name: Unique member name (e.g., etcd-server-7-12). * --data-dir: Data storage location (/data/etcd/etcd-server). * --listen-peer-urls: URL for peer communication (e.g., https://10.4.7.12:2380). * --listen-client-urls: URL for client traffic (e.g., https://10.4.7.12:2379,http://127.0.0.1:2379). * --advertise-client-urls: Advertised client URLs. * --initial-advertise-peer-urls: Advertised peer URLs. * --initial-cluster: List of all initial members in the cluster (e.g., etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380...). * --ca-file, --cert-file, --key-file: Paths to TLS certificates.

Ensure the script is executable^[400-devops-06-kubernetes-k8s-paas-02-k8s.md]:

chmod +x /opt/etcd/etcd-server-startup.sh

Process Management (Supervisor)

Use supervisor to manage the etcd service, ensuring it restarts automatically on failure^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

  1. Install supervisor: yum install supervisor -y^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].
  2. Create a configuration file /etc/supervisord.d/etcd-server.ini specific to the node hostname^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].
  3. Update supervisor: supervisorctl update^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].
  4. Check status: supervisorctl status^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

Verification

Verify the cluster health and membership status from any node using the etcdctl command-line tool^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

./etcdctl cluster-health
./etcdctl member list

Successful deployment will show the cluster is healthy and lists all members. The node that initializes the cluster first typically becomes the leader^[400-devops-06-kubernetes-k8s-paas-02-k8s.md].

  • [[TLS certificates]]
  • [[Raft consensus algorithm]]
  • [[Kubernetes architecture]]

Sources

  • 400-devops-06-kubernetes-k8s-paas-02-k8s.md