High Availability Article Series #3 - NGINX High Availability Implementation

This article was last updated on: July 24, 2024 am

  1. Part 1 of a highly available series - Overview - Dongfeng Weiming Technology Blog (e-whisper.com)
  2. High Availability Series Article 2 - Traditional Layered Architecture Technology Solution - Dongfeng Weiming Technology Blog (e-whisper.com)

Four NGINX highly available implementations

For a highly available implementation, the main steps are outlined below:

  1. Installation and basic configuration of NGINX
  2. Load balancing layer is highly available: NGINX + Keepalive configuration
  3. Application service layer highly available: NGINX -> application service layer forwarding configuration

Detailed configuration list of system hardware and software

According to the design of the high-availability architecture of the manufacturing industry and the business requirements, the deployment model is recommended to be configured as follows:

  • Server balancing (i.e. NGINX + Keepalived): 2, operating system Linux. The configuration recommendations are as follows:
Name Specifications Notes
CPU 2 core
Memory 4 GB
Hard disk 50 GB
Operating System SUSE12 64-bit and patches
Network card At least 1 network card that supports VRRP technology
  • Software running configuration environment:
Software Specifications
NGINX 1.16.1
Keepalived 2.0.10

4.1 NGINX Installation and Configuration

4.1.1 Partitions and directories

It is recommended to be divided into at least the following 3 zones:

Partitions and directories Size Notes
Primary partition ( /) Default The nginx program and configuration files are located in this partition (/etc/nginx)
Log partitions (/var/log/nginx) 10G-20G
Program directory partition (/usr/share/nginx/html) 10G Optionally, this directory is required for nginx to be used as a web server.

4.1.2 Programs and dependent versions

Program Components Installation package name Version md5
nginx nginx-1.16.1-1.sles12.ngx.x86_64.rpm 1.16.1 396A359F26DD0100CD59545BAFFAFE85

4.1.3 NGINX Procedure Specification

  • nginx program directory: /etc/nginx
  • Executor path: /usr/sbin/nginx
  • Main configuration file path:/etc/nginx/conf/nginx.conf
  • Each application system forwards the configuration file directory:/etc/nginx/conf.d/
  • Log directory:/var/log/nginx
  • Static file directories for each application:/usr/share/nginx/html

4.1.4 System-level configuration optimization

❕ Note:

need root User execution.

  1. Install the components: logrotate
  2. To modify the number of connections:
1
2
3
4
vi /etc/security/limits.conf
# vi编辑
*               soft    nofile          65535
*               hard    nofile          65535
  1. To modify the system kernel configuration:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
vi /etc/sysctl.conf
# vi编辑

# NGINX Tuning Performance
fs.file-max = 65535

vm.zone_reclaim_mode = 0

net.core.somaxconn = 2048

net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_mtu_probing = 1

# 生效
sysctl -p

4.1.5 Configuring NGINX repo

❕ Note:

The commands in this section can be operated on the company’s internal repo warehouse machine on a case-by-case basis.

Other machines only need to configure the internal repo address.

Type the following zypper command to add the zypper repository for SLES

1
$ sudo zypper addrepo -G -t yum -c 'http://nginx.org/packages/sles/12' nginx

Add Pre-Built SLES Packages Repo for Stable Nginx version

Next, you must verify the digital signature to maintain the integrity and origin of the downloaded package. Use the wget command to get the nginx signing key:

1
$ wget http://nginx.org/keys/nginx_signing.key

Sample output:

1
2
3
4
5
6
7
8
9
10
--2020-01-09 23:48:48--  http://nginx.org/keys/nginx_signing.key
Resolving nginx.org (nginx.org)... 206.251.255.63, 95.211.80.227, 2001:1af8:4060:a004:21::e3, ...
Connecting to nginx.org (nginx.org)|206.251.255.63|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1561 (1.5K) [text/plain]
Saving to: ‘nginx_signing.key’
 
100%[==================================================>] 1,561       --.-K/s   in 0s      
 
2020-01-09 23:48:49 (117 MB/s) - ‘nginx_signing.key’ saved [1561/1561]

Use the rpm command to import the key into rpm:

1
$ sudo rpm --import nginx_signing.key

4.1.6 Install NGINX on SUSE

Type the following zypper Command:

1
$ sudo zypper install nginx=1.16.1

How To Install Nginx on SUSE Linux using zypper command

4.1.7 Optional: Configure the firewall

❕ Note:

If there are other dedicated firewalls in the data center traffic ingress, you can disable the firewall on the nginx server without this step.

Start by creating a configuration file for the Nginx specific service, opening port 80 using a text editor such as the vi command:

1
$ sudo vi /etc/sysconfig/SuSEfirewall2.d/services/nginx

Add the following configuration:

1
2
3
4
5
## Name: Nginx web server
## Description: Open ports for Nginx Server
 
# space separated list of allowed TCP ports
TCP="http"

(If you don’t need HTTPS support, you only need to allow traffic on TCP port number 80.) Save and exit the VI/VIM text editor. Now, just run the following command to open port 80:

1
$ sudo yast firewall

Must be used TAB and arrow keys to jump in YaST. In YaST, jump to Allowed Services and press Enter Key:

SLES Firewall Config for Nginx server

use TAB Jump to "Allowed Servicesand press the Down Arrow key to select Nginx web serverand press Enter. Must be pressed Alt-A to add the Nginx server to the firewall:

Adjust the SUSE Linux Firewall for Nginx

press Alt-N and Alt-F Save and complete the firewall settings on SLES. When the shell prompt is returned, list all iptables rules on SLE:

1
$ sudo iptables -S

Sample output:

1
2
-A input_ext -p tcp -m limit --limit 3/min -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -j LOG --log-prefix "SFW2-INext-ACC-TCP " --log-tcp-options --log-ip-options
-A input_ext -p tcp -m tcp --dport 80 -j ACCEPT

Use a combination of the sudo command and the grep command to determine whether port 80 is open:

1
sudo sh -c 'iptables -L -n -v | grep :80'

4.1.8 Start NGINX Server

Type the following systemctl command to enable nginx at system startup: (boot-on)

1
$ sudo systemctl enable nginx

Start the NGINX web server:

1
$ sudo systemctl start nginx

Verify:

1
$ systemctl status nginx

Turn On Check Nginx Web Server Service

To determine if port 80 is listening, run the following netstat command or ss command:

1
2
$ sudo netstat -tulpn | grep :80
$ sudo ss -tulpn | grep :80

Is port 80 open on SLES

4.1.9 NGINX Basic Operations

Stop it:

1
$ sudo systemctl stop nginx

Initiate:

1
$ sudo systemctl start nginx

Restart the service:

1
$ sudo systemctl restart nginx

Reload Ngnix after changing the configuration:

1
$ sudo systemctl reload nginx

ℹ Recommendation:

Use after configuration updates reload to reload nginx.

Visit the nginx page:

Assume that the NGINX IP is: 192.168.0.1. Use a browser or curl Visit:

http://serve_IP
http://your-domain
http://192.168.0.1

img

1
$ curl -I 192.168.122.43

Sample output:

HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Sat, 03 Feb 2020 19:18:53 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 17 Oct 2019 13:30:50 GMT
Connection: keep-alive
ETag: "59e6060a-264"
Accept-Ranges: bytes

4.1.10 Find information about Nginx configuration files on SLES

Now Nginx is up and running. Next, you can customize the configuration.

Server configuration file for SLES:

  • /etc/nginx/: nginx default configuration directory
  • /etc/nginx/nginx.conf: nginx master profile
  • /etc/nginx/conf.d/default.conf: The configuration of the default virtual host

To modify the Configuration Master:

1
$ sudo vi /etc/nginx/nginx.conf

4.1.11 NGINX logs

  • /var/log/nginx/access.log: Access logs
  • /var/log/nginx/error.log: Error log

4.1.12 nginx.conf

The main configuration file is detailed below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
#### 全局块 开始 #####
# 配置允许运行Nginx服务器的用户和用户组
#user  nginx nginx;
# 配置允许Nginx进程生成的worker process数
worker_processes  4;
#worker_cpu_affinity 0001 0010 0100 1000;

# 配置Nginx服务器运行时的错误日志文件存放路径和名称
#error_log  logs/error.log;
error_log  logs/error.log  info;

# 配置Nginx服务器运行时的pid文件存放路径和名称
pid        logs/nginx.pid;
#### 全局块 结束 #####


#### events块 开始 ####
events {
    # 配置事件驱动模型
    use epoll;
    accept_mutex off;
    multi_accept off;
    worker_connections  65535;	
}
#### events块 结束 ####


#### http块 开始 ####
http {
    # 定义MIME-Type
    include       mime.types;
    default_type  application/octet-stream;
	
    # 配置请求处理日志的格式
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent $request_time $upstream_response_time "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" "$host"';

    access_log  logs/access.log  main;

    # 配置允许使用sendfile方式传输
    sendfile        on;
    #tcp_nopush     on;
	
    # 配置连接超时时间
    #keepalive_timeout  0;
    keepalive_timeout  65;
	
    # nginx允许的客户端请求头部的缓冲区大小
    client_header_buffer_size 4k;

    # gzip conf
    gzip  on;
    gzip_min_length 1024;
    gzip_buffers 32 4k;
    gzip_http_version 1.1;
    gzip_comp_level 6;
    gzip_types text/plain application/xml image/x-icon image/svg+xml image/png text/css image/jpeg image/gif application/x-javascript application/javascript application/json;
    gzip_vary on;
    gzip_disable "MSIE [1-6]\.";
	
    # security
    port_in_redirect off;
    server_tokens off;
	
    # proxy buffer 
    proxy_buffers 8 4k;
    proxy_buffer_size 4k;
    proxy_temp_file_write_size 4k;
    proxy_temp_path proxy_temp;
	
    # proxy cache
    # proxy_cache_path cache/ keys_zone=cache_all:10m;

	
    #### server块 开始 ####
    ## 配置虚拟主机localhost
    server {
        listen       80 reuseport;
        server_name  localhost;

        #charset koi8-r;

        access_log  logs/host.access.log  main;

        location / {
           root   html;
            index  index.html index.htm;
        }

        error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

    }
	#### server 块 结束 ####
	

    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}
	
    # virtual hosts
    include conf.d/default.conf;

}
#### http块 结束 ####

✔ Recommendation:

In order to ensure that the main configuration file is clean. It is recommended to pass include conf.d/default.conf; Other virtual hosts configurations are introduced in a way like this.

4.1.13 Log dumps

  1. sudo vi /etc/logrotate.d/nginx

  2. Edits:

    /var/log/nginx/*.log {
    daily
    rotate 90  # 保留90天, 按需调整
    create
    dateext
    #compress  # 是否启用压缩, 按需调整
    #minsize 1M
    #create 0644 nginx nginx  # nginx日志所属用户和组, 按需调整
    # copytruncate   用于还在打开中的日志文件，把当前日志Backup并截断；是先拷贝再清空的方式，拷贝和清空之间有一个时间差，可能会丢失部分日志数据。
    # delaycompress 和compress 一起使用时，转储的日志文件到下一次转储时才压缩
    missingok
    ifempty # default
    nomail
    #noolddir # default
    sharedscripts   # 运行postrotate脚本，作用是在所有日志都轮转后统一执行一次脚本。如果没有配置这个，那么每个日志轮转后都会执行一次脚本
    postrotate  # 在logrotate转储之后需要执行的指令，例如重新启动 (kill -HUP) 某个服务！必须独立成行
        if [ -f /var/log/nginx/nginx.pid ]; then
            kill -USR1 `cat /var/log/nginx/nginx.pid`
        fi
    endscript
}

  3. Force to run once to test:logrotate -f -v /etc/logrotate.d/nginx(The corresponding directory can only have user with w permission, otherwise an error will be reported)

  4. Just configure it, logrotate will automatically read it/etc/logrotate.dand automate the configuration.

📓 Remark:

cron.daily will be 3:22+(5,45) This time period is executed

4.2 NGINX + Keep Lived configuration

ℹ Assumption:

Assume that the relevant IP addresses are as follows:

  1. VIP: 192.168.0.100
  2. NGINX - Main IP: 192.168.0.1
  3. NGINX - From IP: 192.168.0.2
  4. NGINX - The NIC corresponding to the master-slave IP address is eth0

4.2.1 Keepalived installation configuration

1
$ sudo zypper install keepalived=2.0.10

4.2.2 Keepalived configuration

Keep alive starts reading the configuration file path by default /etc/keepalived/keepalived.conf to add a configuration file:

1
sudo vi /etc/keepalived/keepalived.conf

NGINX - Primary The machine configuration is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
vrrp_script chk_nginx {
    script "/etc/keepalived/nginx_pid.sh" # 检测nginx状态的脚本路径
    interval 2
    weight -20 
    fall 3
}
vrrp_instance VI_1 {
    state MASTER # 从机为 BACKUP
    interface eth0  # 具体的网卡接口看情况进行填写
    virtual_router_id 51 
    priority 110 # 备机权值为100
    advert_int 1
    authentication {
    auth_type PASS
    auth_pass 123456
    }
    track_script {
        chk_nginx
    }
    virtual_ipaddress {
    192.168.0.100/24 brd 192.168.0.255 dev eth0 label eth0:vip  # 根据实际情况进行修改
    }
}

The configuration instructions are as follows:

  1. weight -20: keepalived configuration priority 110 is the initial weight; The primary one is initially set to 110, the standby one is set to 100, and if the detection fails, the weight is -20. Master-slave switching.

  2. virtual_router_id 51 The ID of the self-defined virtual route.vrrp_instance VI_1 block virtual_router_id The value of 51 for the directive is an example value; Change it as needed to make it unique in your environment.

  3. Add to

NGINX - from The machine configuration is as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
vrrp_script chk_nginx {
    script "/etc/keepalived/nginx_pid.sh" # 检测nginx状态的脚本路径
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state BACKUP # 辅机为 BACKUP
    interface eth0  # 具体的网卡接口看情况进行填写
    virtual_router_id 51
    priority 100 # 备机权值为100
    advert_int 1
    authentication {
    auth_type PASS
    auth_pass 123456
    }
    track_script {
        chk_nginx
    }
    virtual_ipaddress {
    192.168.0.100/24 brd 192.168.0.255 dev eth0 label eth0:vip  # 根据实际情况进行修改
    }
}

4.2.3 Configure the detection script

❕ Notes:

use root user

Configure the detection script: /etc/keepalived/nginx_pid.sh

1
2
3
4
5
6
7
8
9
#!/bin/bash
A=`ps -C nginx --no-header|wc -l`
if [ $A -eq 0 ]
then
   systemctl start nginx
   exit 1
else 
   exit 0
fi

Grant execute permissions: chmod 750 /etc/keepalived/nginx_pid.sh

4.2.4 Start Keepalived

Type the following systemctl command to enable keepalive at system startup: (boot-on)

1
$ sudo systemctl enable keepalived

Start the NGINX web server:

1
$ sudo systemctl start keepalived

Verify:

1
$ systemctl status keepalived

4.2.4 Display node status

To see which node is currently the primary node for a given VIP, run the command on the interface that defines the VRRP instance ip addr show(In the following command,eth0nodesuse12-1and interfaces on suse12-2）：

suse12-1 # ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
     UP qlen 1000
    link/ether 52:54:00:33:a5:a5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global dynamic eth0
       valid_lft 3071sec preferred_lft 3071sec
    inet 192.168.0.100/32 scope global eth0
       valid_lft forever preferred_lft forever

suse12-2 # ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
     UP qlen 1000
    link/ether 52:54:00:33:a5:87 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.2/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever

In this output, the second line inet denote suse12-1 is the master node to which a defined VIP has been assigned (192.168.0.100）。inet Other lines in the output show the actual IP address of the master node (192.168.0.1) and the IP address of the standby (or slave) node (192.168.0.2）。

So far, nginx + keepalivedThe load balancing layer is highly availableThe configuration is complete.

4.3 NGINX -> Application Service Layer forwarding configuration

ℹ Assumption:

Suppose the application system is related to the following:

  1. The request and business protocol is HTTP protocol;
  2. The IP address of application node 1 is: 172.30.0.1
  3. The IP address of application node 2 is: 172.30.0.2
  4. The listening port of the service system is: 8080

4.3.1 Modify the configuration file

Take the high-availability configuration of the mes system as an example (the protocol is HTTP protocol), it is added /etc/nginx/conf.d/mes.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
# mes access

upstream mes{
   # ip_hash;  # 源地址会话保持, 按需开启
   server 172.30.0.1:8080;
   server 172.30.0.2:8080;
}

server {
    listen       80;
    #server_name  mes.example.com 192.168.0.100;  # server_name为对应的访问域名, 按需开启

    #set max uploading file size to 20m
    client_max_body_size 20m;

    #charset koi8-r;
	
    # keepalive
    # 设置代理的HTTP协议版本（默认是1.0版本）
    proxy_http_version 1.1;    # 按需调整
    # 允许重新定义或追加字段到传递给代理服务器的请求头信息（默认是close）
    proxy_set_header Connection "";
	
    proxy_set_header HOST $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
	
    access_log  logs/mes_access.log main buffer=16k flush=5m;
    error_log logs/mes_error.log;
    location / {
        #root   html;
        #index  index.html index.htm;
       	proxy_pass http://mes;
        #proxy_redirect off;
        # proxy_cache cache_all;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    #error_page   500 502 503 504  /50x.html;
    #location = /50x.html {
    #    root   html;
    #}
}

4.3.2 Main configuration file include

Add the following content to the HTTP block

1
include conf.d/mes.conf;

4.3.2 Reload nginx

Reload Ngnix after changing the configuration for the configuration to take effect:

sudo systemctl reload nginx

4.3.3 Verification

Access through the VIP, view the corresponding application system logs, and observe whether both nodes receive requests.

5. Change control process

❗ Strictly follow the change control specifications of manufacturing companies. Conduct relevant change control and approval processes.

A new application needs to be launched or changed, which usually involves the following steps:

Changes that can be implemented at any time:

  1. The system project team must provide at least the port, path, and backend address (domain name is optional);
  2. at /etc/nginx/conf.d Add a configuration file;
  3. at /etc/nginx/nginx.conf Add one includeto point to the newly added configuration file;

❗ Changes that need to be made in the change window:

  1. Reload (sudo systemctl reload nginx) configuration.
  2. Inform the user that the IP of the server accessed is adjusted from the original application server IP to the VIP of NGINX.
  3. Modify the client and browser configuration, and adjust the server IP of the original application server to the VIP of NGINX.
  4. Verify that the change was successful.

5.1 Rollback Steps

Fallback is simpler, the steps are as follows:

  1. Notify the user that the accessed server IP is adjusted to: original application server IP.
  2. Modify the client and browser configuration, adjust the access server IP to: original application server IP.
  3. The fallback is complete
  4. Verify that the fallback was successful.

Reference documents

Reference files
High Availability Support for NGINX
CloudComputing
#Nginx #Architecture #HA #Keepalived
High Availability Article Series #3 - NGINX High Availability Implementation
https://e-whisper.com/posts/32355/
Author
east4ming
Posted on
September 28, 2021
Licensed under