Consul+Registrator+Nginx方案的部署

详细介绍Consul+Registrator+Nginx方案的部署细节。

Consul集群

Consul 集群架构

consul集群

图中Consul集群有两个数据中心,DATACENTER1 和DATACENTER2 。DATACENTER1中,有3个Consul server(一个leader 和2个follewer),3个Consul Client。实际上Consul Server也可以充当Client,比如DATACENTER2只有Server。Server具备Client所有的功能,此外还会把服务的信息持久化下来。服务一般是通过Consul Client注册到Consul集群中,但是也可以直接通过Server注册。

集群节点

IP 角色 节点名称
192.168.0.223 Consul Server leader consul_server_master
192.168.0.226 Consul Server follower consul_server_slave
192.168.0.245 Consul Server follower consul_server_slave2

如果机器是阿里云ECS ,注意三台机器安全组配置打开端口:

tcp 端口 8300 8301 8302 8400 8500 8600

udp端口 8301 8302

通过Docker部署Consul

每个节点上的consul service 的 name 都叫 consul_server。

业务应用的 docker container 通过 --link consul_server:consul 启动 ,就可以在应用内访以 host : consul 来访问 consul API。

阿里云的docker compose 不支持从文件中读取变量信息,无法一次性部署3个节点,只能一个个部署。

对于192.168.0.223节点的docker-compose.yml :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
version: '2'
services:
consul_server:
image: consul:latest
hostname: consul_server
ports:
- "8300:8300"
- "8301:8301"
- "8301:8301/udp"
- "8302:8302"
- "8302:8302/udp"
- "8400:8400"
- "8500:8500"
- "8600:8600"
volumes:
- "/consul/data:/consul/data:rw"
command: consul agent -server -bootstrap-expect 1 -advertise 192.168.0.223 -data-dir /consul/data -node consul_server_master -client 0.0.0.0 -ui
networks:
default:
external:
name: multi-host-network

consul agent -server 是以server模式启动consul。

-bootstrap-expect 1 consul集群最少1个实例即可运行。如果配置为3,需要3个consul server节点都启动,集群才能正常工作。

multi-host-network 是docker集群的网络(该网络实际上由阿里云容器服务默认创建)。也可以自行创建网络。

可以启动该consul节点。

对于 192.168.0.226 节点的docker-compose.yml :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
version: '2' 
services:
consul_server:
image: consul:latest
hostname: consul_server
ports:
- "8300:8300"
- "8301:8301"
- "8301:8301/udp"
- "8302:8302"
- "8302:8302/udp"
- "8400:8400"
- "8500:8500"
- "8600:8600"
volumes:
- "/consul/data:/consul/data:rw"
command: consul agent -server -retry-join=192.168.0.223 -advertise 192.168.0.226 -data-dir /consul/data -node consul_server_slave -client 0.0.0.0
networks:
default:
external:
name: multi-host-network

-retry-join=192.168.0.223 表明该节点启动后(或者异常退出重启后)尝试加入192.168.0.223的集群中。

同样的对于 192.168.0.245 节点的docker-compose.yml :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
version: '2'
services:
consul_server:
image: consul:latest
hostname: consul_server
ports:
- "8300:8300"
- "8301:8301"
- "8301:8301/udp"
- "8302:8302"
- "8302:8302/udp"
- "8400:8400"
- "8500:8500"
- "8600:8600"
volumes:
- "/consul/data:/consul/data:rw"
command: consul agent -server -retry-join=192.168.0.223 -advertise 192.168.0.245 -data-dir /consul/data -node consul_server_slave2 -client 0.0.0.0
networks:
default:
external:
name: multi-host-network

只是-node consul_server_slave2 不同。

把两个follower启动起来。

通过Docker部署LB+Registrator

接下来部署 consul-template+Ngnix+registrator

并没有跟consul集群的docker compose 文件合并成一个文件, 因为consul集群一旦启动,基本上不需要重新部署。而consul-template+Ngnix`可能后续会修改。

对于192.168.0.223节点的docker-compose.yml :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
version: '2'
services:
load_balancer:
image: registry.cn-hangzhou.aliyuncs.com/zongwu233/nginx-consul-template:v1.1
hostname: lb
external_links:
- consul_server:consul
ports:
- "80:80"
- "8788:8788"
- "9000:9000"
- "9001:9001"
registrator:
image: gliderlabs/registrator:latest
hostname: registrator
external_links:
- consul_server:consul
volumes:
- "/var/run/docker.sock:/tmp/docker.sock"
command: -ip 192.168.0.223 consul://192.168.0.223:8500
networks:
default:
external:
name: multi-host-network

nginx-consul-template 镜像带有consul-template 和nginx两个服务。详细介绍见https://github.com/liberalman/nginx-consul-template

该镜像中的nginx.conf.ctmpl文件是consul-template模版,会在启动后生成合适的nginx.conf并调用nginx的reload命令加载该文件。实际上,我们定制了nginx-consul-template docker镜像,后面再介绍。

对于load_balancer通过external_links 链接 同一台机器上的consul_server

对于registrator同样通过external_links链接到consul_server

对于192.168.0.226节点的docker-compose.yml :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
version: '2' 
services:
load_balancer:
image: registry.cn-hangzhou.aliyuncs.com/zongwu233/nginx-consul-template:v1.1
hostname: lb
external_links:
- consul_server:consul
ports:
- "80:80"
- "8788:8788"
- "9000:9000"
- "9001:9001"
registrator:
image: gliderlabs/registrator:latest
hostname: registrator
external_links:
- consul_server:consul
volumes:
- "/var/run/docker.sock:/tmp/docker.sock"
command: -ip 192.168.0.226 consul://192.168.0.226:8500
networks:
default:
external:
name: multi-host-network

对于192.168.0.245节点的docker-compose.yml :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
version: '2'
services:
load_balancer:
image: registry.cn-hangzhou.aliyuncs.com/zongwu233/nginx-consul-template:v1.1
hostname: lb
external_links:
- consul_server:consul
ports:
- "80:80"
- "8788:8788"
- "9000:9000"
- "9001:9001"
registrator:
image: gliderlabs/registrator:latest
hostname: registrator
external_links:
- consul_server:consul
volumes:
- "/var/run/docker.sock:/tmp/docker.sock"
command: -ip 192.168.0.245 consul://192.168.0.245:8500
networks:
default:
external:
name: multi-host-network

配置完成之后分别启动。

这样,整个框架部署完成,通过 http://192.168.0.223:8500/ui/dc1/services 可以访问consul 的UI 界面。

Test应用

部署一个testHello 应用来测试。先构建一个简单的SpringBoot应用,端口是9123。并实现简单的/health 接口,直接返回200,供consul 调用。

docker-compose文件内容:

1
2
3
4
5
6
7
8
9
10
11
12
version: '2'
services:
test_hello:
image: registry.cn-hangzhou.aliyuncs.com/zongwu233/testhello:latest
environment:
- SERVICE_9123_NAME=my-web-server
- SERVICE_9123_TAGS=test-service-01
- SERVICE_9123_CHECK_INTERVAL=10s
- SERVICE_9123_CHECK_TIMEOUT=2s
- SERVICE_9123_CHECK_HTTP=/health
ports:
- "9123:9123"

服务注册到consul,需要配置特定的environment信息。格式是SERVICE_PORT_XXX

访问 http://192.168.0.223:8500/ui/dc1/services 即可看到 test_hello的相关信息。

外部还不能直接访问testHello服务,该服务虽然注册到了consul,但是Ngnix并没有设置相应的代理。

调整nginx-consul-templatenginx.conf.ctmpl文件添加:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

upstream my-web-server{
{{range service "my-web-server"}}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
{{else}}server 127.0.0.1:65535; # force a 502{{end}}
}

server{
listen 8788;
server_name hello.zongwu233.com;
location / {
proxy_pass http://my-web-server;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}

这里配置了一个upstream server。其中

1
{{range service "my-web-server"}}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;

是获取 consul中所有的my-web-server ,生成多个 server配置 ,由于我们只启动了一个实例,最终生成:

1
server 172.17.0.2 9123 max_fails=3 fail_timeout=60 weight=1;

172.17.0.2是 testHello 实例的容器ip。

这里的8788端口用于Ngnix对接我们使用的SLB。如果没有SLB需要依据实际情况修改(比如写成80)。

OpenResty

将Nginx替换为OpenResty,不能直接使用 nginx-consul-template 了,自己定义Dockerfile。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
FROM openresty/openresty:1.13.6.2-2-alpine
MAINTAINER zongwu233

RUN wget -P /usr/local/openresty/lualib/resty/ https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http.lua \
&& wget -P /usr/local/openresty/lualib/resty/ https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http_headers.lua

RUN ln -s /usr/local/openresty/nginx/sbin/nginx /usr/sbin/nginx

# install runit curl
RUN apk --update --no-cache add curl runit

#ENV CT_URL http://releases.hashicorp.com/consul-template/0.19.0/consul-template_0.19.0_linux_amd64.tgz
#RUN curl -L $CT_URL | tar -C /usr/local/bin/ --strip-components 1 -zxf -
ADD consul-template_0.19.0_linux_amd64.tgz /usr/local/bin/

ADD nginx.service /etc/service/nginx/run
RUN chmod a+x /etc/service/nginx/run
ADD consul-template.service /etc/service/consul-template/run
RUN chmod a+x /etc/service/consul-template/run

RUN rm -v /etc/nginx/conf.d/*
RUN mkdir -p /run/nginx/
ADD nginx.conf.ctmpl /etc/consul-templates/nginx.conf.ctmpl

ADD lualib/access-auth.lua /usr/local/openresty/site/lualib/access-auth.lua
ADD lualib/limit-invite.lua /usr/local/openresty/site/lualib/limit-invite.lua

CMD ["runsvdir", "/etc/service"]

构建镜像并且替换刚才的 registry.cn-hangzhou.aliyuncs.com/zongwu233/nginx-consul-template:v1.1 即可。

其他

关于结合consul集群,定制nginx.conf.ctmpl文件,支持蓝绿部署策略,有时间再介绍。