背景

有两台minio文件服务器已做好集群配置，一台是192.168.56.41:9000；另一台是192.168.56.42:9000。

应用程序通过Nginx负载均衡调用这两台minio服务，减轻单点压力、防止单点故障。

Nginx装在192.168.56.41上。

安装nginx

1、官网下载nginx，上传到linux服务器上

http://nginx.org/en/download.html

2、解压，进入nginx，配置

tar -xzf nginx-1.24.0.tar.gz
 mv nginx-1.24.0 nginx
 cd nginx/
 #指定 nginx 的安装路径为 /usr/local/nginx，同时启用了 SSL 和状态监控模块。
 ./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-pcre

在新装的centos7上面安装nginx到时候，执行./config 时候 出现错误。

checking for OS
 + Linux 3.10.0-1127.el7.x86_64 x86_64
checking for C compiler ... not found

./configure: error: C compiler cc is not found

执行下列命令安装gcc

yum -y install gcc gcc-c++ autoconf automake make

再次配置，报错

./configure: error: the HTTP rewrite module requires the PCRE library.
You can either disable the module by using --without-http_rewrite_module
option, or install the PCRE library into the system, or build the PCRE library
statically from the source with nginx by using --with-pcre=<path> option.

安装pcre-devel

yum install -y pcre-devel

再次配置，报错

 ./configure: error: SSL modules require the OpenSSL library. You can either do not enable the modules, or install the OpenSSL library into the system, or build the OpenSSL library statically from the source with nginx by using --with-openssl= option.

安装openssl

yum -y install openssl openssl-devel

再次配置，成功

Configuration summary
  + using system PCRE library
  + using system OpenSSL library
  + using system zlib library

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx modules path: "/usr/local/nginx/modules"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
  nginx http uwsgi temporary files: "uwsgi_temp"
  nginx http scgi temporary files: "scgi_temp"

3、安装

make && make install

4、启动

#添加软连接，以后可在任意目录使用nginx命令
ln -s /usr/local/nginx/sbin/nginx /usr/bin/nginx
#启动
nginx
#重启
nginx -s reload

浏览器访问http://192.168.56.41/，直接访问nginx欢迎页面

5、开机启动

方法一

1.找到/etc/rc.local文件，在文件最后一行新增启动命令(nginx默认安装目录为：/usr/local/nginx)：

/usr/local/nginx/sbin/nginx

方法二（推荐，方便指定启动时机）

(1).进入/etc/systemd/system文件夹，新增文件 nginx.service

(2).文件内容：

[Unit]
Description=nginx service
After=network.target

[Service]
User=root
Type=forking
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=/usr/local/nginx/sbin/nginx -s reload
ExecStop=/usr/local/nginx/sbin/nginx -s stop
ExecStartPre=/bin/sleep 10

[Install]
WantedBy=multi-user.target

(3).开启开机启动

chmod +x /etc/systemd/system/nginx.service
systemctl enable nginx
nginx -s stop
systemctl start nginx

负载均衡

过程

1、配置nginx.conf，http下增加

upstream  minio_server {
        server 192.168.56.41:9000 weight=1;
        server 192.168.56.42:9000 weight=1; 
}
server {
        listen       9100;
        server_name  localhost;
        charset utf-8;
        location /{
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-For $remote_addr;
            client_body_buffer_size 10M;
            client_max_body_size 1G;
            proxy_buffers 1024 4k;
            proxy_read_timeout 300;
            proxy_next_upstream error timeout http_404;
            proxy_pass http://minio_server;
        }
}

2、修改应用程序对Minio服务的访问地址，指向Nginx 9100端口

minio:
  endpoint: http://192.168.56.41:9100

3、重启程序，实现对minio api的负载均衡

nginx https

自己签发证书

创建SSL证书

1、创建证书目录

cd /usr/local/nginx/conf
#创建ssl_ok目录并进入
mkdir ssl_ok && cd $_

2、创建一个ssl配置文件，指定证书请求时的一些参数，注意commonName_default（域名）为*.http://kunsam.com，以便扩展子域名

[ req ]
default_bits       = 4096
distinguished_name = req_distinguished_name
req_extensions     = req_ext

[ req_distinguished_name ]
countryName                 = Country Name (2 letter code)
countryName_default         = cn
stateOrProvinceName         = State or Province Name (full name)
stateOrProvinceName_default = sc
localityName                = Locality Name (eg, city)
localityName_default        = cd
organizationName            = Organization Name (eg, company)
organizationName_default    = my
organizationalUnitName            = Organizational Unit Name (eg, section)
organizationalUnitName_default    = as
commonName                  = Common Name (e.g. server FQDN or YOUR name)
commonName_max              = 64
commonName_default          = *.kunsam.com

[ req_ext ]
subjectAltName = @alt_names

# 此段落标题的方括号两边【没有空格】，只有同时配有IP和域名，才能在IP和域名访问时都成功识别。
[alt_names]
IP.1    = 192.168.56.41
DNS.1   = *.kunsam.com
DNS.2   = *.kunsam.net

3、生成一个RSA密钥

[root@localhost ssl_ok]# openssl genrsa -out server.key 4096
Generating RSA private key, 4096 bit long modulus
...............................................................................................................................................................................................................................................................++
....................++
e is 65537 (0x10001)

4、生成证书请求文件

CSR是Certificate Signing Request的英文缩写，即证书请求文件，也就是证书申请者在申请数字证书时由CSP(加密服务提供者)在生成私钥的同时也生成证书请求文件，证书申请者只要把CSR文件提交给证书颁发机构后，证书颁发机构使用其根证书私钥签名就生成了证书公钥文件，也就是颁发给用户的证书。

[root@localhost ssl_ok]# openssl req -new -sha256 -out server.csr -key server.key -config mySsl.conf 
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [cn]:
State or Province Name (full name) [sc]:
Locality Name (eg, city) [cd]:
Organization Name (eg, company) [my]:
Organizational Unit Name (eg, section) [as]:
Common Name (e.g. server FQDN or YOUR name) [kunsam.com]:
[root@localhost ssl_ok]

5、签发证书(由于是测试自己签发，实际应该将自己生成的csr文件提交给SSL认证机构认证)

[root@localhost ssl_ok]# openssl x509 -req -days 3650 -in server.csr -signkey server.key -out server.crt -extensions req_ext -extfile mySsl.conf
Signature ok
subject=/C=cn/ST=sc/L=cd/O=my/OU=as/CN=*.kunsam.com
Getting Private key
[root@localhost ssl_ok]#

配置Nginx SSL

1、在nginx.conf http下增加对http://minio.kunsam.com域名的监听

server {
        listen      80 default_server;
        listen      [::]:80 default_server;
        server_name minio.kunsam.com;
        #include    /etc/nginx/default.d/*.conf;
        return      301 https://$server_name$request_uri;    #在80监听端口 配置跳转
    }
    server {
        listen       443 ssl http2 default_server;
        listen       [::]:443 ssl http2 default_server;
        server_name minio.kunsam.com;
        client_max_body_size 1024M;
        charset utf-8;
        ssl_certificate "/usr/local/nginx/conf/ssl_ok/server.crt";   #公钥，它会被发送到连接服务器的每个客户端
        ssl_certificate_key "/usr/local/nginx/conf/ssl_ok/server.key"; #私钥是用来解密的
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout  10m;
        ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; #加密套件
        ssl_prefer_server_ciphers on;
        location /{
            proxy_set_header  Host       $host;
            proxy_set_header  X-Real-IP    $remote_addr;
            proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://minio_server;
        }
    }

2、测试nginx

[root@localhost conf]# nginx -t
nginx: [emerg] the "http2" parameter requires ngx_http_v2_module in /usr/local/nginx/conf/nginx.conf:130
nginx: configuration file /usr/local/nginx/conf/nginx.conf test failed

提示未安装http2模块

增加http2模块

无需覆盖原来安装的nginx，只需配置、编译、替换执行文件

1、新配置

cd /opt/apps/nginx
./configure --prefix=/usr/local/nginx --with-http_ssl_module --with-http_stub_status_module --with-pcre --with-http_v2_module

2、编译【特别注意】这里不要进行make install，否则就是覆盖你之前的安装

make

3、将刚刚编译好的nginx覆盖掉原有的nginx（这个时候【特别注意】这里nginx要停止状态）

systemctl stop nginx
cp ./objs/nginx /usr/local/nginx/sbin/

4、启动nginx

systemctl start nginx

HTTP2只对HTTPS生效，所以要同时配置，HTTP2的加载速度比HTTP1.1快很多

完整nginx配置如下

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    #gzip  on;

    upstream  minio_server {
        server 192.168.56.41:9000 weight=1;
        server 192.168.56.42:9000 weight=1; 
    }

    server {
        listen       9100;
        server_name  localhost;
        charset utf-8;
        location /{
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-For $remote_addr;
            client_body_buffer_size 10M;
            client_max_body_size 1G;
            proxy_buffers 1024 4k;
            proxy_read_timeout 300;
            proxy_next_upstream error timeout http_404;
            proxy_pass http://minio_server;
        }
    }

    server {
        listen       80;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
    server {
        listen      80 default_server;
        listen      [::]:80 default_server;
        server_name minio.kunsam.com;
        #include    /etc/nginx/default.d/*.conf;
        return      301 https://$server_name$request_uri;    #在80监听端口 配置跳转
    }
    server {
        listen       443 ssl http2 default_server;
        listen       [::]:443 ssl http2 default_server;
        server_name minio.kunsam.com;
        client_max_body_size 1024M;
        charset utf-8;
        ssl_certificate "/usr/local/nginx/conf/ssl_ok/server.crt";   #公钥，它会被发送到连接服务器的每个客户端
        ssl_certificate_key "/usr/local/nginx/conf/ssl_ok/server.key"; #私钥是用来解密的
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout  10m;
        #ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; #加密套件
        #ssl_protocols TLSv1 TLSv1.1 TLSv1.2;   #配置协议
        ssl_prefer_server_ciphers on;
        location /{
            proxy_set_header  Host       $host;
            proxy_set_header  X-Real-IP    $remote_addr;
            proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_pass http://minio_server;
        }
    }
}

客户机配置域名

访问minio的机器都需要配置，包括java应用程序服务器

1、打开C:\Windows\System32\drivers\etc\hosts文件，增加如下内容

192.168.56.41 minio.kunsam.com

2、客户机浏览器 https://minio.kunsam.com ，转发到http://minio.kunsam.com:9001/

3、修改Java程序中配置的minio服务地址

minio:
  endpoint: https://minio.kunsam.com

4、将证书导入到应用服务器的jdk，否则无法执行https访问。先把server.crt下载到应用程序服务器，执行下列命令

keytool -import -alias nginx_minio_server -keystore cacerts -file ./server.crt

或者，不导入证书则修改java代码，构建MinioClient时取消SSL认证

@Bean
public MinioClient minioClient() throws NoSuchAlgorithmException, KeyManagementException {
        //取消ssl认证
        final TrustManager[] trustAllCerts = new TrustManager[]{
                new X509TrustManager() {
                    @Override
                    public void checkClientTrusted(X509Certificate[] x509Certificates, String s) {
                    }
                    @Override
                    public void checkServerTrusted(X509Certificate[] x509Certificates, String s) {
                    }
                    @Override
                    public X509Certificate[] getAcceptedIssuers() {
                        return new X509Certificate[]{};
                    }
                }
        };
        X509TrustManager x509TrustManager = (X509TrustManager) trustAllCerts[0];
        final SSLContext sslContext = SSLContext.getInstance("SSL");
        sslContext.init(null, trustAllCerts, new SecureRandom());
        final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
        OkHttpClient.Builder builder = new OkHttpClient.Builder();
        builder.sslSocketFactory(sslSocketFactory,x509TrustManager);

        builder.hostnameVerifier((s, sslSession) -> true);
        OkHttpClient okHttpClient = builder.build();

        return MinioClient.builder().endpoint(endpoints).httpClient(okHttpClient).region("eu-west-1").credentials(accessKey
                , secretKey).build();
    }

5、启动程序，postman调用上传成功

申请免费SSL证书

前面的SSL证书是自己签发的，浏览器认为该证书无效，可以到http://freessl.cn申请免费的被浏览器认可的SSL证书，有效期30天并自动续期

前提

公网域名，公网服务器

过程

1、登录http://freessl.cn，输入如http://example.com点击“创建免费的SSL证书”按钮

2、点击下一步，到域名服务控制台添加 _acme-challenge 主机记录，记录类型: CNAME

3、到公网服务器安装acme

curl https://get.acme.sh | sh -s email=my@example.com

如果上面官方下载地址失败 或者 太慢，可以选用国内的备用地址

curl https://gitcode.net/cert/cn-acme.sh/-/raw/master/install.sh?inline=false | sh -s email=my@example.com

4、将freessl上域名的acme.sh部署命令拷贝到服务器上执行

A1D60901-574E-4317-A678-E301030198FA

5、将公钥、私钥拷贝到nginx，确保 /usr/local/nginx/conf/ssl/example.com目录存在

acme.sh --install-cert -d example.com \
--key-file       /usr/local/nginx/conf/ssl/example.com/key.pem  \
--fullchain-file /usr/local/nginx/conf/ssl/example.com/cert.pem \
--reloadcmd     "service nginx force-reload"

证书进入到30天有效期，acme.sh 会自动完成续期。

6、nginx配置ssl证书

server {
    #监听所有的ipv4的地址
    listen 80 default_server;

    #监听所有的ipv6的地址
    listen [::]:80

    #填写绑定证书的域名
    server_name example.com;

    #把http的域名请求转成https
    return 301 https://$host$request_uri;
}
server {
        listen       443 ssl;
        server_name  example.com;

        ssl_certificate      /usr/local/nginx/conf/ssl/example.com/cert.pem;
        ssl_certificate_key  /usr/local/nginx/conf/ssl/example.com/key.pem;

        ssl_session_cache    shared:SSL:1m;
        ssl_session_timeout  5m;

        ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; #加密套件
        ssl_prefer_server_ciphers  on;

        location / {
            root   html;
            index  index.html index.htm;
        }
    }

default_server属性是标识符，用来将此虚拟主机设置成默认主机。所谓的默认主机指的是如果没有匹配到对应的address:port，则会默认执行的。如果不指定默认使用的是第一个server。

7、启动nginx，可通过https://example.com或者https://www.example.com访问到。

建议每个子域名都单独申请证书，而不是用 *.http://example.com，以免证书同时到期系统无法使用。和自己签发证书不同，自己签发可规定有效期

注意：域名和服务器提供商不同可能导致域名的http访问被服务器提供商拦截，https访问不会。

8、acme.sh 删除域名

acme.sh --remove -d example.com
rm -rf /root/.acme.sh/example.com_ecc

9、acme.sh卸载

acme.sh --uninstall
rm -rf /root/.acme.sh

参考文献

CentOS7操作系统安装nginx实战（多种方法，超详细）

linux配置nginx开机启动

Nginx单独开启SSL模块和HTTP2模块，无需重新覆盖安装Nginx如果未开启SSL模块，配置Https时提示错误

JDK导入ssl证书

浏览器显示“SSL证书无效”怎么办（SSL证书不是由受信的CA机构所签发的。）

关于SSL认证的小坑 SSLPeerUnverifiedException

生成带subjectAltName的ssl服务端证书【亲测有效】（SSLPeerUnverifiedException: Hostname minio.kunsam.com not verified:问题解决）

Java实现minio上传、下载、删除文件，支持https访问 （取消SSL认证）

完全卸载nginx及安装的详细步骤