前言

部署mysql之前我们需要先了解一个概念有状态服务。这是一种特殊的服务，简单的归纳下就是会产生需要持久化的数据，并且有很强的I/O需求，且重启需要依赖上次存储到磁盘的数据。如典型的mysql，kafka，zookeeper等等。

在我们有比较优秀的商业存储的前提下，灰常推荐使用有状态服务进行部署，计算和存储分离那是相当的爽的。在实际生产中如果没有这种存储，才有localPV也是不错的选择，当然local pv其实呢和hostPath是一样的。当然我们在开发测试环境也是可以自己搭建一套简单的如NFS服务，来享受存储和计算分离的爽快感。

kubernetes中定义一种了资源类型Stateful Service即有状态服务，有状态服务需要的持久化数据动态绑定我们可以利用存储的API PersistentVolume（PV）和PersistentVolumeClaim（PVC）来进行需要的相关数据的绑定和存储。

PV & PVC

PV就好比是一个仓库，我们需要先购买一个仓库，即定义一个PV存储服务，例如CEPH,NFS,Local Hostpath等等。PVC就好比租户，pv和pvc是一对一绑定的，挂载到POD中，一个pvc可以被多个pod挂载。大致一个流程如下，可以从这里,以及官网看到更多的额关于PV的细节

创建PV --> 创建PVC --> 绑定 --> 可写入数据

有了这个理解之后，我们接下来实战一下（mysql(pvc)+NFS（PV））

NFS Server

首先我们需要创建一个nfs server。我从hub.docker找到一个nfs-server。下面我们将它部署到kubernetes中。

部署nfs到kubernetes

# nfs-server.yaml
apiVersion: v1
kind: Service
metadata:
  name: nfs-service
  labels:
    app: nfs-service
spec:
  ports:
    - port: 2049
      name: nfs-service
  clusterIP: 10.96.0.14
  selector:
    app: nfs-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-service
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-service
  template:
    metadata:
      labels:
        app: nfs-service
    spec:
      # 采用node选择器
      nodeName: node1
      containers:
        - name: nfs-service
          image: itsthenetwork/nfs-server-alpine:latest
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            capabilities:
              add:
                - SYS_ADMIN
                - SETPCAP
          livenessProbe:
            tcpSocket:
              port: 2049
            timeoutSeconds: 5
          readinessProbe:
            tcpSocket:
              port: 2049
            timeoutSeconds: 5
          ports:
            - containerPort: 2049
          env:
            - name: SHARED_DIRECTORY
              value: /nfsshare
          volumeMounts:
            - mountPath: /nfsshare
              name: nfsshare
      volumes:
        - name: nfsshare
          hostPath:
            path: /nfsshare
            type: DirectoryOrCreate

部署到集群

script
1kubectl apply -f nfs-server.yaml

查看日志

script
root@server1:~# kubectl logs -f deploy/nfs-service
Starting rpcbind...
Displaying rpcbind status...
   program version netid     address                service    owner
    100000    4    tcp       0.0.0.0.0.111          -          superuser
    100000    3    tcp       0.0.0.0.0.111          -          superuser
    100000    2    tcp       0.0.0.0.0.111          -          superuser
    100000    4    udp       0.0.0.0.0.111          -          superuser
    100000    3    udp       0.0.0.0.0.111          -          superuser
    100000    2    udp       0.0.0.0.0.111          -          superuser
    100000    4    local     /var/run/rpcbind.sock  -          superuser
    100000    3    local     /var/run/rpcbind.sock  -          superuser
Starting NFS in the background...
rpc.nfsd: knfsd is currently down
rpc.nfsd: Writing version string to kernel: -2 -3 +4 +4.1 +4.2
rpc.nfsd: Created AF_INET TCP socket.
Exporting File System...
exporting *:/nfsshare
/nfsshare       <world>
Starting Mountd in the background...These
Startup successful.

验证

安装nfs-client。

CentOS

script
1sudo yum install nfs-utils

Ubuntu

script
1sudo apt-get install nfs-common -y

挂载到node的目录上

script
1sudo mount -t nfs -o vers=4,minorversion=0,noresvport 10.96.0.14:/ /nfs

向挂载的NFS木写入数据，

script
echo 1 > /nfs/1.txt

前往node1查看，看到如下结果表示NFS服务器搭建成功。

script
root@node1:/nfsshare# ls
1.txt
root@node1:/nfsshare# cat 1.txt 
1

静态PV

搭建完了nfs之后，我们就可以创建PV了

部署pv

apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-static-pv
spec:
  capacity:
    storage: 10Gi
  storageClassName: mysql-scn
  #ReadWriteOnce - 卷可以由单个节点以读写方式挂载
  #ReadOnlyMany - 卷可以由许多节点以只读方式挂载
  #ReadWriteMany - 卷可以由许多节点以读写方式挂载
  accessModes:
    - ReadWriteOnce
  #Retain，不清理, 保留 Volume（需要手动清理）
  #Recycle，删除数据，即 rm -rf /thevolume/*（只有 NFS 和 HostPath 支持）
  #Delete，删除存储资源，比如删除 AWS EBS 卷（只有 AWS EBS, GCE PD, Azure Disk 和 Cinder 支持）
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /
    server: 10.96.0.14
  mountOptions:
    - vers=4
    - minorversion=0
    - noresvport

部署PV到集群中

script
$ kubectl apply -f mysql-pv.yaml 
persistentvolume/mysql-static-pv created

查看PV状态，看到status为Available表示PV创建成功。

script
$ kubectl get pv
NAME              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
mysql-static-pv   10Gi       RWO            Retain           Available           mysql-scn               3s

部署PVC

script
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-static-pvc
spec:
  #RWO - ReadWriteOnce
  #ROX - ReadOnlyMany
  #RWX - ReadWriteMany
  accessModes:
    - ReadWriteOnce
  storageClassName: mysql-scn
  resources:
    requests:
      storage: 10Gi

部署到集群中

script
$ kubectl apply -f mysql-static-pvc.yaml 
persistentvolumeclaim/mysql-static-pvc created

查看状态，看到STATUS=Bound表示PVC和PV绑定成功了。可以看到我们有一个名称为mysql-static-pvc的pvc可以用了。

script
freemandeMacBook-Pro:pv-pvc freeman$ kubectl get pvc
NAME               STATUS   VOLUME            CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql-static-pvc   Bound    mysql-static-pv   10Gi       RWO            mysql-scn      4s

部署mysql使用pvc存储

# mysql-server.yaml
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  ports:
    - port: 3306
      name: mysql
  clusterIP: 10.96.0.15
  selector:
    app: mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql/mysql-server:8.0.17
          imagePullPolicy: IfNotPresent
          livenessProbe:
            tcpSocket:
              port: 3306
            timeoutSeconds: 5
          readinessProbe:
            tcpSocket:
              port: 3306
            timeoutSeconds: 5
          ports:
            - containerPort: 3306
          volumeMounts:
            - mountPath: /var/lib/mysql
              name: mysql-data
              readOnly: false
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: "root"
      volumes:
        - name: mysql-data
          persistentVolumeClaim:
            # 这里名称需要和pvc一一对应
            claimName: mysql-static-pvc

部署到集群

script
kubectl apply -f mysql-server.yaml

查看日志

$ kubectl logs -f deploy/mysql
...
[Entrypoint] Starting MySQL 8.0.17-1.1.12
2019-08-22T09:44:49.137554Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.17) starting as process 1
2019-08-22T09:44:49.754020Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2019-08-22T09:44:49.795415Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.17'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2019-08-22T09:44:49.943852Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: '/var/run/mysqld/mysqlx.sock' bind-address: '0.0.0.0' port: 33060

查看运行状态

script
$ kubectl get pods -owide | grep mysql
mysql-f475dc6c8-xnb4x                      1/1     Running            0          5m20s   172.224.3.158   node1               <none>           <none>

查看node1 nfs目录下的数据，可以看到mysql的文件都存储到了我们的nfs属猪鸡的文件系统中了。而我们在部署mysql时，是不需要指定node也没有挂载hostpath，这意味着我们的mysql计算是可以运行在任意集群中的noe上。

script
root@node1:/nfsshare# ls /nfsshare/
 auto.cnf        binlog.index   client-cert.pem   ibdata1       ibtmp1          mysql.ibd         performance_schema   server-cert.pem   undo_001
 binlog.000001   ca-key.pem     client-key.pem    ib_logfile0  '#innodb_temp'   mysql.sock        private_key.pem      server-key.pem    undo_002
 binlog.000002   ca.pem         ib_buffer_pool    ib_logfile1   mysql           mysql.sock.lock   public_key.pem       sys

测试mysql 迁移到其他node运行

我们先看一下目前mysql是运行在哪个节点上，可以看到目前是运行在node1上。

script
$ kubectl get pods -owide | grep mysql
mysql-f475dc6c8-xnb4x                      1/1     Running            0          5m20s   172.224.3.158   node1               <none>           <none>

为了方便测试我修改一下mysql的yaml文件加入node选择器,将mysql强行调度到node2运行。

spec.template.spec.nodeName: node2

重新apply

$ kubectl apply -f mysql-server.yaml 
service/mysql unchanged
deployment.apps/mysql configured

查看mysql是否还运行正常,可以看到运行正常，这就是存储和计算分离的爽点。

script
$ kubectl get pods -owide | grep mysql
mysql-6f5944cbcd-nd7c4                     1/1     Running            0          38s     172.224.4.18    node2               <none>           <none>

授权远程访问

script
$ kubectl get pods | grep mysql
mysql-6f5944cbcd-nd7c4                     1/1     Running            0          7m58s
$ kubectl exec -it mysql-6f5944cbcd-nd7c4 bash
$ mysql -uroot -proot
$ CREATE USER 'root'@'%' IDENTIFIED BY 'root';
$ GRANT ALL ON *.* TO 'root'@'%';
$ ALTER USER 'root'@'%' IDENTIFIED WITH mysql_native_password BY 'root';
$ FLUSH PRIVILEGES;

在开发机进行访问

script
$ mysql -uroot -proot -h 10.96.0.15
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 136
Server version: 8.0.17 MySQL Community Server - GPL

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

到此我们完成了基于Mysql+nfs的部署工作。