MySQL-MGR集群搭建
一、架构图
二、前决条件
- 一个可用的K8S集群已经搭建完成，所有节点可通过
kubectl访问- 拥有一台NFS服务器（例如IP地址为192.168.228.160），并且已经安装了nfs-utils，并且创建共享目录（如/data/k8s/mysql）
- 所有K8S节点上已经安装nfs-utils
- 确保所有的K8S节点上都能够挂载NFS共享出的目录
- mysql基础镜像版本为
8.0
三、环境准备
所有主机上都安装nfs-utils
yum-yinstallnfs-utils在K8S的Worker节点上导入mysql:8.0镜像
dockerload-imysql.tar四、部署流程
4.1、部署NFS服务器
为所有的MySQL实例创建一个公有的文件夹，用于存放K8S动态生成的PV
# 创建共享目录mkdir-p/data/k8s/mysql# 为该共享目录赋予权限chmod0755 /data/k8s/mysql编辑/etc/exports文件，将上述创建好的/data/k8s/mysql共享出去给K8S集群中的节点
vim/etc/exports#================== 添加如下内容 ======================#/data/k8s/mysql192.168.228.0/24(rw,sync,no_subtree_check,no_root_squash)启动NFS服务
systemctl start nfs查看exports配置是否生效
exportfs-arv#========== 会输出如下类似信息 ==================#exporting192.168.228.0/24:/data/k8s/mysql检查是否可以共享成功(在本机测试挂载)
showmount-elocalhost#========== 会输出如下类似信息 ==================#Export listforlocalhost: /data/k8s/mysql192.168.228.0/24验证NFS客户端是否能够挂载共享目录（在任意一台K8S节点上进行测试即可）
# 对共享目录进行挂载测试mount192.168.228.160:/data/k8s/mysql /mnt#============== 查看是否挂载成功 ===============#df-Th|grep/mnt192.168.228.160:/data/k8s/mysql nfs4 69G3.0G 62G5% /mnt在NFS客户端测试共享目录写入文件是否成功
# 在刚刚挂载好的/mnt挂载点创建一个空间建进行测试echo"test">/mnt/testfile回到NFS服务器的共享目录中查看是否创建成功
cat/data/k8s/mysql/testfiletest卸载NFS客户端中的挂载点即可
umount/mnt如果上述测试全部成功，说明 NFS 服务端配置正确，K8s 节点可以正常使用该 NFS 共享!!!
4,2、部署NFS StorageClass动态供给器
我们使用官方提供的nfs-subdir-external-provisioner为NFS提供动态PV支持
我们在K8S中的Master节点上创建一个MySQL的工程目录，执行如下命令
mkdir-p~/mysql在这个~/mysql工程目录下创建一个存放ServiceAccount资源清单的文件夹，执行如下命令
mkdir-p~/mysql/serviceaccount在~/mysql/serviceaccount目录下创建一个名为nfs-rbac.yaml的资源清单文件，执行如下命令
vim ~/mysql/serviceaccount/nfs-rbac.yaml#============== 添加如下内容 -============#apiVersion:v1kind:ServiceAccountmetadata:name:nfs-client-provisionernamespace:kube-system---apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRolemetadata:name:nfs-client-provisioner-runnerrules:-apiGroups:[""]resources:["persistentvolumes"]verbs:["get","list","watch","create","delete"]-apiGroups:[""]resources:["persistentvolumeclaims"]verbs:["get","list","watch","update"]-apiGroups:["storage.k8s.io"]resources:["storageclasses"]verbs:["get","list","watch"]-apiGroups:[""]resources:["events"]verbs:["create","update","patch"]-apiGroups:[""]resources:["endpoints"]verbs:["get","list","watch","create","update","patch"]---apiVersion:rbac.authorization.k8s.io/v1kind:ClusterRoleBindingmetadata:name:run-nfs-client-provisionersubjects:-kind:ServiceAccountname:nfs-client-provisionernamespace:kube-systemroleRef:kind:ClusterRolename:nfs-client-provisioner-runnerapiGroup:rbac.authorization.k8s.io---apiVersion:rbac.authorization.k8s.io/v1kind:Rolemetadata:name:leader-locking-nfs-client-provisionernamespace:kube-systemrules:-apiGroups:[""]resources:["endpoints"]verbs:["get","list","watch","create","update","patch"]---apiVersion:rbac.authorization.k8s.io/v1kind:RoleBindingmetadata:name:leader-locking-nfs-client-provisionernamespace:kube-systemsubjects:-kind:ServiceAccountname:nfs-client-provisionernamespace:kube-systemroleRef:kind:Rolename:leader-locking-nfs-client-provisionerapiGroup:rbac.authorization.k8s.io4.3、部署Provisioner Deployment
**创建一个用于存放Deployment资源对象文件夹，路径为~/mysql/deployment，并且创建一个资源清单文件nfs-provisioner-deploy.yaml**执行如下命令
mkdir-p~/mysql/deployment&&touch~/mysql/deployment/nfs-provisioner-deploy.yaml对~/mysql/deployment/nfs-provisioner-deploy.yaml配置文件进行编辑，配置内容如下
注意：NFS_SERVER和NFS_PATH两个的VALUE值都要替换为你自己NFS服务器的地址和路径
apiVersion:apps/v1kind:Deploymentmetadata:name:nfs-client-provisionernamespace:kube-systemspec:replicas:1strategy:type:Recreateselector:matchLabels:app:nfs-client-provisionertemplate:metadata:labels:app:nfs-client-provisionerspec:serviceAccountName:nfs-client-provisionercontainers:-name:nfs-client-provisionerimage:registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2imagePullPolicy:IfNotPresentvolumeMounts:-name:nfs-client-rootmountPath:/persistentvolumesenv:-name:PROVISIONER_NAMEvalue:k8s-sigs.io/nfs-subdir-external-provisioner-name:NFS_SERVERvalue:192.168.228.160# 替换为你的 NFS 服务器 IP-name:NFS_PATHvalue:/data/k8s/mysql# 替换为你的 NFS 共享目录volumes:-name:nfs-client-rootnfs:server:192.168.228.160# 替换为你的 NFS 服务器 IPpath:/data/k8s/mysql# 替换为你的 NFS 共享目录4.4、创建StorageClass
创建一个用于存放StorageClass资源对象的文件夹，路径为~/mysql/storageclass，并创建一个名为nfs-storageclass.yaml的资源清单文件执行如下命令
mkdir~/mysql/storageclass&&touch~/mysql/storageclass/nfs-storageclass.yaml编辑刚刚创建好的资源清单文件~/mysql/storageclass/nfs-storageclass.yaml，添加如下内容
apiVersion:storage.k8s.io/v1kind:StorageClassmetadata:name:mysql-nfs-storageprovisioner:k8s-sigs.io/nfs-subdir-external-provisionerparameters:archiveOnDelete:"false"reclaimPolicy:Delete4.5、应用配置并验证
将我们4.1-4.4章节中写好的资源清单文件一并提交。
kubectl apply-f~/mysql/serviceaccount/nfs-rbac.yaml kubectl apply-f~/mysql/deployment/nfs-provisioner-deploy.yaml kubectl apply-f~/mysql/storageclass/nfs-storageclass.yaml进行验证（检查 provisioner Pod 是否运行）
kubectl get pods-nkube-system|grepnfs-client-provisioner#=============== 输出如下类似内容 ==================#nfs-client-provisioner-859d8bff96-4dfdg1/1 Running011m进行验证（检查 StorageClass）
kubectl get storageclass mysql-nfs-storage#=============== 输出如下类似内容 =================#NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE mysql-nfs-storage k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediatefalse12m4.6、部署MySQL MGR集群
4.6.1、创建MySQL集群的命名空间
创建一个目录用于存放Namespace资源对象的资源清单文件，路径为~/mysql/namespace/，文件名为mysql-mgr-namespace.yaml，执行如下命令
mkdir-p~/mysql/namespace&&touch~/mysql/namespace/mysql-mgr-namespace.yaml编辑这个mysql-mgr-namespace.yaml文件，并添加如下内容：
apiVersion:v1kind:Namespacemetadata:name:mysql-mgr-namespacelabels:name:mysql-workspaceproject:database提交这个mysql-mgr-namespace.yaml资源清单文件
kubectl apply-f~/mysql/namespace/mysql-mgr-namespace.yaml查看命名空间是否创建成功
kubectl get namespace --show-labels|grepname=mysql-workspace#=================== 输出如下内容 ======================#mysql-mgr-namespace Active 50s kubernetes.io/metadata.name=mysql-mgr-namespace,name=mysql-workspace,project=database4.6.2、创建ConfigMap（配置文件和初始化脚本）
创建一个目录用于存放configMap资源对象的清单文件，并且配置文件名为mysql-mgr-configmap.yaml,路径为~/mysql/configmap/mysql-mgr-configmap.yaml，执行如下命令：
mkdir-p~/mysql/configmap&&touch~/mysql/configmap/mysql-mgr-configmap.yaml编辑~/mysql/configmap/mysql-mgr-configmap.yaml这个清单文件，内容如下：
apiVersion:v1kind:ConfigMapmetadata:name:mysql-mgr-confignamespace:mysql-mgr-namespacedata:my.cnf:|[mysqld] server_id=0 log_bin=mysql-bin binlog_format=ROW gtid_mode=ON enforce_gtid_consistency=ON master_info_repository=TABLE relay_log_info_repository=TABLE binlog_checksum=NONE log_slave_updates=ON transaction_write_set_extraction=XXHASH64plugin_load_add='group_replication.so' group_replication_group_name="22aa1aee-c2b4-4df9-876a-bf370f983b29" group_replication_start_on_boot=OFF group_replication_local_address=:33061group_replication_group_seeds="mysql-mgr-0.mysql-mgr-headless.mysql-mgr.svc.cluster.local:33061,mysql-mgr-1.mysql-mgr-headless.mysql-mgr.svc.cluster.local:33061,mysql-mgr-2.mysql-mgr-headless.mysql-mgr.svc.cluster.local:33061" group_replication_bootstrap_group=OFF group_replication_ip_whitelist="0.0.0.0/0" group_replication_recovery_use_ssl=OFF group_replication_ssl_mode=DISABLED group_replication_recovery_reuse_ssl=OFF loose-group_replication_single_primary_mode=ON loose-group_replication_enforce_update_everywhere_checks=OFF datadir=/var/lib/mysql socket=/var/run/mysqld/mysqld.sock pid-file=/var/run/mysqld/mysqld.pid user=mysqlinit-mgr.sh:|#!/bin/bash set -eHOSTNAME=$(hostname) ORD=${HOSTNAME##*-}SERVER_ID=$((100 + ORD))# 准备 my.cnfcp /etc/mysql/my.cnf.template /etc/mysql/my.cnf sed-i "s/^server_id=0$/server_id=$SERVER_ID/" /etc/mysql/my.cnf# 数据目录初始化（如果为空）if[!-d "/var/lib/mysql/mysql"]; then echo "Initializing MySQL data directory..." mysqld--initialize-insecure--user=mysql--datadir=/var/lib/mysql fi# 启动 MySQL 后台运行mysqld--user=mysql--datadir=/var/lib/mysql \--socket=/var/run/mysqld/mysqld.sock \--pid-file=/var/run/mysqld/mysqld.pid & mysql_pid=$!# 等待 MySQL 启动for i in{1..60}; do if mysqladmin ping-uroot--silent; then break fi sleep 2 if[$i-eq 60]; thenecho "ERROR:MySQL failed to start" exit 1 fi done# 设置 root 密码mysql-uroot-e "ALTER USER 'root'@'localhost' IDENTIFIED BY '${MYSQL_ROOT_PASSWORD}';"# 创建复制用户（用于 MGR 恢复通道）mysql-uroot-p"${MYSQL_ROOT_PASSWORD}" <<EOF CREATE USER IF NOT EXISTS 'repl'@'%' IDENTIFIED BY '${MYSQL_REPL_PASSWORD}'; GRANT REPLICATION SLAVE,BACKUP_ADMIN,GROUP_REPLICATION_STREAM ON*.*TO 'repl'@'%'; FLUSH PRIVILEGES; EOF# 安装 MGR 插件（幂等）mysql-uroot-p"${MYSQL_ROOT_PASSWORD}"-e "INSTALL PLUGIN group_replication SONAME 'group_replication.so';"||true# 配置恢复通道mysql-uroot-p"${MYSQL_ROOT_PASSWORD}"-e "CHANGE MASTER TO MASTER_USER='repl',MASTER_PASSWORD='${MYSQL_REPL_PASSWORD}' FOR CHANNEL 'group_replication_recovery';"# 引导或加入集群if["$ORD"-eq 0]; then echo "Bootstrapping MGR cluster on $HOSTNAME" mysql-uroot-p"${MYSQL_ROOT_PASSWORD}"-e "SET GLOBAL group_replication_bootstrap_group=ON;" mysql-uroot-p"${MYSQL_ROOT_PASSWORD}"-e "START GROUP_REPLICATION;" mysql-uroot-p"${MYSQL_ROOT_PASSWORD}"-e "SET GLOBAL group_replication_bootstrap_group=OFF;" else echo "Joining MGR cluster from $HOSTNAME" for i in{1..30}; do if mysql-uroot-p"${MYSQL_ROOT_PASSWORD}"-e "START GROUP_REPLICATION;" 2>/dev/null; then echo "Join succeeded" break fi echo "Waiting for primary node...($i/30)" sleep 2 done fi# 保持 MySQL 前台运行wait $mysql_pid应用创建好的ConfigMap~/mysql/configmap/mysql-mgr-configmap.yaml，执行如下命令
kubectl apply-f~/mysql/configmap/mysql-mgr-configmap.yaml4.6.3、创建Service（Headless+Client）
创建一个service目录用于存放Service资源对象的相关资源清单文件,文件名为mysql-mgr-services.yaml，路径为~/mysql/service/mysql-mgr-services.yaml，执行如下指令
mkdir-p~/mysql/service/&&touch~/mysql/service/mysql-mgr-services.yaml编辑~/mysql/service/mysql-mgr-services.yaml配置文件，内容如下：
apiVersion:v1kind:Servicemetadata:name:mysql-mgr-headlessnamespace:mysql-mgr-namespacespec:clusterIP:NonepublishNotReadyAddresses:trueselector:app:mysql-mgrports:-name:mysqlport:3306targetPort:3306-name:mgrport:33061targetPort:33061---apiVersion:v1kind:Servicemetadata:name:mysql-mgr-clientnamespace:mysql-mgr-namespacespec:type:ClusterIPselector:app:mysql-mgrports:-name:mysqlport:3306targetPort:3306应用这个~/mysql/service/mysql-mgr-services.yaml资源清单文件
kubectl apply-f~/mysql/service/mysql-mgr-services.yaml4.6.4、创建StatefulSet（使用动态PVC）
创建一个statefulset目录用于存放StatefulSet资源对象相关的资源清单文件，文件名为mysql-mgr-statefulset.yaml，路径为~/mysql/statefulset/mysql-mgr-statefulset.yaml,执行如下命令：
mkdir-p~/mysql/statefulset/&&touch~/mysql/statefulset/mysql-mgr-statefulset.yaml编辑~/mysql/statefulset/mysql-mgr-statefulset.yaml，内容如下：
apiVersion:apps/v1kind:StatefulSetmetadata:name:mysql-mgrnamespace:mysql-mgr-namespacespec:serviceName:mysql-mgr-headlessreplicas:3selector:matchLabels:app:mysql-mgrtemplate:metadata:labels:app:mysql-mgrspec:containers:-name:mysqlimage:mysql:8.0env:-name:MYSQL_ROOT_PASSWORDvalue:"Root@123456"# 请修改为强密码-name:MYSQL_REPL_PASSWORDvalue:"Repl@123456"# 复制用户密码command:-/bin/bash-/scripts/init-mgr.shports:-containerPort:3306name:mysql-containerPort:33061name:mgrvolumeMounts:-name:datamountPath:/var/lib/mysql-name:confmountPath:/etc/mysql-name:scriptsmountPath:/scripts-name:configmountPath:/etc/mysql/my.cnf.templatesubPath:my.cnf.templatereadinessProbe:exec:command:-sh--c-"mysqladmin ping -uroot -p${MYSQL_ROOT_PASSWORD}"initialDelaySeconds:30periodSeconds:10livenessProbe:exec:command:-sh--c-"mysqladmin ping -uroot -p${MYSQL_ROOT_PASSWORD}"initialDelaySeconds:60periodSeconds:30volumes:-name:confemptyDir:{}-name:scriptsconfigMap:name:mysql-mgr-configdefaultMode:0755-name:configconfigMap:name:mysql-mgr-configitems:-key:my.cnfpath:my.cnf.templatevolumeClaimTemplates:-metadata:name:dataspec:accessModes:["ReadWriteOnce"]storageClassName:"mysql-nfs-storage"# 动态 StorageClassresources:requests:storage:20Gi提交应用这个~/mysql/statefulset/mysql-mgr-statefulset.yamlStatefulSet
kubectl apply-f~/mysql/statefulset/mysql-mgr-statefulset.yaml五、观察部署进度
5.1、实时查看Pod状态
kubectl get pods-nmysql-mgr-namespace-w
