一 实现方案

RabbitMQ Cluster Kubernetes Operator是一个开源的Kubernetes Operator，提供了一种一致且简便的方法，将RabbitMQ集群部署到Kubernetes并运行它们，包括"第二天"(连续）操作，它可以自动的执行Kubernetes上运行的RabbitMQ集群的配置并对集群进行管理和操作。

1.1 安装限制

• Kubernetes 1.16+
• RabbitMQ Docker镜像版本 3.8.8+

二 实现过程

2.1 安装

（1）项目结构

├─scripts
  ├─rabbitmq-cluster-operator.yml
  ├─rabbitmq-definition.yaml
├─software
  ├─rabbitmq_delayed_message_exchange-3.8.9-0199d11c.ez
├─Dockerfile

（2）Dockerfile

由于通过rabbitmq-cluster-operator方式安装，没有找到plugins的安装方式，为此通过新建Docker镜像方式安装rabbitmq_delayed_message_exchange插件

FROM rabbitmq:3.8.9-management

MAINTAINER will

ADD rabbitmq_delayed_message_exchange-3.8.9-0199d11c.ez /opt/rabbitmq/plugins/

RUN set -x \
    && chown -R rabbitmq:rabbitmq /opt/rabbitmq/plugins/rabbitmq_delayed_message_exchange-3.8.9-0199d11c.ez \
    && chmod 775 /opt/rabbitmq/plugins/rabbitmq_delayed_message_exchange-3.8.9-0199d11c.ez \
    && rabbitmq-plugins enable rabbitmq_delayed_message_exchange \

(3) 构建镜像及上传

docker build -t harbor-sit.wxt.com/public/rabbitmq:3.8.9 . -f Dockerfile
docker push  harbor-sit.wxt.com/public/rabbitmq:3.8.9

注意在k8s的node节点手动pull harbor-sit.wxt.com/public/rabbitmq:3.8.9镜像，否则在kubeclt部署时会出现无法pull镜像问题（harbor其他项目是可正常pull）。

（4）下载配置文件

进入github选择k8s匹配版本cluster-operator，并将文件重命名为：rabbitmq-cluster-operator.yml

https://github.com/rabbitmq/cluster-operator/tree/0.49.0

注意cluster-operator与k8s版本的匹配关系，cluster-operator-0.49.0支持Kubernetes 1.16 or newer

（5）安装cluster-operator

可按照我们自己的需求修改相关配置，如namespace等

kubectl apply -f rabbitmq-cluster-operator.yml

输出信息：

customresourcedefinition.apiextensions.k8s.io/rabbitmqclusters.rabbitmq.com created
serviceaccount/rabbitmq-cluster-operator created
role.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-role created
clusterrole.rbac.authorization.k8s.io/rabbitmq-cluster-operator-role created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-operator-rolebinding created
deployment.apps/rabbitmq-cluster-operator created

（4）配置MQ实例

新建 rabbitmq-definition.yaml文件，进行如下配置：

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq #服务名称
  namespace: yrz-sit-software
spec:
  image: harbor-sit.wxt.com/public/rabbitmq:3.8.9
  replicas: 3 #副本数
  service:
    type: NodePort # 服务类型
  persistence:
    storageClassName: nfs-client #存储类的名称
    storage: 5Gi
  resources:
    requests:
      cpu: 200m
      memory: 400Mi
    limits:
      cpu: 300m
      memory: 600Mi
  rabbitmq:
    additionalPlugins:
      - rabbitmq_delayed_message_exchange   # 启用插件

注意：resources根据具体环境配置

(5) 部署MQ实例

kubectl apply -f rabbitmq-definition.yaml

获取：

kubectl get all -l app.kubernetes.io/name=rabbitmq -n yrz-sit-software

结果：

NAME                    READY   STATUS    RESTARTS   AGE
pod/rabbitmq-server-0   1/1     Running   0          112s
pod/rabbitmq-server-1   1/1     Running   0          80s
pod/rabbitmq-server-2   1/1     Running   0          54s

NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
service/rabbitmq         NodePort    10.68.125.158   <none>        5672:36939/TCP,15672:29706/TCP   112s
service/rabbitmq-nodes   ClusterIP   None            <none>        4369/TCP,25672/TCP               112s

NAME                               READY   AGE
statefulset.apps/rabbitmq-server   3/3     112s

（6）修改svc默认端口

根据实际情况决定是否修改， nodePort: 36939修改为nodePort: 20749，nodePort: 29706修改为nodePort: 20057

kubectl edit svc/rabbitmq -n yrz-sit-software -o yaml --save-config

spec:
  clusterIP: 10.68.125.158
  externalTrafficPolicy: Cluster
  ports:
  - name: amqp
    nodePort: 20749
    port: 5672
    protocol: TCP
    targetPort: 5672
  - name: management
    nodePort: 20057
    port: 15672
    protocol: TCP
    targetPort: 15672
  selector:
    app.kubernetes.io/name: rabbitmq
  sessionAffinity: None
  type: NodePort

（7）获取账号和密码

kubectl -n yrz-sit-software get secret rabbitmq-default-user  -o jsonpath="{.data.username}" | base64 --decode
kubectl -n yrz-sit-software get secret rabbitmq-default-user -o jsonpath="{.data.password}" | base64 --decode

三 集群健康检查

# 1. rabbitmq访问路径
http://10.168.xx.xx:20057

# 用户名/密码
Gfz7siXNrPBCu24T_bCQeOhp2TZWqKk3/6itjHADQz3D0SvVOVhELcJLhGgYZjSaw

# 2 获取域名 rabbitmq StatefulSet
[root@yrz-k8s-master01 rabbitmq]# for i in 0 1 2; do kubectl exec -n yrz-sit-software rabbitmq-server-$i -- hostname -f; done

rabbitmq-server-0.rabbitmq-nodes.yrz-sit-software.svc.cluster.local.
rabbitmq-server-1.rabbitmq-nodes.yrz-sit-software.svc.cluster.local.
rabbitmq-server-2.rabbitmq-nodes.yrz-sit-software.svc.cluster.local.

四 问题记录

# 出现无法删除crd
kubectl delete crd rabbitmqclusters.rabbitmq.com

# 解决
kubectl patch crd/rabbitmqclusters.rabbitmq.com -p '{"metadata":{"finalizers":[]}}' --type=merge

---

欢迎关注公众号： 编码是个技术活