Saturday, August 15, 2020

Containerizing MySQL Enterprise Monitor (MEM) on Kubernetes

 A DBA holds a big responsibility for keeping mission critical databases in the organization to be always up and running and meeting all the SLAs assigned to the database, such as availability, recovery, security, performance, and more. This is really a difficult and risky job to maintain mission critical databases without proper tools providing monitoring data. In real life, it is very important for DBAs to be equipped with good database monitoring tools to ensure they can quickly identify, resolve and fix any potential problems before those problems even impact operations.

MySQL Enterprise Monitor (MEM) is a comprehensive, centralized, real-time MySQL database monitoring tool that provides a complete range of database monitoring aspects: from availability to performance. It is able to monitor all MySQL database instances in the environment and present information in a web-based interface which visually displays a global and detailed view of all monitored databases, including MySQL cluster (NDB Cluster and InnoDB Cluster), replication, backup, all security features on MySQL Enterprise Edition, etc. More information about MEM can be found here: https://dev.mysql.com/doc/mysql-monitor/8.0/en/

 

This article discusses on how to run the MEM on Kubernetes to monitor closely all MySQL Enterprise Edition database that running on the same Kubernetes cluster. We will start by creating a docker image from scratch. (Disclaimer: Please note, this is unofficial and DIY way to create docker image for MEM. If you require your MEM to run on container, you can follow below steps, test those, but support is by your own in case of issues). 

 

1. Obtain MySQL Enterprise Monitor 8.0.21 binary 

Download mysql-enterprise-monitor-8.0.21 for Linux x86_64 from edelivery.oracle.com (file name: V999555-01.zip). Unzip V999555-01.zip to get installation file: mysqlmonitor-8.0.21.1248-linux-x86_64-installer.bin

  

 2. Prepare and run docker-build to build image


Create Dockerfile on the same directory: 

FROM oraclelinux:7-slim

 

RUN groupadd mysql

RUN useradd -g mysql mysql

 

RUN mkdir /mem

 

COPY mysqlmonitor-8.0.21.1248-linux-x86_64-installer.bin docker-entrypoint.sh /mem/

WORKDIR /mem

 

RUN chmod u+x /mem/docker-entrypoint.sh

RUN yum install -y libaio

RUN yum install -y numactl-libs

RUN yum install -y hostname

RUN yum install -y procps-ng

 

ENTRYPOINT ["/mem/docker-entrypoint.sh"]

Create docker-entrypoint.sh file on the same directory: 

#! /bin/bash

 

if [ "$ADMINUSER" = "" ]; then

   ADMINUSER="admin"

fi

 

if [ "$ADMINPASSWORD" = "" ]; then

   ADMINPASSWORD="admin"

fi;

 

if [ "$SYSTEMSIZE" = "" ]; then

   SYSTEMSIZE="small"

fi;

 

echo "Admin user is set to $ADMINUSER"

echo "Admin password is set to $ADMINPASSWORD"

echo "System size is set to $SYSTEMSIZE"

 

if [ `ls -l /opt/mysql/enterprise/monitor | grep total | awk '{print $2}'` -eq 0 ]; then

 

       echo "Install new Service Manager"

       /mem/mysqlmonitor-8.0.21.1248-linux-x86_64-installer.bin --mode unattended --system_size $SYSTEMSIZE --adminuser $ADMINUSER --adminpassword $ADMINPASSWORD --mysql_installation_type bundled

 

       /opt/mysql/enterprise/monitor/mysqlmonitorctl.sh status

 

fi;

 

while true; do

       echo "Checking if MySQL is Up"

       if [ `/opt/mysql/enterprise/monitor/mysqlmonitorctl.sh status mysql | grep "not running" | wc -l` -gt 0 ]; then

             echo "MySQL is not running, starting up"

             /opt/mysql/enterprise/monitor/mysqlmonitorctl.sh start mysql

       else

             echo "MySQL is already running"

       fi       

 

       echo "Checking if Tomcat is Up"

       if [ `/opt/mysql/enterprise/monitor/mysqlmonitorctl.sh status tomcat | grep "not running" | wc -l` -gt 0 ]; then

             echo "Tomcat is not running, starting up"

             /opt/mysql/enterprise/monitor/mysqlmonitorctl.sh start tomcat

       else

             echo "Tomcat is already running"

       fi

      

       /opt/mysql/enterprise/monitor/mysqlmonitorctl.sh status

       sleep 15;

done; 

Check our installed docker images: 

$ docker images

REPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZE

mysql/enterprise-server                                          8.0                 106673d79487        8 weeks ago         452MB

mysql/mysql-cluster                                              latest              ca2cf722118d        3 months ago        415MB

k8s.gcr.io/kube-proxy                                            v1.16.2             8454cbe08dc9        10 months ago       86.1MB

k8s.gcr.io/kube-apiserver                                        v1.16.2             c2c9a0406787        10 months ago       217MB

k8s.gcr.io/kube-controller-manager                               v1.16.2             6e4bffa46d70        10 months ago       163MB

k8s.gcr.io/kube-scheduler                                        v1.16.2             ebac1ae204a2        10 months ago       87.3MB

quay.io/kubernetes-ingress-controller/nginx-ingress-controller   0.26.1              29024c9c6e70        10 months ago       483MB

k8s.gcr.io/etcd                                                  3.3.15-0            b2756210eeab        11 months ago       247MB

kubernetesui/dashboard                                           v2.0.0-beta4        6802d83967b9        11 months ago       84MB

k8s.gcr.io/coredns                                               1.6.2               bf261d157914        12 months ago       44.1MB

k8s.gcr.io/kube-addon-manager                                    v9.0.2              bd12a212f9dc        12 months ago       83.1MB

kubernetesui/metrics-scraper                                     v1.0.1              709901356c11        13 months ago       40.1MB

k8s.gcr.io/kube-addon-manager                                    v9.0                119701e77cbc        19 months ago       83.1MB

k8s.gcr.io/kubernetes-dashboard-amd64                            v1.10.1             f9aed6605b81        20 months ago       122MB

k8s.gcr.io/k8s-dns-sidecar-amd64                                 1.14.13             4b2e93f0133d        23 months ago       42.9MB

k8s.gcr.io/k8s-dns-kube-dns-amd64                                1.14.13             55a3c5209c5e        23 months ago       51.2MB

k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64                           1.14.13             6dc8ef8287d3        23 months ago       41.4MB

k8s.gcr.io/pause                                                 3.1                 da86e6ba6ca1        2 years ago         742kB

gcr.io/k8s-minikube/storage-provisioner                          v1.8.1              4689081edb10        2 years ago         80.8MB

Execute the following command to build the image: 

$ set -x

$ set -e

$ docker build -m 2g -c 4 -t private/mem:8.0.21 .

Check again our installed docker images: 

$ docker images

REPOSITORY                                                       TAG                 IMAGE ID            CREATED             SIZE

private/mem                                                      8.0.21              57ddbccb6758        39 seconds ago      1.74GB

oraclelinux                                                      7-slim              153f8d73287e        4 weeks ago         131MB

mysql/enterprise-server                                          8.0                 106673d79487        8 weeks ago         452MB

mysql/mysql-cluster                                              latest              ca2cf722118d        3 months ago        415MB

k8s.gcr.io/kube-proxy                                            v1.16.2             8454cbe08dc9        10 months ago       86.1MB

k8s.gcr.io/kube-controller-manager                               v1.16.2             6e4bffa46d70        10 months ago       163MB

k8s.gcr.io/kube-scheduler                                        v1.16.2             ebac1ae204a2        10 months ago       87.3MB

k8s.gcr.io/kube-apiserver                                        v1.16.2             c2c9a0406787        10 months ago       217MB

quay.io/kubernetes-ingress-controller/nginx-ingress-controller   0.26.1              29024c9c6e70        10 months ago       483MB

k8s.gcr.io/etcd                                                  3.3.15-0            b2756210eeab        11 months ago       247MB

kubernetesui/dashboard                                           v2.0.0-beta4        6802d83967b9        11 months ago       84MB

k8s.gcr.io/coredns                                               1.6.2               bf261d157914        12 months ago       44.1MB

k8s.gcr.io/kube-addon-manager                                    v9.0.2              bd12a212f9dc        12 months ago       83.1MB

kubernetesui/metrics-scraper                                     v1.0.1              709901356c11        13 months ago       40.1MB

k8s.gcr.io/kube-addon-manager                                    v9.0                119701e77cbc        19 months ago       83.1MB

k8s.gcr.io/kubernetes-dashboard-amd64                            v1.10.1             f9aed6605b81        20 months ago       122MB

k8s.gcr.io/k8s-dns-sidecar-amd64                                 1.14.13             4b2e93f0133d        23 months ago       42.9MB

k8s.gcr.io/k8s-dns-kube-dns-amd64                                1.14.13             55a3c5209c5e        23 months ago       51.2MB

k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64                           1.14.13             6dc8ef8287d3        23 months ago       41.4MB

k8s.gcr.io/pause                                                 3.1                 da86e6ba6ca1        2 years ago         742kB

gcr.io/k8s-minikube/storage-provisioner                          v1.8.1              4689081edb10        2 years ago         80.8MB

 

3.  Prepare YAML

Deploying MEM on Kubernetes requires the following sample YAML (mem.yaml).  Please note the pod needs to run as privileged container, or else tomcat service will fail to start. We will configure db repository with user memadmin:manager. MEM will be for small number of monitored database instances (up to 5 - 10 databases). If there’re more number of instances, consider to change SYSTEMSIZE parameter from “small” to “medium” (up to 100 databases) or “large” (more than 100 databases).

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

  name: mem

  namespace: mysql-cluster

spec:

  replicas: 1

  serviceName: mem

  selector:

    matchLabels:

      app: mem

  template:

    metadata:

      labels:

        app: mem

    spec:

     containers:

     - image: private/mem:8.0.21

       name: mem

       env:

         - name: ADMINUSER

           value: memadmin

         - name: ADMINPASSWORD

           value: manager

         - name: SYSTEMSIZE

           value: small

       resources:

         limits:

           cpu: "2"

           memory: "4096Mi"

         requests:

           cpu: "2"

           memory: "2048Mi"

       volumeMounts:

         - name: mem-vol

           mountPath: /opt/mysql/enterprise/monitor

       securityContext:

         privileged: true

  volumeClaimTemplates:                               

  - metadata:

      name: mem-vol                                  

    spec:

      accessModes: [ "ReadWriteOnce" ]                

      resources:

        requests:

          storage: 1Gi

---


4. Deploy MEM

 

Deploy the YAML file to Kubernetes

 

$ kubectl apply -f mem.yaml

 

Check Pod Status

$ kubectl -n mysql-cluster get pod

NAME    READY   STATUS    RESTARTS   AGE

mem-0     1/1           Running      0                   3s 

 

Wait for 1 min to give time for MEM configuration process. Then, check the status of our MEM on Kubernetes

 

$ kubectl -n mysql-cluster exec -it mem-0 -- /opt/mysql/enterprise/monitor/mysqlmonitorctl.sh status

MySQL Enterprise MySQL is running

MySQL Enterprise Tomcat is running


Both MySQL instance and Tomcat are running!


5. Access our MEM

In this article, I just simply used port-forward to test if my MEM is running:

 

$ kubectl -n mysql-cluster port-forward pods/mem-0 18443:18443


Then, I opened my web browser and accessed https://127.0.0.1:18443

The MEM initial setup page appeared after few minutes.

 

It's working !

 

No comments:

Post a Comment