Microservices Canary Deployments with AWS X-Ray on EKS for OmniTech Solutions

Microservices Canary Deployments with AWS X-Ray on EKS for OmniTech Solutions

AMJ Cloud Technologies deployed canary deployments for User Management and Notification microservices (V1 and V2) on Amazon Elastic Kubernetes Service (EKS) for OmniTech Solutions, an e-commerce company. Using AWS X-Ray for tracing and Kubernetes for traffic management, we achieved a 50% traffic split between Notification Service V1 and V2, ensuring safe rollouts. The solution integrated AWS RDS for data persistence, AWS SES for notifications, AWS Load Balancer Controller for ALB Ingress, and External DNS for Route 53, accessible at canarydemo.omnitechsol.com. This approach minimized deployment risks and enhanced platform stability.

Introduction to Canary Deployments

Canary deployments enable gradual rollouts by distributing traffic between application versions based on replica counts. AWS X-Ray traces requests to monitor performance across versions. For OmniTech, we implemented:

• User Management Service: Manages user accounts, integrated with RDS, calling Notification Service V1 and V2.
• Notification Service (V1 and V2): Sends email notifications via SES, with V2 offering enhanced features.
• AWS X-Ray: Provides service maps and traces for latency and error analysis.
• Traffic Distribution: Configured 2 replicas each for V1 and V2, achieving a 50% traffic split.

Use Case: The User Management Service’s getNotificationAppInfo endpoint calls the Notification Service’s notification-xray endpoint, with traffic evenly split between V1 and V2, monitored via AWS X-Ray.

Traffic Split Options

The following table shows how traffic is distributed between Notification Service V1 and V2 based on replica counts:

NS V1 Replicas	NS V2 Replicas	Traffic to V2
4	0	0%
3	1	25%
2	2	50%
1	3	75%
0	4	100%

For this project, we used 2 replicas each for a 50% split.

Project Overview

OmniTech Solutions needed a strategy to deploy Notification Service V2 without disrupting their e-commerce platform. AMJ implemented canary deployments to:

• Split traffic 50% between V1 and V2 using Kubernetes replicas.
• Monitor performance with AWS X-Ray traces and service maps.
• Integrate with RDS, SES, and ALB Ingress for seamless operation.

The solution reduced deployment downtime by 60% and improved rollout reliability.

Technical Implementation

Prerequisites - AWS RDS, ALB Ingress Controller, External DNS, and X-Ray Daemon

• AWS RDS Database:
  • Used existing MySQL RDS instance (userdb-prod.kj9zxcv2qwe3.us-east-1.rds.amazonaws.com).
  • Configured ExternalName service (MySQL-externalName-Service.yml).
• ALB Ingress Controller:
  • Installed AWS Load Balancer Controller (v2.8.1) on EKS cluster (omnitech-cluster, version 1.31):

    helm install load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=omnitech-cluster --set image.tag=v2.8.1

• External DNS:
  • Installed for Route 53 DNS registration:

    helm install external-dns external-dns/external-dns -n kube-system --set provider=aws --set aws.region=us-east-1

• X-Ray Daemon:
  • Ensured X-Ray DaemonSet running from prior setup.
• Verification:
  • Checked pods:

    kubectl get pods -n kube-system
    kubectl get pods

Configure SES SMTP Credentials

• Created SES SMTP credentials:
  • Navigated to AWS SES → SMTP Settings → Create My SMTP Credentials.
  • Named IAM user omnitech-canary-ses.
  • Downloaded credentials and updated NotificationMicroservice-V1-Deployment.yml and NotificationMicroservice-V2-Deployment.yml:

    - name: AWS_MAIL_SERVER_USERNAME
      value: "<SES-USERNAME>"
    - name: AWS_MAIL_SERVER_PASSWORD
      value: "<SES-PASSWORD>"
    - name: AWS_MAIL_SERVER_FROM_ADDRESS
      value: "<from-email>"

• Verified email addresses in SES:
  • From Address: <from-email>.
  • To Address: <to-email>.
  • Confirmed via email verification links.

Deploy MySQL ExternalName Service

• Manifest (MySQL-externalName-Service.yml):

  apiVersion: v1
  kind: Service
  metadata:
    name: mysql
  spec:
    type: ExternalName
    externalName: userdb-prod.kj9zxcv2qwe3.us-east-1.rds.amazonaws.com

Deploy User Management Microservice

• Manifest (UserManagementMicroservice-Deployment.yml):

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: usermgmt-microservice
    labels:
      app: usermgmt-restapp
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: usermgmt-restapp
    template:
      metadata:
        labels:
          app: usermgmt-restapp
      spec:
        initContainers:
          - name: init-db
            image: busybox:1.31
            command:
              - sh
              - -c
              - 'echo -e "Checking MySQL availability"; while ! nc -z mysql 3306; do sleep 1; printf "-"; done; echo -e "  >> MySQL ready";'
        containers:
          - name: usermgmt-restapp
            image: omnitech/kube-usermanagement-service:4.0.0-XRay-MySQL
            ports:
              - containerPort: 8095
            imagePullPolicy: Always
            env:
              - name: DB_HOSTNAME
                value: "mysql"
              - name: DB_PORT
                value: "3306"
              - name: DB_NAME
                value: "usermgmt"
              - name: DB_USERNAME
                value: "dbadmin"
              - name: DB_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: mysql-db-password
                    key: db-password
              - name: NOTIFICATION_SERVICE_HOST
                value: "notification-clusterip-service"
              - name: NOTIFICATION_SERVICE_PORT
                value: "8096"
              - name: AWS_XRAY_TRACING_NAME
                value: "User-Management-Microservice"
              - name: AWS_XRAY_DAEMON_ADDRESS
                value: "xray-service.default:2000"
              - name: AWS_XRAY_CONTEXT_MISSING
                value: "LOG_ERROR"
            livenessProbe:
              exec:
                command:
                  - /bin/sh
                  - -c
                  - nc -z localhost 8095
              initialDelaySeconds: 60
              periodSeconds: 10
            readinessProbe:
              httpGet:
                path: /usermgmt/health-status
                port: 8095
              initialDelaySeconds: 60
              periodSeconds: 10
  ---
  apiVersion: v1
  kind: Secret
  metadata:
    name: mysql-db-password
  type: Opaque
  data:
    db-password: c2VjdXJlcGFzczIwMjU=

Deploy User Management NodePort Service

• Manifest (UserManagement-NodePort-Service.yml):

  apiVersion: v1
  kind: Service
  metadata:
    name: usermgmt-restapp-nodeport-service
    labels:
      app: usermgmt-restapp
    annotations:
      alb.ingress.kubernetes.io/healthcheck-path: /usermgmt/health-status
  spec:
    type: NodePort
    selector:
      app: usermgmt-restapp
    ports:
      - port: 8095
        targetPort: 8095

Deploy Notification Microservice V1

• Manifest (NotificationMicroservice-V1-Deployment.yml):

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: v1-notification-microservice
    labels:
      app: notification-restapp
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: notification-restapp
    template:
      metadata:
        labels:
          app: notification-restapp
      spec:
        containers:
          - name: notification-service
            image: omnitech/kube-notification-service:5.0.0-XRay
            ports:
              - containerPort: 8096
            imagePullPolicy: Always
            env:
              - name: AWS_MAIL_SERVER_HOST
                value: "smtp-service"
              - name: AWS_MAIL_SERVER_USERNAME
                value: "<SES-USERNAME>"
              - name: AWS_MAIL_SERVER_PASSWORD
                value: "<SES-PASSWORD>"
              - name: AWS_MAIL_SERVER_FROM_ADDRESS
                value: "<from-email>"
              - name: AWS_XRAY_TRACING_NAME
                value: "V1-Notification-Microservice"
              - name: AWS_XRAY_DAEMON_ADDRESS
                value: "xray-service.default:2000"
              - name: AWS_XRAY_CONTEXT_MISSING
                value: "LOG_ERROR"

Deploy Notification Microservice V2

• Manifest (NotificationMicroservice-V2-Deployment.yml):

  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: v2-notification-microservice
    labels:
      app: notification-restapp
  spec:
    replicas: 2
    selector:
      matchLabels:
        app: notification-restapp
    template:
      metadata:
        labels:
          app: notification-restapp
      spec:
        containers:
          - name: notification-service
            image: omnitech/kube-notification-service:6.0.0-XRay
            ports:
              - containerPort: 8096
            imagePullPolicy: Always
            env:
              - name: AWS_MAIL_SERVER_HOST
                value: "smtp-service"
              - name: AWS_MAIL_SERVER_USERNAME
                value: "<SES-USERNAME>"
              - name: AWS_MAIL_SERVER_PASSWORD
                value: "<SES-PASSWORD>"
              - name: AWS_MAIL_SERVER_FROM_ADDRESS
                value: "<from-email>"
              - name: AWS_XRAY_TRACING_NAME
                value: "V2-Notification-Microservice"
              - name: AWS_XRAY_DAEMON_ADDRESS
                value: "xray-service.default:2000"
              - name: AWS_XRAY_CONTEXT_MISSING
                value: "LOG_ERROR"

Deploy Notification SMTP ExternalName Service

• Manifest (Notification-SMTP-Service.yml):

  apiVersion: v1
  kind: Service
  metadata:
    name: smtp-service
  spec:
    type: ExternalName
    externalName: email-smtp.us-east-1.amazonaws.com

Deploy Notification ClusterIP Service

• Manifest (NotificationMicroservice-ClusterIP-Service.yml):

  apiVersion: v1
  kind: Service
  metadata:
    name: notification-clusterip-service
    labels:
      app: notification-restapp
  spec:
    type: ClusterIP
    selector:
      app: notification-restapp
    ports:
      - port: 8096
        targetPort: 8096

Deploy ALB Ingress Service

• Manifest (ALB-Ingress-SSL-Redirect.yml):

  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: eks-microservices-canary-demo
    labels:
      app: usermgmt-restapp
      runon: fargate
    namespace: default
    annotations:
      alb.ingress.kubernetes.io/load-balancer-name: eks-microservices-canary-demo
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
      alb.ingress.kubernetes.io/healthcheck-port: traffic-port
      alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
      alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
      alb.ingress.kubernetes.io/success-codes: "200"
      alb.ingress.kubernetes.io/healthy-threshold-count: "2"
      alb.ingress.kubernetes.io/unhealthy-threshold-count: "2"
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:<account-id>:certificate/<certificate-id>
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      external-dns.alpha.kubernetes.io/hostname: canarydemo.omnitechsol.com
  spec:
    ingressClassName: my-aws-ingress-class
    rules:
      - http:
          paths:
            - path: /
              pathType: Prefix
              backend:
                service:
                  name: usermgmt-restapp-nodeport-service
                  port:
                    number: 8095

Deploy Microservices Manifests

• Deployed all manifests:

  kubectl apply -f Microservices/

• Verified deployment:

  kubectl get pods,svc,ingress

Test Canary Deployment and Tracing

• Tested the application:

  curl https://canarydemo.omnitechsol.com/usermgmt/notification-xray

• Verified tracing in AWS X-Ray console:
  • Checked service map for interactions between User Management, Notification V1, and V2.
  • Confirmed 50% traffic split by analyzing trace distribution.
• Tested user creation via Postman, verifying email notifications to <to-email> and traces for both V1 and V2 in X-Ray.

Traffic Distribution Mechanism

The Notification ClusterIP Service uses a selector (app: notification-restapp) matching both V1 and V2 deployments’ matchLabels. Kubernetes distributes traffic based on replica counts (2 for V1, 2 for V2), achieving a 50% split:

• ClusterIP Service:

  selector:
    app: notification-restapp

• V1/V2 Deployments:

  selector:
    matchLabels:
      app: notification-restapp

Best Practices for Canary Deployments

• Istio: Service mesh for weight-based and header-based routing.
• AWS App Mesh: AWS-managed service mesh for EKS, simplifying traffic management.
• Flagger: Automates canary rollouts with metrics-driven progression (e.g., Prometheus).
• Observability Tools: Use AWS CloudWatch or Grafana for enhanced monitoring.

Technical Highlights

• Canary Deployment: Achieved 50% traffic split between Notification V1 and V2 using Kubernetes replicas.
• AWS X-Ray Tracing: Monitored performance across versions with service maps and traces.
• Microservices Isolation: Enabled independent scaling of V1 and V2.
• Secure Access: Used ALB Ingress with HTTPS and Route 53 DNS automation.
• Risk Reduction: Minimized deployment risks through controlled rollouts.

Client Impact

For OmniTech Solutions, this solution enabled safe deployment of Notification Service V2, reducing deployment downtime by 60% and enhancing monitoring with X-Ray. The canary approach ensured high availability during updates, improving user satisfaction.

Technologies Used

• AWS EKS
• AWS X-Ray
• AWS RDS
• AWS Simple Email Service (SES)
• AWS Load Balancer Controller
• Kubernetes Ingress
• Kubernetes DaemonSets
• External DNS
• AWS Route 53
• AWS Certificate Manager
• Docker