Skip to main content
IntroFundamentalsObservabilitySecurityNetworkingAutomationAI/MLTroubleshooting
Switch to Essentials →

Advanced troubleshooting

In this section, we will use Kiro CLI and the MCP server for Amazon EKS to troubleshoot a complex issue in the EKS cluster that would be difficult to resolve without knowledge of Kubernetes, EKS, and other AWS services.

First, let's reconfigure the carts service to use a DynamoDB table that has been created for us. The application loads most of its configurations from a ConfigMap. Let's examine the current ConfigMap:

~$kubectl -n carts get -o yaml cm carts
apiVersion: v1
data:
  AWS_ACCESS_KEY_ID: key
  AWS_SECRET_ACCESS_KEY: secret
  RETAIL_CART_PERSISTENCE_DYNAMODB_CREATE_TABLE: "true"
  RETAIL_CART_PERSISTENCE_DYNAMODB_ENDPOINT: http://carts-dynamodb:8000
  RETAIL_CART_PERSISTENCE_DYNAMODB_TABLE_NAME: Items
  RETAIL_CART_PERSISTENCE_PROVIDER: dynamodb
kind: ConfigMap
metadata:
  name: carts
  namespace: carts


We'll use the following kustomization to update the ConfigMap. This removes the DynamoDB endpoint configuration, instructing the SDK to use the real DynamoDB service instead of our test Pod. We've also configured the DynamoDB table name in environment variable RETAIL_CART_PERSISTENCE_DYNAMODB_TABLE_NAME that's already been created for us:


~/environment/eks-workshop/modules/aiml/kiro-cli/troubleshoot/dynamo/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../../../../base-application/carts
configMapGenerator:
  - name: carts
    namespace: carts
    env: config.properties
    behavior: replace
    options:
      disableNameSuffixHash: true


Let's verify the DynamoDB table name and apply the new configuration:


~$echo $CARTS_DYNAMODB_TABLENAME
eks-workshop-carts
~$kubectl kustomize ~/environment/eks-workshop/modules/aiml/kiro-cli/troubleshoot/dynamo \
  | envsubst | kubectl apply -f-


Verify the updated ConfigMap:


~$kubectl -n carts get cm carts -o yaml
apiVersion: v1
data:
  RETAIL_CART_PERSISTENCE_DYNAMODB_TABLE_NAME: eks-workshop-carts
  RETAIL_CART_PERSISTENCE_PROVIDER: dynamodb
kind: ConfigMap
metadata:
  labels:
    app: carts
  name: carts
  namespace: carts


You can see that RETAIL_CART_PERSISTENCE_DYNAMODB_TABLE_NAME attribute now points to an actual DynamoDB table in your account instead of the table within this EKS cluster.


tip
You should be able to see this DynamoDB table in your account using console under DynamodDB > Tables menu.


Now, let's redeploy the carts deployment to pick up the new ConfigMap contents:


~$kubectl rollout restart -n carts deployment/carts
deployment.apps/carts restarted
~$kubectl rollout status -n carts deployment/carts --timeout=20s
Waiting for deployment "carts" rollout to finish: 1 old replicas are pending termination...
error: timed out waiting for the condition


The deployment appears to have failed. Let's check the Pod status:


~$kubectl -n carts get pod
NAME                              READY   STATUS             RESTARTS        AGE
carts-5d486d7cf7-8qxf9            1/1     Running            0               5m49s
carts-df76875ff-7jkhr             0/1     CrashLoopBackOff   3 (36s ago)     2m2s
carts-dynamodb-698674dcc6-hw2bg   1/1     Running            0               20m


Let's use Kiro CLI to investigate this issue. Start a new Kiro CLI session:


~$kiro-cli chat


Ask Kiro CLI to help troubleshoot the issue:


I have a pod in my eks-workshop cluster that is with status CrashLoopBackOff. Troubleshoot the issue and resolve it for me.


To address the prompt, Kiro CLI will use a variety of tools from the MCP server. In addition to the tools seen in the previous example, it may also:


    

  1. Describe IAM roles and policies in the scope using get_policies_for_role tool from EKS MCP server
    2. 

  2. Fetch further information related to AWS resources with the use_aws tools that is built in to Kiro CLI
    3. 

  3. Take corrective action to solve the problem
    4. 



Follow the suggestions offered by Kiro CLI to solve this issue. In an ideal scenario, the problem should be fixed. At the end, Kiro CLI will present you a final status summary of the steps it took.


Expand for sample response
  The issue is resolved. Here's a summary:
  
  Root Cause: The carts pod (carts-6956bbbbf6-9lsd6) was crashing because it couldn't access DynamoDB. The pod's service account had no IRSA role configured, so it fell back to the node instance role (eksctl-eks-workshop-nodegroup-defa-NodeInstanceRole-nH1a0rWkr5Um), which lacked DynamoDB permissions.
  
  Fix Applied: Added an inline IAM policy CartsDynamoDBAccess to the node instance role granting the necessary DynamoDB actions (Query, Scan, GetItem, PutItem, UpdateItem, DeleteItem, DescribeTable, BatchGetItem, BatchWriteItem) on the eks-workshop-auto-carts table and its indexes.
  
  Result: The replacement pod carts-6956bbbbf6-2qsbw is now Running and Ready with zero restarts.

▸ Credits: 1.94 • Time: 3m 43s


Once you are done, enter the following command to exit Kiro CLI session.


/quit


Finally, verify that the pods are now running correctly:


~$kubectl -n carts get pod
NAME                              READY   STATUS    RESTARTS   AGE
carts-596b6f94df-q4449            1/1     Running   0          9m5s
carts-dynamodb-698fcb695f-zvzf5   1/1     Running   0          2d1h


This concludes our introduction to Kiro CLI. You've seen how this powerful tool, combined with the MCP server for EKS, can help diagnose and resolve complex issues in your EKS cluster.