Category

AWS

API GATEWAY part 1-AWS API GATEWAY Monitoring and Authentication

By | AWS, Blogs | No Comments

Written by Mudita Misra, Cloud Engineer, Powerupcloud Technologies.

Authentication and Monitoring are the MUST DO steps for all services in today’s scenario.

In this blog, we are going to show how authentication can be enabled for AWS APIs as well as the usage monitoring.

API KEYS:

Why we should use API keys?

Before implementing something we should understand the use case well.

USE CASE: If the API is strictly for one’s own application’s consumption then API Keys aren’t really going to do much, but If the API is going to be consumed by 3rd parties, then API Keys are a mechanism that will allow to monitor each 3rd party consumption of the API.

Let’s do some hands-on with AWS API:

Let us begin the API gateway setup:

  1. Create an API Key.
  2. Associate an API Key to the API.
  3. Configure the API to require API Keys.

Step 1. Create an API Key

  1. Launch the AWS API Gateway Console.
  2. Here we have created one example API from AWS console: PetStore

3. From the API menu select the API Keys menu.

4. Click the Create API Key button.

5. Provide a Name and Description for the API.

6. We have given mention below values:

  • Name: mudita-key
  • Description: API Key for user microservice

7. Choose Auto Generate or Custom key

  • Auto Generate: a key will be generated by AWS and you will get from the console after API key creation.
  • Custom key: You can give any key of your choice in the below box.

8. Click Save.

9. Get the key from console now by clicking on Show.

Step 2. Associate the API Key to an API Stage

  1. We will be able to see the key created above, here we have mudita-key. Now we have to create a usage plan for API Monitoring.

2. Create your Usage Plan:

  • Click on the Usage plan from the left pane.
  • Click on create then we will be navigated to a page like this:
  • Name the plan, add a description.
  • According to the plan for the API usage, mention the Throttling and Quota values.
  • Then save it, the usage plan is ready.

Now come back to the API Keys section.

3. Click the Add to usage plan button and add the API and stage to the key.

4. Verify that the API/State association now shows up the Staged Enabled section of the page.

Step 3. Configure the API to Require API Keys

  1. From the AWS API Gateway Console select the API menu to get back to the API settings page.
  2. Select the PetStore API.
  3. From the PetStore Methods page, note that the API Key setting is set to Not Required.

4. Click the POST method.

5. From the POST Method Execution page, click the Method Request link.

6. Select true from the API Key Required to drop down and click the tick mark.

7. In the POST section, the API KEY will be changed from NONE to Required.

And that’s it. This is the process of creating an API Key, associate the API Key to an API/stage, and configuring the API to require API Keys.

8. Once we deploy the updated API all clients who call our API will require a valid API Key to access it. Specifically, clients will need to include a new HTTP header, “x-API-key” with the value that is the API key.

Deploy the API again !!

Let’s deploy the updates made to our API so we can do some testing to prove this change works.

9. Now when hitting the API URL, here the URL is below:

https://bbmagiuhh0.execute-api.us-east-2.amazonaws.com/stage1

We will get the “message: forbidden” on the page due to the API key required setup.

The output from Postman:

10. Let us add the API KEY to get the response from API.

In the HEADER section add

key: x-API-key

value: <the API key>

Save and click Send.

All set with API key authentication for the API.

Now we will create the Usage data for the API. Click on the Usage plan, then click on API keys as shown below:

Click on Usage and Extension you will get the detail graphs as below for you API:

We can increase or decrease the requests per month from the Extension tab.

The next method is IAM authentication via access key and secret key.

IAM AUTHENTICATION:

  1. Create an IAM group with the permission AmazonAPIGatewayInvokeFullAccess and then create add the users to permit for accessing the API URL in the group.
  2. We will use the IAM user keys later to access the API URL, save the access and secret key safely.
  3. Now go to API console, select API and choose resources under it. Then choose the POST and choose the Method Request.

4. In the settings section for Authentication choose AWS_IAM and click the tick mark and deploy your API again.

5. Now choose the stage we created now and get the section for /pets/{pet ID}, get the URL.

URL: https://bbmagiuhh0.execute-api.us-east-2.amazonaws.com/stage1/pets/{petId}

6. Change the petID with any number between 1–555 (according to the example API)

7. Hit the URL from the postman and you will get the “missing authentication token” due to Authentication set to AWS_IAM now.

8. Now we will use the IAM credentials: Access key and Secret keys, we created above and have saved safely. We can download the CSV credential files also.

9. Give the IAM credentials in the postman to get the page via authentication:

Click on Authorization choose TYPE: AWS Signature

Give the Access key and Secret key.

10. Save it and send again. Get the response page now.

All set now the API URL is secure with IAM Authentication.

Hope this blog post will give an idea to start the API. Please comment below the queries and reviews.

We will be showing how to do Private integration with API in our next upcoming blog post.

Securing AWS Elasticsearch Service by enabling Cognito Authentication

By | AWS, Blogs | No Comments

Written by Priyanka Sharma, DevOps Architect, Powerupcloud Technologies.

In this blog, we are discussing on enabling the Authentication for Elasticsearch / Kibana. Recently, AWS announced the Cognito Authentication support for Elasticsearch (link in the References section below). We have implemented the same thing in our scenario too. This article will cover how one can access the Kibana endpoint and elastic search endpoint securely which exists in private subnet.

Go to AWS Cognito. Create a User pool.

Review defaults and creates a pool.

On the same page, now go to the domain and provide a domain name.

Also, you can put a logo in UI customization.

Switch to the Federated Identities on top to create a Cognito Identity Pool. Provide a name and select “Enable access to unauthenticated identities”.

Allow the IAM Role to be created on the next screen and create the Identity pool.

Now go to your existing Elasticsearch Domain. Configure Cluster. Select “Enable Amazon Cognito for Authentication”.

Select the pools i.e. UserPool and IdentityPool created in the above step. Submit.

Now modify Access Policy as mentioned below. Give IAM Role created by Cognito service fr Authenticated Users:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::xxxxxx:role/Cognito_xxxxx_es_id_poolAuth_Role"
]
},
"Action": "es:*",
"Resource": "arn:aws:es:ap-south-1:xxxxxx:domain/dev-k8s-es/*"
}
]
}

Once the Elasticsearch domain is modified with the above settings, Hit the Kibana URL:

Cognito authentication will be enabled for your Kibana Endpoint. The users created in the Cognito UserPool will be able to access the endpoint.

Till here we have secured the kibana endpoint. But sometimes we have to access the Elasticsearch endpoint too for example,

  • If we want to list the existing indices, we can execute a curl command with GET request to view the indices available.
  • Similarly, if we want to delete any index, it’s easy to execute a curl command with DELETE request.
  • Even if you want to put some data or mappings, a curl command with POST request will be useful in this case.

In our case, we have provisioned an EC2 server with IAM Instance Profile attached to access the Elasticsearch Endpoint. Here we are discussing two scenarios now:

  • Accessing the Elasticsearch domain without Cognito Authentication Enabled

A basic curl command will work in this case. In the below screenshot, we are executing a basic curl command to list the existing indices and here is the output:

  • Accessing the Elasticsearch domain with Cognito Authentication Enabled

Similar curl command here will give the output as:

Once we enable Cognito Authentication, the Elasticsearch will only support signed requests with AWS Signature Version 4. On this note, we are using aws-es-curl utility (link in the References section below) to execute the curl commands on our Cognito Authentication enabled Elasticsearch Domain. You can use any other curl like utility which supports AWS Sig4 requests.

Hope you found it useful.

References:

Terraform – Getting started

By | AWS, Blogs | No Comments

Written by Vasudha N, Cloud Engineer & Navya M, Cloud Engineer, Powerupcloud Technologies.

Infrastructure as a code is increasingly becoming popular with the demand of Automation, a search over the internet will fetch us a list of open source tools one such popular tool is Terraform.

Terraform is an open-source simple powerful tool, that allows you to create, manage and improve IT infrastructures. It can be shared among members of a team and treated exactly as code, so it can be easily changed, reviewed, versioned.

Advantages of Terraform:

  • Terraform is a cross-platform tool it can interact with many like AWS, GCE, VMware, OpenStack, and Azure.
  • Terraform can be Integrated with third-party solutions like Github, PagerDuty, and Consul.
  • Terraform gives us the possibility of planning any modification and control changes previews before apply.
  • Terraform can be packaged as modules and reused which avoids rewriting the code.

Below is the step by step procedure for getting started and is targeted at AWS and Terraform…

Pre-requisites :

  • Create an AWS account or use existing if you have one.
  • Generate AWS Access key and Secret key.

Note: We can use the AWS credential file or IAM role ARN and mention in the provider block in the code itself.

Use the command $ aws configure and put the access keys, secret key, region and start designing in the server where you are planning to execute terraform.

The workflow of Terraform:

Here is a simple example to bring up an EC2 instance using a single configuration file:

  1. Create a json file with .tf file extension in a directory with the necessary configuration to launch an instance

provider “aws” {

region=”xx-xxxx-xx”

}

resource “aws_instance” “example” {

ami = “ami-xxxxxxxx”

instance_type = “t2.micro”

tags{

Name= “test”

}

}

NOTE: Replace region and ami-id as per your configuration

2. Run the command $ terraform init to initialize the configuration

3. Use the command $ terraform plan to check the execution plan

4. Use the command $ terraform apply to the configuration

5. Once applied the verify the launch in the console

6. Use the command $ terraform destroy to delete all the resources created

Reusable infrastructure with Terraform modules

Terraform modules can be used to avoid code duplication for the same application deployed in multiple environments.

We can maintain the templates of the resources in a particular folder which may be used more than once for the deployment of environments. All we need to do is specify the path of the folder which will fetch the resource.

Important aspects of reusable modules:

  • We can make the modules configurable by using input variables. These variables can be defined in .tfvars files.
  • We can set a “source” parameter in the file. The source parameter tells Terraform where the module can be found. Terraform supports the following sources: Local file paths, Terraform Registry, GitHub, Bitbucket, HTTP URLs, S3 buckets.
  • “Data sources” allow data to be fetched to be used in Terraform configuration, a data source may retrieve information from modules/AWS resource by filtering on its tags
  • Terraform records information about what infrastructure is created in a Terraform state file i.e terraform. tfstate, using this file we can lock the state when the file has been shared and various members are trying to modify the module to avoid a race condition.

Example:

The resource module contains a basic template of all the AWS resources.

The reusable module contains the required AWS resource with the necessary configuration which can be modified according to our environment need.

The main module calls the reusable module to create the resource.

Considering the above, we are going to launch an elastic search service using terraform.

  • Make a directory in any path containing three folders for resource, reusable and main modules
  • Give the source path appropriately which calls the preceding modules as shown in the figure
  • Initialize and plan the terraform in the main module
  • Once the plan is satisfactory apply the plan to bring up the resources

I hope this helps to get a grip of terraforming!!!

AWS-Windows CloudWatch Monitoring part 3

By | AWS, Blogs | One Comment

Written by Mudita Misra, Cloud Engineer, Powerupcloud Technologies

In the last two blogs, we discussed configuring cloudWatch custom metrics and logs using the EC2config and SSM Agent. There are new developments in that space. AWS has launched a unified cloudWatch agent which can handle both custom metrics and logs seamlessly in both window and Linux environment.

Another important feature of this agent is, it can run in both the cloud environment and also your on-premises.

To enable custom metrics to AWS CloudWatch and set CloudWatch-Alarms on Windows follow the blog part-II, link below:

AWS-Windows CloudWatch Monitoring (part-II):Stream Windows/IIS log to AWS CloudWatch with Custom…In series of Monitoring the AWS Windows instances, here is how we can get the custom metrics to AWS CloudWatch and set…blog.powerupcloud.com

Unified CloudWatch agent supports both 32/64-bit Windows/Linux both on-prem and cloud. We can deploy it using AWS Systems Manager (SSM) Run CommandSSM State Manager.

Below are the features why we should go for the new Unified agent announced by AWS:

  • Simplified agent — only a single agent is used to collect metrics and logs, simplifying installation and collection.
  • Cross-platform — we may install the agent on either 64-bit Linux or Windows. It also includes HTTP proxy server support.
  • Configurable — agent automatically tracks useful system metrics, but may be modified to collect others such as CPU threads, mounted filesystems, and network interfaces.
  • CloudWatch — supports the normal 1-minute metrics and the new 1-second high-resolution metrics option. Includes EC2 dimensions like Instance ID, Image ID, and Autoscaling Group Name, as well as custom dimensions.
  • Migration — We may migrate existing AWS SSM and EC2Config settings for the new agent.

The new CloudWatch Agent is available now for download and uses for all AWS Regions except for GovCloud and China.

Let us start implementations for Windows server:

  1. We will be implementing the AWS Unified agent for windows instance.
  2. Launch a windows server in the AWS account.
  3. While launching the machine attach the IAM role with below permissions:

Now the SSM Agent is installed on the server as the role is attached to it.

4. Going ahead, we will do the magic with the help of the System manager here. Next, we have to install the CloudWatch Agent using the AWS Systems Manager.

5. Go to EC2 console from left pane scroll down and click on the Run command:

i) Choose AWS-ConfigureAWSPackage as a document.

ii) Select your target(instance) from the list, as the role attached the SSM is being installed on your server so it will be present in the list here.

Note: If the instances are not listed then install the SSM agent from below link:

https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html

iii) In Actions, put the values as below:

iv) Click Run.

v) This takes just a few seconds. Now we can use a simple wizard to set up the configuration file for the agent. This is just a sample file that we are going to create by the Unified agent. If you already have your configuration file you can directly switch to step 14.

6. Login/RDP to the windows server/box and go to below location:

C:\Program Files\Amazon\AmazonCloudWatchAgent

7. Click on the Amazon CloudWatch agent config wizard as shown in the screenshot. It will open a wizard-like below where the implementation will take place on the basis of the inputs given :

8. Start the manual implementation to customize the windows monitoring as per your requirements by just hitting some options like 1 or 2 and enter.

9. Finally, check the configuration file created automatically here.

10. Let’s customize the log file too by answering some questions to unified agent

11. After this, the configuration file has to be stored in the AWS Parameter store. Give a name as per your preference.

12. Next, choose region here us-east-1 by default came as our region is the same, verify and hit enter.

13. As we have attached a role which is having read-only access to EC2 so unified will take the access key automatically from there, you need to hit enter for the default value. Finally, the press enters to complete the process.

14. A config file is being stored on the parameter store in the same region, verify.

Below is the sample config file value created by Unified agent stored in AWS Parameter store:

{
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "",
"log_group_name": "."
}
]
},
"windows_events": {
"collect_list": [
{
"event_format": "xml",
"event_levels": [
"VERBOSE",
"INFORMATION",
"WARNING",
"ERROR",
"CRITICAL"
],
"event_name": "System",
"log_group_name": "System"
}
]
}
}
},
"metrics": {
"append_dimensions": {
"AutoScalingGroupName": "${aws:AutoScalingGroupName}",
"ImageId": "${aws:ImageId}",
"InstanceId": "${aws:InstanceId}",
"InstanceType": "${aws:InstanceType}"
},
"metrics_collected": {
"LogicalDisk": {
"measurement": [
"% Free Space"
],
"metrics_collection_interval": 60,
"resources": [
"*"
]
},
"Memory": {
"measurement": [
"% Committed Bytes In Use"
],
"metrics_collection_interval": 60
},
"Paging File": {
"measurement": [
"% Usage"
],
"metrics_collection_interval": 60,
"resources": [
"*"
]
},
"PhysicalDisk": {
"measurement": [
"% Disk Time"
],
"metrics_collection_interval": 60,
"resources": [
"*"
]
},
"Processor": {
"measurement": [
"% User Time",
"% Idle Time",
"% Interrupt Time"
],
"metrics_collection_interval": 60,
"resources": [
"_Total"
]
}
}
}
}

Copy the file name from here, we will be using this later.

15. Now we can deploy the configuration file to multiple instances at a time using the Run command with below steps:

i)Go running Command as we did above for installing the CloudWatch agent.

ii)Click on Run Command

iii)Choose Document: AmazonCloudWatch-ManageAgent

iv)Choose the server from the Target list.

v)Put the values as given below:

vi)Optional Configuration location: paste the File name copied from the Parameter store here: mudita-unified agent-config.

Action values in Run command

Click on Run.

16. Wait for some time and see the output as below from console:

The first output from Run Command:

A second output from Run Command:

17. Go to CloudWatch to see the logs and metrics for your target server :

i)Go to logs, since we have chosen Default location- System here, so logs must be present on the same location in CloudWatch:

Since we have chosen XML format while creating configuration file via Unified agent so the output is here in XML format, we can choose normal text format also :

ii)Go to Metrics now:

Choose CWAgent:

We can see the format we have chosen above while doing the setup from the wizard:
ImageID,InstanceID, InstanceType,objectname etc.

We can see the metrics now

This is how we can use a Unified agent which can help us in collecting the custom metrics on multiple Linux/windows servers EC2 or On-prem both.

This automation is done by using AWS services without logging in to each and every server.

For installing Unified agent on Linux server follow the AWS blog:

https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-agent-with-aws-systems-manager-integration-unified-metrics-log-collection-for-linux-windows/

Note: There is no fee to use the new agent, but CloudWatch will be charged as per AWS costing: https://aws.amazon.com/cloudwatch/pricing/

Thanks to AWS for this new feature.

Happy windows monitoring!!

AWS-Windows CloudWatch Monitoring (part-II): Stream Windows/IIS log to AWS CloudWatch with Custom metrics(Memory, FreeDisk)

By | AWS, Blogs, windows | No Comments

Written by Mudita Misra, Cloud Engineer, Powerupcloud Technologies

In a series of Monitoring the AWS Windows instances, here is how we can get the custom metrics to AWS CloudWatch and set CloudWatch-Alarms for the same.

  1. To enable CloudWatch on Windows to follow the blog part-I, link below:

Stream Windows/IIS logs to AWS CloudWatchTo enable CloudWatch on Windows follow the below steps:medium.com

2. Now next is to edit AWS.EC2.Windows.CloudWatch.json file, go to the location for this file C:\Program Files\Amazon\EC2config\Settings\AWS.EC2.Windows.CloudWatch.json, new paste the json file there as below:

{
"EngineConfiguration": {
"PollInterval": "00:00:15",
"Components": [
{
"Id": "ApplicationEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Application",
"Levels": "1"
}
},
{
"Id": "SystemEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "System",
"Levels": "7"
}
},
{
"Id": "SecurityEventLog",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Security",
"Levels": "7"
}
},
{
"Id": "ETW",
"FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogName": "Microsoft-Windows-WinINet/Analytic",
"Levels": "7"
}
},
{
"Id": "IISLog",
"FullName": "AWS.EC2.Windows.CloudWatch.IisLog.IisLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\inetpub\\logs\\LogFiles\\W3SVC1"
}
},
{
"Id": "CustomLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"LogDirectoryPath": "C:\\CustomLogs\\",
"TimestampFormat": "MM/dd/yyyy HH:mm:ss",
"Encoding": "UTF-8",
"Filter": "",
"CultureName": "en-US",
"TimeZoneKind": "Local"
}
},
{
"Id": "PerformanceCounterMemory",
"FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"CategoryName": "Memory",
"CounterName": "Available MBytes",
"InstanceName": "",
"MetricName": "Memory",
"Unit": "Megabytes",
"DimensionName": "InstanceId",
"DimensionValue": "{instance_id}"
}
},
{
"Id": "PerformanceCounterDisk-C",
"FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"CategoryName": "LogicalDisk",
"CounterName": "Free Megabytes",
"InstanceName": "C:",
"MetricName": "FreeDisk",
"Unit": "Megabytes",
"DimensionName": "InstanceId",
"DimensionValue": "{instance_id}"
}
},
{
"Id": "PerformanceCounterDiskC",
"FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"CategoryName": "LogicalDisk",
"CounterName": "% Free Space",
"InstanceName": "C:",
"MetricName": "C.FreeDiskPercent",
"Unit": "Percent",
"DimensionName": "Instance",
"DimensionValue": "Hostname: {hostname} IP Address: {ip_address} InstanceId: {instance_id}"
}
},
{
"Id": "PerformanceCounterDiskD",
"FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"CategoryName": "LogicalDisk",
"CounterName": "% Free Space",
"InstanceName": "D:",
"MetricName": "D.FreeDiskPercent",
"Unit": "Percent",
"DimensionName": "Instance",
"DimensionValue": "Hostname: {hostname} IP Address: {ip_address} InstanceId: {instance_id}"
}
},
{
"Id": "CloudWatchLogs",
"FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"AccessKey": "",
"SecretKey": "",
"Region": "ap-south-1",
"LogGroup": "Windows-logs",
"LogStream": "{instance_id}"
}
},
{
"Id": "CloudWatch",
"FullName": "AWS.EC2.Windows.CloudWatch.CloudWatch.CloudWatchOutputComponent,AWS.EC2.Windows.CloudWatch",
"Parameters": {
"AccessKey": "",
"SecretKey": "",
"Region": "ap-south-1",
"NameSpace": "WindowsCustom-Metrics"
}
}
],
"Flows": {
"Flows": [
"(ApplicationEventLog,SystemEventLog),CloudWatchLogs",
"(PerformanceCounterMemory,PerformanceCounterDiskC,PerformanceCounterDiskD,PerformanceCounterDisk-C ),CloudWatch"
]
}
}
}

3. Mention the region according to the instance as we have chosen ap-south-1

4. Save the file and delete the file at location C:\Program Files\Amazon\SSM\Plugins\awsCloudWatch\AWS.EC2.Windows.CloudWatch.json (if any present).

5. Restart the SSMAgent we can do this through the services.msc application or issue the following command:

net stop AmazonSSMAgent && net start AmazonSSMAgent

6. We should now have enabled the CloudWatch plugin on the instance. We can now visit the CloudWatch console to see the logs and metrics.

7. Please note, in the AWS.EC2.Windows.CloudWatch.json make sure we have configured the correct region for the CloudWatch parameters.

8. For reference find the below screen-shots:

Windows logs:

9.Custom metrics(Memory and FreeDisk)

i)Go to Metrics and click on WindowsCustom-Metrics,

ii)Click on InstanceId,

iii)Now we can see the Instance name, instance id, metrics name.

iv)Now create Alarms for the Custom metrics: Click on the metrics and go to the next column Graphed metrics, we will be able to see a window like below for Memory metrics, click on the bell icon and create the alarm to send the Custom memory alerts.

Note: As this Memory metrics are custom metric it will incur the extra charge of $0.30 per instance per month, and also general CloudWatch Alarm charges will be applied.

In our next blog, we will write on how to use the unified CloudWatch agent to achieve more detailed metrics.

Happy Windows Custom Metrics Monitoring!!

Kubernetes Event Notifications to a Slack Channel- Part V

By | AWS, Blogs | No Comments

Written by Priyanka Sharma, DevOps Architect, Powerupcloud Technologies.

In the previous parts of our Kubernetes blog series, we have discussed installing, deploying, monitoring and debugging the Kubernetes Cluster. In this article, we are covering the notification part i.e. how to get notified through Slack for each activity (creation/termination/update/restart) of Kubernetes resources.

Get the Slack Token

Go to Slack and create a new bot.

https://.slack.com/apps/new/bot

Save the API Token that you got on the next screen.

Now switch to slack channels and click on the slack channel on which you want the notifications from the Kubernetes cluster.

Execute:

/invite @<bot-username>

Hit “https://<your_organisation>.slack.com/apps/manage/custom-integrations”

Click Bots and you will be able to view the Bots. Select the bot which you have created and edit. It will show the channel that is added to your bot.

Using Kubewatch for Create/Delete/Update Notifications

We are using Kubewatch explained on the below link to get notified if the Kubernetes resources are being created, terminated or updated.

https://github.com/bitnami-labs/kubewatch

Create Configmap for kubewatch. Replace Slack API token and channel name.

kubewatch-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
name: kubewatch
data:
.kubewatch.yaml: |
handler:
slack:
token: <SLACK_API_TOKEN>
channel: <SLACK_CHANNEL_NAME>
resource:
deployment: true
replicationcontroller: true
replicaset: true
daemonset: true
services: true
pod: true

Execute command:

kubectl create -f kubewatch-config.yaml

Create service account, cluster role and role bindings with the below yaml file:

rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
name: kubewatch
namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kubewatch
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
- replicationcontrollers
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kubewatch
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kubewatch
subjects:
- kind: ServiceAccount
name: kubewatch
namespace: monitoring

Create it using the command:

kubectl create -f rbac.yaml

Create the Pod yaml file:

kubewatch.yaml

apiVersion: v1
kind: Pod
metadata:
name: kubewatch
namespace: monitoring
spec:
containers:
- image: tuna/kubewatch:v0.0.1
imagePullPolicy: Always
name: kubewatch
volumeMounts:
- name: config-volume
mountPath: /root
- image: gcr.io/skippbox/kubectl:v1.3.0
args:
- proxy
- "-p"
- "8080"
name: proxy
imagePullPolicy: Always
restartPolicy: Always
serviceAccount: kubewatch
serviceAccountName: kubewatch
volumes:
- name: config-volume
configMap:
name: kubewatch

kubectl create -f kubewatch.yaml

The notifications in slack will be like:

Note: Kubewatch will notify for the creation/termination/update of the resources. It won’t notify if the pod restarts.

Using Lifecycle Hook to get Notifications for Pod Restarts

We can get the restart count from kubectl get pods.

NAME                             READY     STATUS    RESTARTS   AGE<#podname#-xxxxxxx-xxxxxxxxxx>   2/2       Running   21         17d

We can see the restart count from the above output. To get notified for the pod restart, we are using a post-Start life cycle hook. Here’s the workflow:

  • Put post start lifecycle hook in the deployment.yaml which will execute whenever the pod starts.
  • Execute a POST curl request from the lifecycle hook to post the message to the Slack channel using API Token.

Refer to the deployment YAML content below for adding post start Life cycle Hook.

spec:
containers:
- image: xxxxxxx.dkr.ecr.ap-south-1.amazonaws.com/dev-tomcatapp:v1.0
imagePullPolicy: IfNotPresent
lifecycle:
postStart:
exec:
command: ["/bin/sh", "-c", "export hostname=`hostname` && curl -X POST -H 'Authorization: Bearer <SLACK_API_TOKEN>' -H 'Content-type: application/json' --data '{\"channel\":\"<SLACK_CHANNEL_NAME>\",\"text\":\"'\"The pod has started: $hostname\"'\"}'
https://slack.com/api/chat.postMessage"]
preStop:
exec:

and create the deployment using the command:

kubectl create -f deployment.yaml

If you have wanted to update the existing deployment, you can execute the below commands to add the post start hook:

kubectl get deployment -l name=<label> -o yaml > deployed-app.yaml

Edit deployed-app.yaml. Add the post start Lifecycle hook with proper indentation and execute the below command to do the deployment:

kubectl apply -f deployed-app.yaml

For more details on the above deployment.yaml, see our previous blog.

The notifications from the post start lifecycle hook will be like:

and that’s it..!! Hope you found it useful. Keep following our Kubernetes series for more interesting articles.

References:

Kubernetes Log Management using Fluentd as a Sidecar Container and preStop Lifecycle Hook- Part IV

By | AWS, Blogs | No Comments

Written b y Priyanka Sharma, DevOps Architect, Powerupcloud Technologies

Application logs play a vital role in any successful deployment. The first thing which is being checked after any deployment is “logs”. Even if someone is getting any error or unable to access the application, everyone needs “logs” for debugging. The logs are particularly useful for debugging problems and monitoring “what is happening from the application server-side”.

In Kubernetes, the application logs from a pod can easily be fetched from the command: “kubectl logs <podname>”.

But what if your container crashes or pod becomes inaccessible and you still want to access the old logs. In such cases, we must have permanent storage for our logs so that we don’t miss any of the application logs.

In this article, we will be discussing logs Management in Kubernetes. We are introducing the following two scenarios here:

  • Scenario 1: Store the Logs in a Centralized Location( S3). So that if the pod is deleted, you can easily retrieve logs from the S3 bucket.
  • Scenario 2: Stream the application logs to Elasticsearch in real-time.

PreStop Lifecycle Hook:

If one pod is deleted due to some reason, here is the workflow which we have followed to get the logs from a container:

  • Execute Prestop Lifecycle Hook which will execute before the pod is terminated. The prestop hook will do the following tasks:
  • Zip the locations where the application logs exist in the container. For example, /usr/local/tomcat/logs for any tomcat application.
  • Put the Zip file to the AWS S3 Bucket.

Prerequisites:

  • S3 Bucket setup for putting logs.
  • AWS CLI installed in the container to put the logs to the S3 Bucket. Use the below three commands to install awscli from Dockerfile:
RUN apt-get install -y python3-pipRUN apt-get install -y zipRUN pip3 install --upgrade awscli
  • Ensure the IAM role, which is attached to K8s cluster nodes, is having permissions to access the S3 bucket which is configured for putting logs.

Creating ConfigMap for the Bash Script:

prestop-config.yaml

apiVersion: v1
data:
prestop.sh: |
#!/bin/bash
DATE=`date '+%Y-%m-%d-%H%M%S'`
HOSTNAME=`hostname`
###custom application log location
APPLOGS="/home/Apps/Logs/*"
###tomcat logs location
TOMCATLOGS="/usr/local/tomcat/logs/*"
zip -r /tmp/${DATE}-${HOSTNAME}.zip ${APPLOGS} ${TOMCATLOGS}
SERVICE=`echo $HOSTNAME | cut -d'-' -f -2`
aws s3 cp /tmp/${DATE}-${HOSTNAME}.zip s3://dev-k8s-prestop-logs/${SERVICE}/
kind: ConfigMap
metadata:
name: dev-prestop-script-config
namespace: default

In our case, we have two logs locations for writing the application logs:

  • /usr/local/tomcat/logs/
  • /home/Apps/Logs/

Execute the below command to create a configMap:

kubectl create -f prestop-config.yaml

Verify whether the config map created or not:

kubectl get configmap

Once the config map is created, but the bash script into the container using volume mount in deployment.YAML file which is used to create the deployment for the application.

        - name: prestop-script
mountPath: /mnt

We are adding the script to /mnt location of the application container. The location can be modified according to the requirements.

Specify the volumes for the config map which we have created in the above step:

      - name: prestop-script
configMap:
name: dev-prestop-script-config

Ensure that you are giving the same names to the prestop volume and volume mount.

And add the prestop lifecycle hook in the spec.containers.

       lifecycle:
preStop:
exec:
command: ["/bin/sh", "/mnt/prestop.sh"]

Refer to the final deployment.yaml in the sections below.

Fluentd as a Sidecar Container:

Workflow:

If you want to stream the pod logs to AWS Elasticsearch Service, here is the workflow:

  • Run two containers for every single pod. (Since a pod is a group of one or more containers)
  • One is an application container where the tomcat application is being deployed
  • Another one is a Fluentd container which will be used to stream the logs to AWS Elasticsearch Service.
  • Share the logs directories from application containers to fluentd containers using volume mounts.
  • Specify those logs directories in fluentd config so that the logs will be taken from them and streamed to Elasticsearch.
  • Hit Kibana URL to view the logs.

Prerequisites:

  • AWS Elasticsearch Service Setup (Cognito enabled Authentication Elastcisearch will also work here).
  • Ensure the IAM role, which is attached to K8s cluster nodes, is having permissions to access the AWS Elasticsearch Domain.

We have prepared a Dockerfile for building the fluentd image which is used to stream the logs to AWS Elasticsearch Service.

Dockerfile

FROM ubuntu:16.04
RUN apt-get update
RUN ulimit -n 65536
RUN apt-get install -y curl
RUN curl https://packages.treasuredata.com/GPG-KEY-td-agent | apt-key add -RUN echo "deb http://packages.treasuredata.com/3/ubuntu/xenial/ xenial contrib" > /etc/apt/sources.list.d/treasure-data.
listRUN apt-get update && apt-get install -y -q curl make g++ && apt-get clean && apt-get install -y td-agent && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN sed -i -e "s/USER=td-agent/USER=root/" -e "s/GROUP=td-agent/GROUP=root/" /etc/init.d/td-agent
RUN /usr/sbin/td-agent-gem install fluent-plugin-aws-elasticsearch-service -v 1.0.0
CMD /usr/sbin/td-agent $FLUENTD_ARGS

Build a docker image out of it and push to a docker registry. We have used ECR as a docker registry provided by AWS.

docker build -t <registry/repo:tag> -f <Dockerfile>docker push <registry/repo:tag>

Create a fluentd config map using below YAML:

fluentd-config.yaml

apiVersion: v1
data:
td-agent.conf: |
<source>
@type tail
format multiline
format_firstline /[0-9]{2}-[A-Za-z]{3}-[0-9]{4}/
format1 /^(?<datetime>[0-9]{2}-[A-Za-z]{3}-[0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}) (?<Log-Level>[A-Z]*) (?<message>.*)$/
path /usr/local/tomcat/logs/catalina*,/usr/local/tomcat/logs/localhost*.log
path_key tailed_tomcat_path
pos_file /usr/local/tomcat/logs/tomcat.catalina.pos
tag tomcat.tomcat.logs
</source> <source>
@type tail
format apache
path /usr/local/tomcat/logs/localhost_access*
path_key tailed_localhost_access_path
pos_file /usr/local/tomcat/logs/tomcat.localhost.access.pos
tag tomcat.localhost.access.logs
</source> <source>
@type tail
format multiline
format_firstline /[0-9]{4}-[0-9]{2}-[0-9]{2}/
format1 /^(?<datetime>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}) (?<Level>[A-Z]*) (?<Message>.*)$/
path /home/Apps/Logs/*.log
path_key tailed_app_path
pos_file /home/Apps/Logs/tomcat.app.pos
tag tomcat.app.logs
</source> <filter tomcat.tomcat.logs>
@type record_transformer
<record>
hostname ${hostname}
</record>
</filter> <filter tomcat.localhost.access.logs>
@type record_transformer
<record>
hostname ${hostname}
</record>
</filter> <filter tomcat.app.logs>
@type record_transformer
<record>
hostname ${hostname}
</record>
</filter> <match **>
@type copy
<store>
@type aws-elasticsearch-service
type_name fluentd
logstash_format true
logstash_prefix dev-tomcatapp
flush_interval 60s
num_threads 8
<endpoint>
url https://AWS_ES_ENDPOINT
region ap-south-1
</endpoint>
</store>
<store>
@type stdout
</store>
</match>kind: ConfigMap
metadata:
name: dev-tomcatapp-fluentd-config

The <source> section can be changed according to the application platform.

Execute the below command to create the configmap:

kubectl create -f fluentd-config.yaml

Refer to the final deployment.yaml file below.

Creating a YAML file for the Deployment

Now that we are aware of the workflows, let’s create the deployment on Kubernetes Cluster.

Here is the deployment.yaml with lifecycle hook and fluentd container:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
name: dev-tomcatapp
tier: dev
name: dev-tomcatapp
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/dev-tomcatapp
spec:
replicas: 1
selector:
matchLabels:
name: dev-tomcatapp
tier: dev
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
name: dev-tomcatapp
tier: dev
spec:
containers:
- image: xxxxxxx.dkr.ecr.ap-south-1.amazonaws.com/dev-tomcatapp:v1.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command: ["/bin/sh", "/mnt/prestop.sh"]
name: dev-tomcatapp
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: 200m
memory: 1Gi
requests:
cpu: 100m
memory: 256Mi
volumeMounts:
- mountPath: /usr/local/tomcat/logs
name: tomcatapp-tomcat-logs
- mountPath: /home/Apps/Logs
name: tomcatapp-app-logs
- name: prestop-script
mountPath: /mnt
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
- name: fluentd
image: xxxxxxx.dkr.ecr.ap-south-1.amazonaws.com/tdagent-aws-es
env:
- name: FLUENTD_ARGS
value: -c /etc/td-agent/td-agent.conf
volumeMounts:
- name: tomcatapp-tomcat-logs
mountPath: /usr/local/tomcat/logs
- name: tomcatapp-app-logs
mountPath: /home/Apps/Logs
- name: config-volume
mountPath: /etc/td-agent
volumes:
- name: tomcatapp-tomcat-logs
emptyDir: {}
- name: tomcatapp-app-logs
emptyDir: {}
- name: config-volume
configMap:
name: dev-tomcatapp-fluentd-config
- name: prestop-script
configMap:
name: dev-prestop-script-config
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: secret01
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30

Execute the below command to create the deployment now:

kubectl create -f deployment.yaml

Verifying Logs

Get the pod name:

kubectl get pods -l name=dev-tomcatapp

and execute

kubectl logs -f pod/<podname> fluentd

It will show the stdout of logs parsing to ES:

Create an index pattern in Kibana:

Once the index is created, hit Discover to browse the logs.

If the pod is deleted, the zip file of logs will be shown in the s3 bucket:

and that’s it..!! Hope you found it very useful. Unlike us, at least now you won’t have to spend hours and days to research for a complete logging solution in Kubernetes. Follow this whole article and there you go without missing any logs. Happy Pod Logging..!! 🙂

Do visit our previous blogs for more interesting stuff on Kubernetes.

Share Files Securely Over Internet Using AWS Cognito and S3

By | AWS, Blogs | No Comments

Written By: Priyanka Sharma, Cloud Architect, Powerupcloud Technologies

One of our clients requested a simple web app that can share files in a secure manner with internal and external users. Out-of-the-box solutions like Dropbox or Gladinet did not suffice their needs as the customer had more requirements for this web app. They are,

  • The web page should list the name and size of all the files stored in an S3 bucket.
  • The webpage should allow the user to select certain files and share them to any email address.
  • Their internal employees can view, upload and share the selected files whereas the external users can only view and share the selected files.
  • External users should not be allowed to upload any file to the S3 bucket.
  • On sharing, the recipient should get an email with the download link and post-authentication, the recipient should be able to download the files with that link.
  • The recipient should be able to download the file from the app console too.

Keeping these requirements in mind, we built a simple web application using which you can share the files securely among internal and external users.

In this post, we will take you through the way in which the application works.

Proposed Solution

The application was built with S3 as the primary storage repo. Following AWS services were also used in this application.

  • IAM Role: To assume after login.
  • IAM SAML Provider: With ADFS Federation Metadata.
  • S3 Bucket: To upload and share the files.
  • AWS Cognito User Pool: To create external users.
  • AWS Cognito Federated Identity Pool: For the authentication providers (SAML and Cognito User Pool).
  • AWS EC2 Ubuntu Server: The application code is kept and services through a web server.
  • AWS EC2 Windows Server: Managing Active Directory and ADFS for Internal Users.

Prerequisites

S3 Bucket

We created an S3 bucket with the name “private-Cognito-s3” with following CORS Configuration:

<?xml version="1.0" encoding="UTF-8"?> 
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

For Internal Users

IAM Role to Assume by the Users

Create an IAM Role with below trust relationship.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Federated": "cognito-identity.amazonaws.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"cognito-identity.amazonaws.com:aud": "<cognito-federated-identity-pool-id>"
},
"ForAnyValue:StringLike": {
"cognito-identity.amazonaws.com:amr": "authenticated"
}
}
}
]
}

SAML Provider in AWS

Before you create a SAML provider, you need to download the SAML metadata document for your ADFS federation server. By default, you can download it from the following address:

https:///FederationMetadata/2007–06/FederationMetadata.xml

Create a SAML Provider in IAM with the above metadata file.

Setup ADFS

We followed the below link to set up a windows server for ADFS.

https://aws.amazon.com/blogs/security/enabling-federation-to-aws-using-windows-active-directory-adfs-and-saml-2-0/

Once we have the server provisioned, we created a relying party trust with the endpoints pointing to our application as shown in the below screenshot.

Also, we added the following Claim Rules for trust.

Transform

Get AD Groups With Rule

c:[Type == "http://schemas.micraosoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]

=> add(store = "Active Directory", types = ("http://temp/variable"), query = ";tokenGroups;{0}", param = c.Value);

Roles

Provide the SAML Arn and Arn of the IAM Role to assume in the value.

=> issue(Type = "https://aws.amazon.com/SAML/Attributes/Role", Value = "arn:aws:iam::xxxxxxxx:saml-provider/cognito-s3-saml,arn:aws:iam::xxxxxxx:role/ADFS-Readonly");

RoleSessionName with Outgoing Claim Type

https://aws.amazon.com/SAML/Attributes/RoleSessionName

For External Users

Create Cognito User Pool

We have created a Cognito user pool with an app client “power app”.

Create Federated Identity Pool for SAML as well as Cognito User Pool. We created a Cognito Federated Identity Pool with authentication providers.

Cognito User Pool: Give user pool id and app client id

SAML: Select SAML Provider from the list

Implementation

We provisioned an Ubuntu Server and configured Nginx Web Server to serve the application. Refer to the application code in the repo below:

https://github.com/powerupcloud/aws-private-file-sharing-tool

Also, an API (code exists in API directory in the repo) has been used to authenticate and redirect the AD users to the correct page. Start the API using below command:

node server.js

The API starts listening on 8443 port.

Main Features of Application

  • The users will be able to view the contents of their particular bucket and a shared bucket. Users will not be able to access the buckets which are owned by other users.
  • The internal users i.e. AD Users will be able to view, upload and share the selected files to any user internally or externally.
  • The external users i.e. Cognito Users will be able to view and share the selected files to any user internally or externally.
  • After login, the user will get the temporary credentials so that he/she will be able to access the bucket.
  • After listing the existing files, the files can be downloaded from the same console by clicking on the file name.
  • Once a file is shared with a user, he/she will get a download link on his/her email address. Also, the file will get copied to his/her folder in the same S3 bucket. For example, if userA is sharing a fileA to userB (userb@gmail.com), the fileA will get copied to userB’s folder i.e. the structure of the bucket will be: s3bucket/userb@gmail.com/usera@gmail.com/fileA.
  • The download link will be sent to the email address.
  • If registered, the user will be asked to enter the credentials with MFA to download the file.
  • It not already registered, the user will be redirected to the registration page. Once registered, the user will be asked to reenter his/her credentials to download the file.

Application in Action: Internal Employees

Step 1: Hit the Application URL

Step 2: Enter the credentials of the AD User

Step 3: After successful login, the user will be redirected to the S3 page.

Note: we are having prefix value the same as the username. Hence, the bucket structure will be,

  • private-Cognito-s3
  • priyanka@puc.com/files
  • user1@puc.com/files
  • user2@gmail.com/files
  • shared/files

Step 4: Since we are logged in as priyanka@puc.com, we will be able to view the contents of the specified folder and won’t be able to view the content of other folders.

Step 5: We can select the files and share them to any email address.

Step 6: The email address is already registered in the AWS Cognito User Pool, hence the download link will ask for the login credentials to enter.

Step 7: After entering the credentials, it also asks for the code sent to the registered Mobile Number.

Step 8: After a successful login, the user will be able to download the file.

If the email is sent to an unregistered user, the workflow will be as shown in the below screenshots.

Step 9: The download link will redirect you to the registration page.

Step 10: Once registered, you need to enter your credentials on the next page.

Note: Even if someone got the download link, he/she wont be able to access the file. Since the file has been shared to a particular user, only that user can download the file by his/her credentials.

Step 11: After a successful login, you will be able to download the file.

Application in Action: External Employees

Step 1: Login as External Users

Step 2: After entering the credentials, you will be asked for the code sent to your Mobile Number

Step 3: After successful login, you will be redirected to the S3 page where you can view and share the files

Demo Video:

I hope you find this app useful in multiple scenarios. Happy File Sharing!! 🙂

References:

If you would like to know more details on this project or if you want us to build something similar for you, please write to us at cloud@powerupcloud.com.

Enabling the leadership of a large OTT access business data just by asking for it

By | Alexa, AWS, Blogs | No Comments

Written By: Kartikeya Sinha, Lead Data Architect, Powerupcloud & Siva S, CEO, Powerupcloud Technologies

Just imagine the work-life of a Chief Executive or someone from the senior leadership team of a company. You would see them getting into meetings after meetings. They always seem to be thinking about something. To make better business decisions, they need to understand their business data. In their super busy schedule, it often turns out to be cumbersome for them to navigate through complex Business Intelligence (BI) dashboards and tens & hundreds of reports to find the metrics they need.

With the introduction of Natural Language Processing (NLP) APIs from leading pubic cloud providers like AWS, Azure & Google, we have started receiving a lot of requirements around integrating these NLP APIs with BI dashboards so that the senior business executives can simply ask for specific data and hear them out instantly.

One such case is discussed in this blog post.

 

Problem Statement

One of our customers is a large video streaming company. They collect several metrics including video streaming, customer behaviour, application usage, network usage, etc. But these metrics were distributed across several software used by them for video streaming including the likes of Mixpanel, Youbora, Appsee, etc. The customer had the following requirements:

 

  1. Build a data lake so that all data can be accessed from one centralized location
  2. Build ML engines for prediction, correlation of the app data
  3. Build a highly responsive and graphically rich reporting dashboard
  4. Enable NLP to search metrics using voice or text query

In this blog, we will be covering the custom reporting dashboard and NLP integration modules.

 

Data Lake Solution

Powerupcloud’s data team built a data lake using Amazon Redshift, Amazon S3 to support the data analysis processes. The data was loaded to Amazon S3 by Talend jobs. An ETL job converts the raw data files to readable CSV files and pushes to a target bucket. This allows the data to be queried either by Redshift Spectrum or Athena directly from Amazon S3 and this brings down the data storage costs quite a bit.

Below is a high-level architecture diagram without the Redshift Spectrum or Athena component.

 

 

Tech Stack

– Amazon Redshift as DWH.

– Amazon Lex to do NLP on the query text and extract intent and slot values.

– Elasticbeanstalk based Query processing engine written in Python3

– Webkit Speech Recognition API to convert speech to text.

– Elasticbeanstalk to host the BI dashboard

– Tech stack for the BI dashboard — Bootstrap, jQuery, Morris.js charts

 

Rich Reporting Dashboard

Once the data lake was implemented, we were faced with the next big problem-how can you integrate NLP into a BI platform? We tried several out-of-the-box BI platforms like Redash, PowerBI, etc. But integrating a browser-based voice-to-text converter was a challenge. So we decided to go with Google Web Kit and a custom reporting dashboard.

As the customer needed a rich UI, we chose morris.js charts running on a bootstrap theme. Morris.js allowed us to have rich colours and graphics in the graphs while the bootstrap theme helped in a high level of customization.

 

 

Integrating Amazon Lex

This architecture gives you a flow of data from the browser to Redshift.

The queries generated by Google Webkit is passed to Amazon NLP for intents and associated slots. Once the slots are identified, the parameters are passed to the Query Processing API which queries the Redshift for relevant data. This data is then presented through the custom reports built.

 

How does the solution work?

 

  1. Click on the ‘mic’ icon and ask your query.
  2. The BI tool does the speech to text conversion using Webkit Speech API.
  3. The text query is then sent to a Query Processing engine.
  4. Query processing engine sends a request to Amazon Lex for extracting intent and slot values from the query.
  5. Amazon Lex responds back with the intent name and slot values.
  6. Query processing engine uses the intent name and slot values to form a SQL query to the backend DWH-Amazon Redshift.
  7. Using the result of the query from Redshift, the query processing engine forms a response back to the frontend dashboard (BI).
  8. The frontend (BI) dashboard uses the response data to plot the graph/display it in the table.

 

Training Amazon Lex

The utterances are trained as below. Please note that the more utterances you train, the smarter the engine gets. The slots can be added as per the reports built in the dashboard. In this example, we chose ‘DeviceOS’, ‘GraphType’ and ‘# of days’ as the slots that are needed to be supplied from the customer’s query.

 

 

 

Challenges Faced

 

  1. Webkit Speech API does a pretty good job of converting speech to text. However, it works only on Google Chrome browser. Firefox has recently launched support for speech recognition, but that is still in very nascent stage.
  2. Although the ideal situation would be that you ask any meaningful query to the BI tool and it should be able to answer it. However, in order to do that Query processing engine needs to be really super smart to form dynamic SQL queries based on the user query. We have not yet achieved that and are evolving the Query processing engine to handle as many queries as possible without a need for modification.

 

Voice-Based BI Engine in Action

The voice search can pull reports based on 3 inputs,

 

  • Metrics-Visitors or Viewers or Video Views
  • Devices-iOS or Android or TV or PWA
  • Time-Last X days
  • Sample Query: Can you show me the number of visitors from iOS for the last 10 days?
  • Note: Voice search for terms like ‘Video Views’ and ‘PWA’ might be a little difficult for Lex to comprehend. Text search works better.

Hope this read was insightful. The future is voice-based platforms, be it apps, reports, customer service, etc.

If you would like to know more details on this project or if you want us to build something similar for you, please write to us at data@powerupcloud.com.

 

Parameterized Automation using Infrastructure as Code Model

By | AWS, Blogs | No Comments

Written by Manoj Kumar, Chief Cloud Architect & Priyanka Sharma, Sr. Cloud Architect, Powerupcloud Technologies

Server-less Architecture! Infrastructure as Code! DevOps Automation! A lot of buzz words are doing rounds in the market. While it is easy to sell these buzz words to customers, a lot of work goes behind implementing such an architecture on the cloud. In this post, we will be covering a recent work done by our cloud consulting team in implementing ‘Infrastructure as Code’ based architecture for one of our customers on AWS. Tip: Infrastructure as Code is a model where operations teams can automatically manage and provision IT infrastructure through code, rather than using a manual process.

Why should you continue to read on?

This is one of the complex project pulled off by our enthusiastic cloud architects, trying fully functional ‘parameterized’ Infrastructure as Code deployment model for the first time.

What were the challenges?

  1. Customer’s environment on AWS was not scalable and there was no fault-tolerant mechanism implemented
  2. The lead time to get a developer operational was high as the developer ended up waiting for a longer duration to access cloud resources like EC2, RDS, etc
  3. The deployment process was manual which increased the chances of unforced human errors
  4. Configuration management took longer time which further slowed down the deployment process
  5. No centralized mechanism for user management, log management, and cron jobs monitoring

The solution proposed by Powerup?

  1. Migrate their monolithic service into smaller independent services which are self deployable, sustainable, and scalable
  2. Set up CI/CD using Jenkins and Ansible
  3. Implement centralized user management using FreeIPA
  4. Implement centralized log management using ELK stack
  5. Implement centralized cron jobs monitoring using Healthcheck.io

Technical assessment of current infrastructure

Before we could start building the Amazon Cloudformation templates, we had to conduct a thorough assessment of their existing infrastructure to identify the gaps and plan the template preparation phase accordingly.

Below were a few key findings from our assessment:

  1. Termination Protection was not enabled to many EC2 instances
  2. IAM Password policy was not implemented and root MFA is not enabled
  3. IAM roles were not used to access the AWS services from EC2 instances
  4. CloudTrail was not integrated with Cloudwatch logs
  5. S3 Access logs for Cloudtrail S3 bucket was not enabled
  6. Log Metric was not enabled for (1) Unauthorised API Calls; (2) Using ROOT Account to access the AWS Console; (3) IAM Policy changes; (4) Changes to CloudTrail, CloudConfig, S3 Bucket policy; (5) Alarm for any security group changes, NACL, RouteTable, VPC
  7. SSH ports of few security groups were open to Public
  8. VPC Flow logs were not enabled for few VPCs

Proposed AWS Architecture

Powerupcloud proposed a highly available, scalable architecture on AWS which can be launched using Cloudformation templates.

Architecture Description

  1. Different VPCs were provisioned for Stage, Production and Infra management
  2. VPC peering was established from Infra VPC to Production and Stage VPC
  3. VPN tunnel was established between customer’s office and AWS VPC for SSH & Infra tool access
  4. All layers except the elastic load balancer were configured in private subnet
  5. Separate security group was configured for each layer including DB, Cache, Queue, App, ELB, Infra security groups, and only the required inbound and outbound rules were enabled
  6. Amazon ECS was configured in auto-scaling mode and hence the ECS workers can scale horizontally based on the load on the entire ECS cluster
  7. Service level scaling was implemented for each service to scale the individual service automatically based on the load
  8. Elasticache (Redis) was used to store the end-user sessions
  9. Highly available RabbitMQ cluster was configured and it was used as a messaging broker between the microservices
  10. Multi-AZ RDS for used to host MySQL and Postgresql
  11. MongoDB was configured in master-slave mode
  12. IAM roles were configured for accessing the AWS resources like S3 from EC2 instances
  13. VPC flow logs, CloudTrail, CloudConfig were enabled for logging purpose
  14. The logs were then streamed to AWS Elasticsearch services using AWS Lambda
  15. Alerts were configured for critical events like instance termination, IAM user deletion, Security group updates, etc
  16. AWS system manager was used to manage and collect the OS, Application, instance metadata of EC2 instances for inventory management
  17. AMIs and backups were configured for business continuity
  18. Jenkins was configured for CI/CD process
  19. CloudFormation template was used for provisioning/updating the environment
  20. Ansible was used as configuration management for all the server configurations like Jenkins, Bastion, FreeIPA, etc
  21. Sensu monitoring system was configured to monitor system performance
  22. New Relic was configured for application performance monitoring and deployment tracking

Deployments

Deployment Flow

Continuous Integration

Webhooks were configured to trigger the deployment pipeline whenever a commit was pushed to the Github repo.

Continuous Deployment

The most complex piece of this entire project was ‘parameterization’. Yes, you read it right. Our team went on to build Jenkins jobs which can accept different parameters on the fly and launch the AWS environment accordingly. Neat, isn’t it? So, how did we achieve this?

  • The Jenkins job we built can accept multiple parameters from the user including
  1. Docker base image
  2. Container CPU and memory
  3. Service name
  4. Environment type-stage/pre-prod/prod
  5. Github repository URL
  6. Git origin (branch or tag)
  7. Host
  8. Container ports
  9. Notification email and few more
  • Six jobs were prepared in a pipeline for each service. Once the upstream project was successful, the downstream project gets triggered.

CloneRepo

  1. Clones the Github repo in the workspace
  2. If a branch was selected in the stage environment for the Git origin, one Git tag with timestamp value gets created and pushed to the Github repo
  3. Notifies the status (failure/success/unstable/stable) through Email/Slack

BuildCode

  1. Prepares config files for the service if applicable
  2. Installs dependencies, eg: npm install
  3. Executes the build process commands, eg: Gradle build
  4. Notifies the status (failure/success/unstable/stable) through Email/Slack

BuildImage

  1. Prepares a Docker image with the building code, and start command as an entry point and tags the image with the Git commit id
  2. Pushes the Image to ECR (there is one ECR repo for each service)
  3. Notifies the status (failure/success/unstable/stable) through Email/Slack

DeployCode

  1. Deploys the Docker image on the ECS Cluster
  2. Waits for the service to be stable on cluster
  3. Notifies the status (failure/success/unstable/stable) through Email/Slack

ValidationTest

  1. Validates the Health Check URL for the service
  2. Notifies the status (failure/success/unstable/stable) through Email/Slack

LatestTag

  1. Tags the docker image with “latest” tag and push to ECR repo (used in rollback job)
  2. Notifies the status (failure/success/unstable/stable) through Email/Slack

Service Rollback Job

  1. Take user inputs from Jenkins parameters
  2. Deploys the Docker image, with a tag as specified in the parameter, on the ECS cluster
  3. Waits for the service to be stable on cluster
  4. Notifies the status (failure/success/unstable/stable) through Email/Slack

Configuration Management

Ansible Playbooks were used to achieve the following:

  • Hardening of EC2 servers and Docker base images
  • Enabling SSH recording for the bastion server
  • Configuring Repo server for APT/PIP private repositories
  • RabbitMQ installation and configuration
  • MongoDB installation and configuration
  • Provisioning of infrastructure components: (1) Jenkins; (2) Metabase; (3) Freeipa Server and Client; (4) Sensu
  • Preparing Ubuntu EC2 Server as ECS Host

Operations Management

  • Nodejs script was written to allow or revoke the rules from a security group
  • The script was stored in a Codecommit Repo
  • One Jenkins Job was configured which takes the user input for script related parameters and executes the action (allow/revoke) as specified

Benchmarking Exercise

  • Load-testing: “Bees with Machine Guns” tool was used to do the load test on the ECS Services

Backup & DR Plan

  • Backup Policy: Lambda functions were set up to create AMIs of EC2 servers and RDS snapshots and delete them after the retention period.
  • Infra: Cloudformation templates and Ansible playbooks were used to provision the Infrastructure
  • Cloudformation templates are stored in a Codecommit repo which was used for provisioning of the following resources:
  1. VPC, Subnets, SecurityGroups
  2. ECS Cluster with Autoscaling Group
  3. RDS DB Instance
  4. Redis Elasticache Cluster
  5. Standalone EC2 servers for MongoDB and RabbitMQ
  • Jenkins Jobs were setup for provisioning of Cloudformation stacks

Monitoring Setup

  • Infra: Sensu tool was used for the infrastructure monitoring
  • Dashboard: Grafana dashboard was built to display the metrics
  • Application Performance Monitoring: Newrelic was set up and configured for the application monitoring
  • Log Analytics-ELK
  1. AWS Elasticsearch cluster was setup
  2. One centralized logstash server was set up and configured
  3. Filebeat was installed in all the ECS Instances that will send the mounted service logs to logstash server
  4. Logstash will stream the logs to Elasticsearch cluster
  5. The service logs were visualized in Kibana dashboard

Hope this read was useful. Welcome to the world of automation. We encourage you to try this and share your learnings with us.

If you would like to know more details on this project or if you want us to build something similar for you, please write to us at cloud@powerupcloud.com.