Category

Blogs

Access Management of GCP instances by configuring Google Cloud Identity LDAP for user login

By | Blogs | No Comments

Written by Madan Mohan K, Associate Cloud Architect

“One Man’s TRASH can be another Man’s IDENTITY”-Identity Access Management

Maintaining two identity management systems for SaaS apps and traditional apps/infrastructure which results in intricacy, fragmented security, and additional cost.

To overcome this Google launched secure LDAP that lets you manage access to SaaS apps and traditional LDAP based apps/infrastructure using a single cloud-based identity and access management (IAM) solution.

Cloud Identity:

A unified identity, access, app, and endpoint management (IAM/EMM) platform that helps IT and security teams maximize end-user efficiency, protect company data, and transition to a digital workspace.

LDAP (Lightweight Directory Access Protocol) is an application protocol for querying and modifying items in directory service providers like Active Directory, which supports a form of LDAP.

Platform Used:

  • G-Suite Admin with cloud Identity Premium 
  • Google Cloud

G-Suite Admin:

Create LDAP client from the Apps in the G-Suite Admin console

G-Suite Admin Console

In the LDAP apps page, click on “ADD CLIENT” button and Key in the required details

LDAP client creation

Under Access Permissions, you will have 3 settings:

  • Verify user credentials,
  • Read user information &
  • Read group information.

In this illustration, we chose to go with the entire domain option. If you wish to have restricted access it can be done by limiting the user access to OU.

In the “Read group information” section, change the option to On and click the “ADD LDAP CLIENT” button to create the client.

Once after the configuration When prompted with a Google SSL certificate Click on “Download certificate” and then “CONTINUE TO CLIENT DETAILS”

The service status should be in ON state. So, in the Status page, select “ON for everyone” and click on “SAVE”

Well, that is all at the G-Suite Admin console.

Google Cloud:

  • Create an instance in the GCP. In this example, we chose to use Ubuntu 16.
  • Update the Instance using sudo apt update -y
  • Install the SSSD package using sudo apt install -y sssd sssd-tools
  • Once the installation is done create a new file in /etc/sssd/ and name it as sssd.conf. You can do it using vi /etc/sssd/sssd.conf or the preferred editor.

The sssd.conf file should include the following and look similar like the image below

Note: Remember to replace the domain with yours. By default, Google Linux instances disable password authentication so change it to Yes.

Configuration in Google Instance:

  • Upload the certificate which was downloaded earlier in the G-Suite Download certificate Page.
  • Change the permission of sssd.conf file using sudo chown root:root /etc/sssd/sssd.conf & sudo chmod 600 /etc/sssd/sssd.conf.
  • Restart the SSSD service using sudo service sssd restart

To verify that SSSD is running and connecting to the LDAP server you can run the following command with any of the users in your G Suite account:

  • Type getent passwd username@powerup.university in the instance created in google cloud and the output should look something like this:

Instance Access Scenario:

Now, when you try to ssh from the open in the browser window you will receive the following error. Well now without the G-Suite user we will not be able to log in to the instance.

Granular Level Access to the G-Suite User: When you need to restrict the user access only to the instance. We need to set the custom metadata as enable-oslogin=TRUE

The following roles must be assigned to the G-Suite user to access the instance using a third-party tool(e.g putty).

  • Computer OS Admin Login
  • Compute OS Login
  • Service Account User

Now open a third-party tool and use the G-Suite user and password to login to the machine.

Inference:

When all the identities and apps are managed in a single window the complexity is reduced and security is enhanced which also leads to an increase in the adoption of cloud technology across your business

Update:

In the forthcoming days, we shall have G-Suite users access the Windows instances using the G-Suite credentials.

AWS EKS Authentication and Authorization using AWS Single SignOn

By | Blogs | No Comments

Written by Priyanka Sharma, DevOps Architect, Powerupcloud Technologies

Amazon EKS uses IAM to provide authentication to the Kubernetes cluster. The “aws eks get-token” command is being used to get the token for authentication. However, IAM is only used for authentication of valid IAM entities. All permissions for interacting with the Amazon EKS cluster’s Kubernetes API is managed through the native Kubernetes RBAC system.

In this article, we are covering the authentication and authorization of AWS EKS through OnPrem ActiveDirectory i.e. in general, how we can provide EKS access to Onprem AD users. We have divided the solution into two parts: One is integrating OnPrem AD to AWS through AWS SingleSignOn and another is implementing RBAC policies on Kubernetes Cluster.

For the solution, we are using the following services:

  • OnPrem Active Directory with predefined users and groups: Ensure the below ports are allowed for your AD:
    • TCP 53/UDP 53/TCP 389/UDP 389/TCP 88/UDP 88/UDP 389
  • AD Connector on AWS which connects to the OnPrem AD
  • AWS Single SignOn: Integrates with AD Connector, allowing users to access the AWS Command Line Interface with a set of temporary AWS credentials.
  • AWS EKS: version 1.14

Integrating OnPrem AD with the AWS Single SignOn

For instance, we have created the following users and groups on OnPrem AD for the demo purpose:

AD UsernameRespective AD Group
user1EKS-Admins
user2EKS-ReadOnly
dev1EKS-Developers

Ensure to setup and AD Connector in the same region as AWS SignOn. Refer to our previous blog for setting up an Active Directory Connector.

Switch to AWS SingleSignOn Console and change the user directory. Select the AD connector created in the above step.

Select the account where you have setup the EKS Cluster.

Search for the AD group for which you want to give the EKS access.

Create a custom permission set.

Attach the below custom permission policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "*"
        }
    ]
 }

Select the Permission set created above.

Finish. Similarly, Create for Readonly and Developers too with the same permission set policy. Verify the permission sets in AWS accounts once.

Note that no specific permissions are assigned to the assumed role for the AD Users/Groups at this stage. By default the assumed role will not have permission to perform any operations. The specific authorization permissions will be defined via Kubernetes RBAC in the next section.

Behind the scenes, AWS SSO performs the following operations in Account B (member):

  • Sets up SAML federation by configuring an Identity Provider (IdP) in AWS IAM. The Identity Provider enables the AWS account to trust AWS SSO for allowing SSO access.
  • Creates an AWS IAM role and attaches the above permission set as a policy to the role. This is the role that AWS SSO assumes on behalf of the Microsoft AD user/group to access AWS resources. The role created will have prefix “AWSReservedSSO”.

Go to the account you have applied permission set to. There will be an IAM role created by SSO. In our case, below is the screenshot from the target account:

Creating RBAC Policies on Kubernetes

A Role is used to grant permissions within a single namespace. Apply the below role to create the multiple K8s role for Admins, Readonly and Dev groups respectively:

---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
   name: default:ad-eks-admins
   namespace: default
 rules:
 - apiGroups: ["*"]
   resources: ["*"]
   verbs: ["*"]
---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
   name: default:ad-eks-readonly
   namespace: default
 rules:
 - apiGroups: [""]
   resources: ["*"]
   verbs: ["get", "list", "watch"]
---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
   name: default:ad-eks-developers
   namespace: default
 rules:
 - apiGroups: ["*"]
   resources: ["services","deployments", "pods", "configmaps", "pods/log"]
   verbs: ["get", "list", "watch", "update", "create", "patch"]
---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
   name: default:ad-eks-monitoringadmins
   namespace: monitoring
 rules:
 - apiGroups: ["*"]
   resources: ["*"]
   verbs: ["*"]

Edit the existing aws-auth configmap through the below command:

kubectl edit configmap aws-auth –namespace kube-system

Add the below contents:

- rolearn: arn:aws:iam::ACCOUNTID:role/AWSReservedSSO_AD-EKS-Admins_b2abd90bad1696ac
      username: adminuser:{{SessionName}}
      groups:
        - default:ad-eks-admins
    - rolearn: arn:aws:iam::ACCOUNTID:role/AWSReservedSSO_AD-EKS-ReadOnly_2c5eb8d559b68cb5
      username: readonlyuser:{{SessionName}}
      groups:
        - default:ad-eks-readonly
    - rolearn: arn:aws:iam::ACCOUNTID:role/AWSReservedSSO_AD-EKS-Developers_ac2b0d744059fcd6
      username: devuser:{{SessionName}}
      groups:
        - default:ad-eks-developers
    - rolearn: arn:aws:iam::ACCOUNTID:role/AWSReservedSSO_AD-EKS-Monitoring-Admins_ac2b0d744059fcd6
      username: monitoringadminuser:{{SessionName}}
      groups:
        - default:ad-eks-monitoring-admins

Ensure to remove: aws-reserved/sso.amazonaws.com/ from the role_arn.

kubectl create rolebinding eks-admins-binding --role default:ad-eks-admins --group default:ad-eks-admins --namespace default

kubectl create rolebinding eks-dev-binding --role default:ad-eks-developers --group default:ad-eks-developers --namespace default

kubectl create rolebinding eks-readonly-binding --role default:ad-eks-readonly --group default:ad-eks-readonly --namespace default

kubectl create clusterrolebinding clusterrole-eks-admins-binding --clusterrole=cluster-admin  --group default:ad-eks-admins

kubectl create clusterrolebinding clusterrole-eks-readonly-binding --clusterrole=system:aggregate-to-view  --group default:ad-eks-readonly

Time for some Action

Hit SSO User Portal URL (highlighted in the below screenshot):

Give user AD credentials which is added to EKS-Admins group:

Click on Programmatic access:

It gives temporary AWS Credentials:

Create a ~/.aws/credentials file in the server with the credentials got from SSO:

[ACCOUNTID_AD-EKS-Admins]
aws_access_key_id = ASIAZMQ74VVIMRLK2RLO
aws_secret_access_key = gzOs61AcQ/vyh0/E9y+naT3GF3PDKUqB5stZLWvv
aws_session_token = AgoJb3JpZ2luX2VjENr//////////wEaCXVzLWVhc3QtMSJIMEYCIQCcs8/t5OK/UlOvSQ/NSXt+giJm9WkxVkfUhY6MFVnJwgIhAMOuJxb/CqhNx12ObPY4Obhe4KmxyEdyosqzqq63BOLaKt8CCKP//////////wEQABoMNjQ1Mzg1NzI3MzEyIgzw9o8jbVmjgcjgTHIqswICmRCh/7qIgBbxjG0kZJdmrGFEHjssv1b4Rl3AnIel7p0RizMDzzY9lQIlsuE5S7xYVB4alVVl1MNQ/1+iNSrSAG4LlCtSIaMrmUZ+hspR1qiQ5cqS2954UhgzEb081QCzYMbPgtvtPWwiiDZ9LkYOU2tp9hWbX7mHAZksFTHgEOO62hEuJWl3bh6dGYJWqyvTO3iwSJZYeqKJ/vY0MNnx5bjcqjgehUA6LnpUES3YlxelAGQPns7nbS0kOzDatoMe4erBIUTiP60vJ4JXJ2CFPsPmX6Doray0MWrkG/C9QlH4s/dZNCIm6In5C3nBWLAjpYWXQGA9ZC6e6QZRYq5EfMmgRTV6vCGJuSWRKffAZduXQJiZsvTQKEI0r7sVMGJ9fnuMRvIXVbt28daF+4ugyp+8MOCXjewFOrMB8Km775Vi0EIUiOOItQPj0354cao+V9XTNA/Pz23WTs8kF+wA5+il7mBOOEkmhLNrxEkuRTOCv0sn52tm9TeO9vSHRbH4e4xaKoJohyBYZTlEAysiu8aRQgahg4imniLYge+qvelQeDl1zYTBsea8Z71oQDcVVtBZzxmcIbS0V+AOOm81NTLRIIM1TNcu004Z7MnhGmD+MiisD0uqOmKVLTQsGLeTKur3bKImXoXNaZuF9Tg=

Update the KubeConfig for EKSAdmins groups as below:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: XXXXXX
    server: https://3B6E58DAA490F4F0DD57DAE9D9DFD099.yl4.ap-south-1.eks.amazonaws.com
  name: arn:aws:eks:ap-south-1:ACCOUNTID:cluster/puck8s
contexts:
- context:
    cluster: arn:aws:eks:ap-south-1:ACCOUNTID:cluster/puck8s
    user: arn:aws:eks:ap-south-1:ACCOUNTID:cluster/puck8s
  name: arn:aws:eks:ap-south-1:ACCOUNTID:cluster/puck8s
current-context: arn:aws:eks:ap-south-1:ACCOUNTID:cluster/puck8s
kind: Config
preferences: {}
users:
- name: arn:aws:eks:ap-south-1:ACCOUNTID:cluster/puck8s
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - ap-south-1
      - eks
      - get-token
      - --cluster-name
      - puck8s
      - -r
      - arn:aws:iam::ACCOUNTID:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_AD-EKS-Admins_b2abd90bad1696ac
      command: aws
      env:
        - name: AWS_PROFILE
          value: "ACCOUNTID_AD-EKS-Admins"

Similarly, update the KubeConfig and get the temporary credentials for ReadOnly and Dev Users.

KubeConfig for ReadOnly Users:

users:
- name: arn:aws:eks:ap-south-1:ACCOUNTID:cluster/puck8s
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - ap-south-1
      - eks
      - get-token
      - --cluster-name
      - puck8s
      - --role
      - "arn:aws:iam::ACCOUNTID:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_AD-EKS-ReadOnly_2c5eb8d559b68cb5"
      command: aws
      env:
        - name: AWS_PROFILE
          value: "ACCOUNTID_AD-EKS-ReadOnly"

AWS Profile for Temporary Credentials:

Export the KUBECONFIG for ReadOnly Users and try out the following commands:

Export the KUBECONFIG for EKS Admin Users and try out the following commands:

Export the KUBECONFIG for EKS ReadOnly Users and try out the following commands:

That’s all..!! Hope you found it useful.

References:

https://aws.amazon.com/blogs/opensource/integrating-ldap-ad-users-kubernetes-rbac-aws-iam-authenticator-project/

Email VA report of Docker Images in ECR

By | Blogs | One Comment

Written by Praful Tamrakar, Senior Cloud Engineer, Powerupcloud Technologies

Amazon ECR is Elastic Container Registry provided by Amazon to store, decrypt and manage container images. Recently AWS announced Image scanning feature for the images stored in the ECR. Amazon ECR uses the Common Vulnerabilities and Exposures (CVEs) database from the open-source CoreOS Clair project and provides you with a list of scan findings.

Building container images in a Continuous Integration (CI) pipeline, pushing these artifacts into ECR has been a common approach adopted widely. But along with this, we would also like to Scan the Container Image and send a Vulnerability Assessment report to the customer. The Email alert will be triggered only if the container has any critical vulnerabilities.

How Can I Scan My Container Images?

Container images can be scanned using lots of third-party tools such as Clair, Sysdig Secure, etc. To use these tools, the required server/database needs to be managed by us. This adds additional effort to the operations team.

To reduce these efforts, we can use the Image scanning feature of the ECR.

  • You can scan your container images stored in ECR manually.
  • Enable Image Scan for the push on your repositories so that each and every image is checked against an aggregated set of Common Vulnerabilities and Exposures (CVEs).
  • Scan Images using an API command thereby allowing you to set up periodic scans for your container images. This ensures continuous monitoring of your images.

Problem Statement

Currently, no direct way to achieve getting the scan results in CloudWatch or CloudTrail.  However,  it can be achieved using the following approach.

Resolution:

  1. Configure an Existing Repository to Scan on Push using AWS CLI
aws ecr put-image-scanning-configuration --repository-name <ECR_REPO_NAME> --image-scanning-configuration scanOnPush=true --region <REGION_CODE>
  1. Getting the image scan findings (scan results from ECR) can be achieved through a Lambda function that will use an API call.
    1. Create an SNS topic with EMAIL as the subscription
    2. Create a lambda function with runtime Python 3.7 or above, and attach AmazonEC2ContainerRegistryPowerUser and AmazonSNSFullAccess to the following AWS provided policy to the Lambda service Role.

b. Paste the following python command which will return the image scan findings summary

import json
from datetime import datetime
from logging import getLogger, INFO
import os
import boto3
from botocore.exceptions import ClientError


logger = getLogger()
logger.setLevel(INFO)

ecr = boto3.client('ecr')
sns = boto3.client('sns')

def get_findings(tag):
    """Returns the image scan findings summary"""
    
    try:
        response = ecr.describe_image_scan_findings(
            repositoryName='<NAME_OF_ECR >',
            registryId='<AWS_ACCOUNT_ID >',
            imageId={
            'imageTag': tag},
        )
        
        criticalresult = {}
        criticalresultList = []
        findings = response['imageScanFindings']['findings']
        for finding in findings:
            if finding['severity'] == "CRITICAL": #Can be CRITICAL | HIGH
                # print(findding['severity'])
                name  = finding['name']
                description = finding['description']
                severity = finding['severity']
                criticalresult["name"] = name
                criticalresult["description"] = description
                criticalresult["severity"] = severity
                criticalresultList.append(criticalresult)
        return criticalresultList
        
            
    except ClientError as err:
        logger.error("Request failed: %s", err.response['Error']['Message'])

def lambda_handler(event, context):
    """AWS Lambda Function to send ECR Image Scan Findings to EMAIL"""
    scan_result = get_findings(event['tag'])
    print (scan_result)
    
    
    sns_response = sns.publish(
    TopicArn='arn:aws:sns:<AWS_REGION_CODE>:<AWS_ACCOUNT_ID>:<SNS_TOPIC>',    
    Message=json.dumps({'default': json.dumps(scan_result)}),
    MessageStructure='json')
    
    print (sns_response)

This email contains the Name, Description and Severity level for the scanned image.

[{
"name": "CVE-2019-2201", 
"description": "In generate_jsimd_ycc_rgb_convert_neon of jsimd_arm64_neon.S, there is a possible out of bounds write due to a missing bounds check. This could lead to remote code execution in an unprivileged process with no additional execution privileges needed. User interaction is needed for exploitation.Product: AndroidVersions: Android-8.0 Android-8.1 Android-9 Android-10Android ID: A-120551338", 
"severity": "CRITICAL"
}]

d. Trigger this from the Jenkins pipeline

stage('Image Scan'){
    node('master'){        
        sh'''

        sleep 60
        if [[ $(aws ecr describe-image-scan-findings --repository-name < ECR_REPO_NAME> --image-id imageTag=${IMAGETAG} --region ap-southeast-1 --registry-id <AWS_ACCOUNT_ID> --output json --query imageScanFindings.findingSeverityCounts.CRITICAL) -gt 0 ]]
          then
           aws lambda invoke --function-name <LAMBDA_FUNCTION_NAME> --invocation-type Event --payload '{"tag":"${IMAGETAG}"}' response.json
        fi

        '''
      
      
         }}

OPTIONAL

You can create a CloudWatch rule that will match the scanning completion event using this event pattern If you don’t want to trigger this lambda function with pipeline but with Cloudwatch event for Event for a Completed Image Push

{
    "version": "0",
    "id": "13cde686-328b-6117-af20-0e5566167482",
    "detail-type": "ECR Image Action",
    "source": "aws.ecr",
    "account": "123456789012",
    "time": "2019-11-16T01:54:34Z",
    "region": "us-west-2",
    "resources": [],
    "detail": {
        "result": "SUCCESS",
        "repository-name": "my-repo",
        "image-digest": "sha256:7f5b2640fe6fb4f46592dfd3410c4a79dac4f89e4782432e0378abcd1234",
        "action-type": "PUSH",
        "image-tag": "latest"
    }
}

This event pattern will match the exact completion’s event API. After that, you can pass all of the matched events to the Lambda function so that it extracts the values of “repository-name”, “image-digest” and “image-tags”, which will be passed to the DescribeImageScanFindings API call. For reference: https://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr-eventbridge.html

Note:

If the ECR repository is not in the same account where Lambda is configured then, you must configure image permission for the image.

  1. Go to ECR console, and select the ECR repository.
  1. Click on Permission in the Left-hand side dashboard.
  2. Click on Edit policy json
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "pull and push",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
                    "arn:aws:sts::<AWS_ACCOUNT_ID>:assumed-role/<LAMBDA_ROLE_NAME>"
        ]
      },
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchCheckLayerAvailability",
        "ecr:BatchGetImage",
        "ecr:BatchGetImage",
        "ecr:CompleteLayerUpload",
        "ecr:DescribeImageScanFindings",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetDownloadUrlForLayer",
        "ecr:InitiateLayerUpload",
        "ecr:PutImage",
        "ecr:UploadLayerPart"
      ]
    }
  ]
}
  1. Save it.
  2. Trigger the lambda.

And that’s it..!! Hope you found it useful. Keep following our Blog for more interesting articles.

Elasticsearch Logstash Kibana (ELK) Authentication using Active Directory

By | Blogs | No Comments

Written by Priyanka Sharma, DevOps Architect, Powerupcloud Technologies

This article covers how you can enable security features on ELK to communicate with AD to authenticate Users. The security features provide two realms to achieve the same: One is LDAP realm and the other one is the ActiveDirectory realm.

We have used the Active_directory realm in our configurations. active_directory realm uses an LDAP bind request so it is similar to the LDAP realm. The Active Directory realm authenticates users using an LDAP bind request. After authenticating the user, the realm then searches to find the user’s entry in Active Directory.

Setup:

  • Elasticsearch version: 7.2
  • Three Master Nodes in private subnets
  • Kibana EC2 standalone server in private subnet
  • Logstash running on the standalone application server in private subnet
  • One Internal ALB with host-based routing for Kibana and Elasticsearch Endpoints.
    • kibana.powerupcloud.com → Pointing to Kibana Server
    • elasticsearch.powerupcloud.com → Pointing to ES Masters
  • Active Directory with ESAdmins AD group and a few users added to it which requires Elasticsearch access. Ensure port TCP 389 and UDP 389 are allowed in the AD.

Elasticsearch Nodes Configuration

Ensure to activate X-Pack on the Elastic Stack. It is an Elastic Stack extension that provides security, alerting, monitoring, reporting, machine learning, and many other capabilities. By default, X-Pack is installed when you install the Elasticsearch.

Create the Certificate to be used by Transport Layer Security (TLS) in the Elastic Stack.

/usr/share/elasticsearch/bin/elasticsearch-certutil ca
/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
mkdir -p  /etc/elasticsearch/certs/
cp -r elastic-certificates.p12 /etc/elasticsearch/certs/

Update the Certificate Path in the /etc/elasticsearch/elasticsearch.yml. Add the realm configuration in the same file. The final elasticsearch.yml looks like as shown below:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 0.0.0.0
cluster.name: puc_elasticsearch
node.name: master-1
http.port: 9200

node.master: true
node.data: true
plugin.mandatory: "discovery-ec2"
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: "us-east-1a, us-east-1b, us-east-1c, us-east-1d"
discovery.zen.minimum_master_nodes: 2
discovery.seed_hosts: ["172.31.xx.xx", "172.31.xx.xx", "172.31.xx.xx"]
cluster.initial_master_nodes: ["172.31.xx.xx", "172.31.xx.xx", "172.31.xx.xx"]

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.license.self_generated.type: trial
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12

xpack:
  security:
    authc:
      realms:
        active_directory:
          my_ad:
            order: 1
            domain_name: AD_DNS
            url: ldap://AD_DNS:389
            user_search:
              base_dn: "cn=users,dc=puc,dc=com"
            group_search:
              base_dn: "cn=users,dc=puc,dc=com"
            files:
              role_mapping: "/etc/elasticsearch/role_mapping.yml"

Replace AD_DNS with the domain name associated with the Active Directory.

The trial license of xpack is available only for 30 days. After that, it is mandatory to purchase a license and update the type of license in the configuration.

Update the role mapping file to map the AD group with an existing ES role.

vim /etc/elasticsearch/role_mapping.yml

superuser:
   - "CN=ESAdmins,CN=Users,DC=puc,DC=com"

ESAdmins is the AD Group Name. Replace it as required.

Superuser is an inbuilt available role in Elasticsearch. By this role_mapping, we are mapping the AD Group to the superuser ES role.

Upload the same certificate “ /etc/elasticsearch/certs/elastic-certificates.p12” in the other two nodes as well. You can use scp commands to achieve it.

Add the same xpack configurations in the other two nodes too.

Validate the ES authentication by executing the curl commands as shown in the screenshot below:

Logstash Configuration

Now that we have the AD authentication enabled on the ES nodes in the above section. Update the logstash configuration to authenticate ES with an AD user. Update the logstash.conf file to add the AD user credentials as highlighted in the below config:

vim /etc/logstash/conf.d/logstash.conf

input {
 file {
   path => ["/var/log/nginx/access.log", "/var/log/nginx/error.log"]
   type => "nginx"
 }
  beats {
    port => 5044
  }
}
filter {
 grok {
   match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
   overwrite => [ "message" ]
 }
 mutate {
   convert => ["response", "integer"]
   convert => ["bytes", "integer"]
   convert => ["responsetime", "float"]
 }
 geoip {
   source => "clientip"
   target => "geoip"
   add_tag => [ "nginx-geoip" ]
 }
 date {
   match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
   remove_field => [ "timestamp" ]
 }
 useragent {
   source => "agent"
 }
}
output {
 elasticsearch {
   hosts => ["http://elasticsearch.powerupcloud.com:80"]
   user => "esadmin@puc.com"
   password => "PASSWORD"
   index => "nginx-%{+YYYY.MM.dd}"
 }
 stdout { codec => rubydebug }
}

In the above configuration, Replace the ES Endpoint (elasticsearch.powerupcloud.com), AD user and password. The AD user must exist in the same AD group as specified in the role_mapping.yml.

Restart logstash: “service logstash restart”. Ensure to look at the logs after restart.

Kibana Configuration

Similar to Logstash, update the Kibana configuration to add the AD User Credentials for Elasticsearch endpoint.

vim /etc/kibana/kibana.yml

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://elasticsearch.powerupcloud.com"]
elasticsearch.username: "esadmin@puc.com"
elasticsearch.password: "PASSWORD"

In the above configuration, Replace the ES Endpoint, elastic search.username and elasticsearch.password. The AD user must exist in the same AD group as specified in the role_mapping.yml.

Restart Kibana: service kibana restart

Hit Kibana Endpoint.

Enter the user credentials that exist in the AD group.

Appendix:

If the Xpack license is not activated earlier before enabling AD authentication, you can execute the below commands to start the trail after adding xpack configuration in elasticsearch.yml.

Create a local user:

/usr/share/elasticsearch/bin/elasticsearch-users useradd  priyanka -r superuser -p PASSWORD
curl -u priyanka http://localhost:9200/_xpack/license
curl -X POST -u priyanka "http://localhost:9200/_license/start_trial?acknowledge=true&pretty"

Validate with AD user:

curl -u esadmin@puc.com http://localhost:9200/_xpack/license

The above commands are executed only in one master server.

curl -u esadmin@puc.com http://elasticsearch.powerupcloud.com/_xpack/license

And that’s all. Hope you found it useful.

How to Convert Historical Data into Parquet Format with Date Partitioning

By | Blogs, data | No Comments

Written by: Nagarjun K, Software engineer at powerupcloud technologies

Given the cloud imperative, a lot of organizations migrate their workloads from on-prem/cloud to AWS. However, while migrating old data into AWS S3, organizations find it hard to enable date-based partitioning. Given the inability to retrospectively implement this feature, organizations usually end-up with disparate storage sources within their AWS environment. The blog equips you with some best practices on implementing date-based partitioning in historical data, as well as, provides key guidelines to convert CSV/Json files to Parquet format before migrating your data.

Source: aws.amazon.com

Source: aws.amazon.com

It is common knowledge that Parquet file format is desirable because of size and cost benefits. Hence a recommended approach  for converting old data to Parquet format is crucial from a migration success point of view. To enable this, organizations often explore AWS EMR and DataProc clusters. However, these approaches introduce other challenges such as large cluster size and associated cost for running the clusters. Therefore, a solution that can address these concerns and also rid the organization from cluster administrative chores is deeply valuable. For these reasons, AWS Glue seems to be a prudent choice. Below is the list on interchangeable format conversions supported by Glue:

  • CSV
  • JSON
  • Parquet
  • Avro
  • ORC

Why Parquet?

Data is usually constrained by storage, which has a bearing on costing aspects. Correspondingly, Parquet is a columnar file format and allows unparalleled storage optimization due to its size benefits. Additionally, there are a great deal of options available in the market for compression and encoding of Parquet files. Date warehousing services such as BigQuery and Snowflake support Parquet file format, enabling granular control on performance and cost.

Why Partition?

As discussed above, partitioning files on the basis of date directly confines the amount of data that needs to be processed and, therefore, allows read-optimization. While unpartitioned data can also be queried, the antiquated approach introduces performance and cost inefficiencies. In essence, partitioning helps optimize data that needs to be scanned by the user, enabling higher performance throughputs.

Steps to convert the files into Parquet

 

Step 1: Extract of Old Data

As first steps, extract historical data from the source database along with with headers in CSV format. To enable better readability of data, you may also use Pipe separator(). After structuring the data with Pipe separator, store the CSV file in S3 bucket.

Step 2: Creating Crawlers for Fetching File Meta-data

With the purpose of identifying the schema of CSV files, you need to create and run Crawlers. Find the steps below:

  • Go to AWS Glue home page.
  • After selecting Crawlers section , click “Add crawler”
  • Name your crawler.

  • Select the path of your CSV folder in S3 (Do not select specific CSV files). As a prerequisite, create a folder that includes all your CSV files.

  • As demonstrated below, we give a path name instead of selecting the filename s3://Bucketname/foldername

  • You may add additional data sources, else click “NO”
  • Since the crawlers need both read and write access in order to read the source file and write the parquet file back to S3, you need to create an IAM that allows both read and write access.

  • Set up the crawler as Run as On Demand

  • Enter the database name to create a table schema for the CSV file

Step 3: Running the Crawler

After you successfully create the crawlers, click “Run it Now” and wait for a few minutes. Shortly you will see a new table that has the same schema as your CSV file in the Data Catalog section

  • Here, we see the csv file table created by the crawler

Step 4: Adding the partition columns to Historical data using Athena

  • Once the table is created by the crawler open athena and click “Run query”.

  • As illustrated in the figure below, the Date Column is in yyyy/mm/dd As part of the partitioning procedure, you can separate columns for year, month and day by running the partitioning query:

Step 5: Running ETL for converting to Parquet format

  • Select ETL Section, go to Jobs and click “Add Job”
  • Name your job and select the IAM role(select the role you created in the earlier step).

  • Select the data source created by the crawler

  • Choose your data target as s3

  • The next screen allows column mapping. If you need to remap or remove any column from CSV, you may modify it from this screen.

  • The following screen shows you the Diagram and source code for the job. As a next step, add PartitionKey and mention the column name for year,month and day to enable partition in that order. See example  below: “partitionKeys”:[“year”,”month”,”day”]

  • Save the changes and click “Run Job” button. Standby for a few mins( based on your total data size) to allow the job to complete. You can see the logs from the bottom.

Step6 : Verifying the files in S3.

Go to s3 bucket where you have saved the parquet file.  You will see that there new folders structured in year–month–date format.

Conclusion:

As organizations continue to move workloads on the cloud, there will be considerable increase  in volume, velocity and variety of data. In order to maintain a healthy trade off between cost and performance, measures such as converting to Parquet format and date-based partitioning can help organizations manage their data requirements with more effectively.

The Role of Artificial Intelligence in Building a Better World

By | AI, Artificial Intelligence, Blogs | No Comments

Written by Jeremiah Peter, Solution specialist-Advanced Services Group, Powerupcloud technologies

A Not So Distant Future

As we usher into an era dominated by technological innovation, Artificial Intelligence continues to draw heated debates for its unparalleled ability to automate tasks and eliminate human dependency. The growing cynicism on automation has captured our imagination in leading cinematic marvels such as ‘2001 A Space Odyssey’ and ‘Terminator’. Painting a deeply poignant future, these movies induce fears of a machine-led holocaust sparked by AI’s transcendence into Singularity- a point where AI supersedes human intelligence. Veering away from the dystopian narrative and objectively analyzing the realm of AI, it seems apparent that we can leverage AI for social good without descending into chaos. The call for beneficence in intelligent design is best captured by American computer scientist, Alan Kay’s words- “The best way to predict the future is to invent it”.

The blog presents a new frame of analysis for the responsible use of Artificial Intelligence technologies to augment human and social development. Additionally, the blog also delineates key Machine Learning and Computer Vision (Object Detection) concepts to solve a real-world problem, outlining a discourse for pragmatic solutions with broad social impact.

The Next Frontier

Under the dense canopy of commercial AI clutter, there are several AI initiatives that continue to garner both awe and adulation in the social sciences and humanities spectrum. Cancer detection algorithms, disaster forecast systems and voice-enabled navigation for the visually impaired are a few notable mentions. Although socially-relevant applications have achieved a fair degree of implementational success, they fail to attain outreach at a global level due to a lack of data accessibility and the dearth of AI talent.

Alternatively, innovative technologies that supplement large scale human-assistance programs in enhancing efficacy could be considered a worthwhile undertaking. Infusing computer vision technology in human-centered programs can dramatically improve last-mile coverage, enhance transparency, mitigate risks and measure the overall impact of assistance programs. In the next section, we delve into some core issues that afflict large-scale human assistance programs and underscore the need for technological intervention.

The State of Human-assistance Programs

According to The State of Food Security and Nutrition in the World Report (2019), around 820 million people in the world are hungry and over 66 million of them are under the age of 5. With numbers increasing steeply in most parts of Africa, South America, and Asia, the fate of Sustainable Development Goal of Zero Hunger by 2030 hangs by a thread. Perturbed by the growing scourge, some nations responded by releasing a slew of measures to take corrective action.

One such initiative called The Midday Meal Scheme (MDMS), launched by the government of India in 1995, serves around 120 million children across government and government-aided schools. Recognized as one of the largest food assistance programs in the world, MDMS was laid out with a bold vision to enhance enrolment, retention, and attendance with the overarching aim of improving nutritional status among children in India. However, not including the logistical and infrastructural shortcomings, the initiative loses significant funds to pilferage each year (Source: CAG Report2015). Shackled by a lack of resources, the program struggles to counter aberrant practices with institutional measures and seeks remediation through innovative solutions.

Given the unprecedented magnitude, large-scale human assistance schemes such as MDMS require well-crafted solutions that can instill governance and accountability into their dispensation process. In our constant endeavor to design Intelligent Apps with profound social impact, Powerup experts examined a few libraries and models to carve out a computer vision model that could reign in governance under such programs. The following section explores the pre-requisites for formulating a desirable solution.

Empowering Social Initiatives with Object Detection

Evidently, the success of an AI application is hugely predicated on a well-conceived algorithm that can handle varying degrees of complexity. However, developing a nuanced program from scratch is often a cumbersome and time-intensive process. To accelerate application development, programmers usually rely on pre-compiled libraries, which are frequently accessed code routines used iteratively in the program. After gleaning several open-source image processing libraries (VXL, AForge.Net, LTI-Lib), Powerup team narrowed down on OpenCV for its unique image processing functions and algorithms.

Besides a good library, the solution also requires a robust image classification engine to parse objects and scenes within images. However, despite several key advances in vision technology, most classification systems fail to interact with the complexities of the physical world in that these systems can only identify a limited set of objects under a controlled environment. To develop advanced object detection capabilities, the application needs to be powered by an intelligent model that can elicit a more refined outcome- to make sense of what it sees.

In order to develop a broad understanding of the real-world, computer vision systems require comprehensive datasets that consist of a vast array of labeled images to facilitate object detection within acceptable bounds of accuracy. Apart from identifying the position of the object/s in the image, the engine should also be able to discern the relationship between objects and stitch a coherent story. Imagenet is a diverse open-source dataset that has over a billion labeled images and, perhaps, serves as a foundation for similar computer vision explorations.

Moreover, computer vision systems also hinge on neural networks for developing self-learning capabilities. Though popular deep-learning based models such as R-CNN, R-FCN, and SSD offer ground-breaking features, YOLO (You Only Look Once) stands out for its capabilities in super real-time object detection, clocking an impressive 45 FPS on GPU. The high processing power enables the application to not only interact with images but also process videos in real-time. Apart from an impressive processing capacity, the YOLO9000 is trained on both the ImageNet classification dataset and the COCO detection dataset that enables the model to interact with a diverse set of object classes. We labeled and annotated local food image sets containing items such as rice, eggs, beans, etc. to sensitize the model toward domain specific data.

As demonstrated in the image above, the model employs bounding boxes to identify individuals and food items in the picture. Acting as a robust deterrent against pilferage, the application can help induce more accountability, better scalability, and improved governance.

A New Reckoning

While a seventh of the population goes hungry every day, a third of the food in the world is wasted. The figure serves as a cause for deep contemplation of the growing disparities in a world spawned by industrialization and capitalism. As we stand on the cusp of modern society, phenomena such as unbridled population growth, disease control, climate change and unequal distribution of resources continue to present grave new challenges. Seeking innovative and sustainable solutions, therefore, becomes not just a moral obligation, but also the 21st century imperative.

Aside from the broad benefits, the domain of AI also presents a few substantive concerns that need general oversight. To that effect, the evolving technological landscape presents two inherent risks: wilful misuse (Eg- Cambridge Analytica Case) and unintended consequences (COMPAS- a biased parole granting application).

While concerns such as wilful misuse raise moral questions pertaining to data governance and preservation of user self-determination, risks such as algorithm bias and inexplicability of decision-making expose design loopholes. However, these apprehensions can be largely mitigated through a commonly accepted framework that is vetted by civil society organizations, academe, tech & business community, and policymakers around the globe. Owing to this pressing need, the European Parliament launched AI4People in February 2018 to design a unified charter for intelligent AI-based design. Upholding values such as protection of human self-determination, privacy, and transparency, the initiative is aimed at proposing recommendations, subsequently leading to policies, for ethical and socially preferable development of AI.

Governed by ethical tenets, innovative solutions such as Object Detection can operate within the purview of the proposed framework to alleviate new-age challenges. With reasonable caution and radical solution-seeking, AI promises to be an indispensable vector of change that can transform society by amplifying human agency (what we can do).

Check out next blog post on

Making the Connected Car ‘Real-time Data Processing’ Dream a Reality

By | Analytics, Automation, AWS, Blogs | No Comments

Written by Jeremiah Peter, Solution specialist-Advanced Services Group, Contributor: Ravi Bharati, Tech Lead and Ajay Muralidhar,  Sr. Manager-Project Management at Powerupcloud Technologies

Connected car landscape

Imagine driving your car on a busy dirt road in the monsoon, dodging unscrupulous bikers, jaywalking pedestrians and menacing potholes. Suddenly, a fellow driver makes wild gestures to inform you that the rear door is unlocked, averting an imminent disaster.

In a connected car system, these events are tracked in near real-time and pushed to the driver’s cell phone within seconds. Although the business relevance of real-time car notifications is apparent, the conception of the underlying technology and infrastructure hardly is. The blog attempts to demystify the inner workings of handling data at scale for an Indian automobile behemoth and equips you with a baseline understanding of storing and processing vast troves of data for IoT enabled vehicles.

The paradigm of shared, electric and connected mobility, which seemed a distant reality a few years ago, is made possible through IoT sensors. Laced with tiny data transmitting devices, vehicles can send valuable information such as Battery Percentage, Distance to Empty (DTE), AC On/Off, Door Locked/Unlocked, etc. to the OEM. The service providers use this information to send near real-time alerts to consumers, weaving an intelligent and connected car experience. Timely analysis and availability of data, thus, becomes the most critical success component in the connected car ecosystem.

Before reaching the OEM’s notification system, data is churned through various phases such as data collection, data transformation, data labeling, and data aggregation. With the goal of making data consumable, manufacturers often struggle to set up a robust data pipeline that can process, orchestrate and analyze information at scale.

The data conundrum

According to Industry Consortium 5GAA, connected vehicles ecosystem can generate up to 100 terabytes of data each day. The interplay of certain key factors in the data transmission process will help you foster a deeper understanding of the mechanics behind IoT-enabled cars. As IoT sensors send data to a TCP/IP server, parsers embedded within the servers push all the time series data to a database. The parsing activity converts machine data (hexadecimal) into a human-readable format (Json) and subsequently triggers a call to a notification service. The service enables OEM’s to send key notifications over the app or through SMS to the end-consumer.

Given the scale and frequency of data exchange, the OEM’s earlier set up was constrained by the slow TCP/IP data transfer rate (Sensor data size: TCP/IP- 360 bytes; MQTT- 440 bytes). The slow transfer rate has far-reaching implications over the user experience, delaying notifications by 6-7 minutes. As part of a solution-driven approach, Powerup experts replaced the existing TCP/IP servers with MQTT servers to enhance the data transfer rate. The change affected a significant drop in notification send-time, which is presently calibrated at around 32-40 seconds.

Furthermore, the OEM’s infrastructure presented another unique challenge in that only 8 out of 21 services were containerized. The rest of the services ran on plain Azure VM’s. To optimize costs, automate scalability and reduce operational overhead, all services are deployed on Docker Containers. Containers provide a comprehensive runtime environment that includes dependencies, libraries, framework and configuration files for applications to run. However, containers require extensive orchestration activities to aid scalability and optimal resource management. AWS Fargate is leveraged to rid the OEM’s infrastructure management team of routine container maintenance chores such as provisioning, patching, cluster and capacity management

Moreover, MQTT and TCP IP brokers were also containerized and deployed on Fargate to ensure that all IoT sensor data is sent to the AWS environment. Once inside the AWS environment, sensor data is pushed to Kinesis Stream and Lambda to identify critical data and to call the AWS notification service-SNS. However, the AWS solution could not be readily implemented since the first generation of electric vehicles operated on 2G sim cards, which did not allow change of IP whitelisting configuration. To overcome the IP whitelisting impediment, we set up an MQTT bridge and configured TCP port forwarding to proxy the request from Azure to AWS. Once the first generation vehicles are called back, the new firmware will be updated over-the-air, enabling whitelisting of new AWS IP addresses. The back-handed approach will help the OEM to fully cut-over to the AWS environment without downtime or loss of sensor data.

On the Database front, the OEM’s new infrastructure hinges on the dynamic capabilities of Cassandra DB and PostgreSQL. Cassandra is used for storing Time Series data from IoT sensors. PostgreSQL database contains customer profile/vehicle data and is mostly used by the Payment Microservice. Transactional data is stored in PostgreSQL, which is frequently called upon by various services. While PostgreSQL holds a modest volume of 150 MB Total, the database size of Cassandra is close to 120 GB.

Reaping the benefits

While consumers will deeply benefit from the IoT led service notifications, fleet management operators can also adopt innovative measures to reduce operational inefficiencies and enhance cost savings. Most fleet management services today spend a significant proportion on administrative activities such as maintaining oversight on route optimization, tracking driver and vehicle safety, monitoring fuel utilization, etc. A modern fleet management system empowers operators to automate most of these tasks.

Additionally, preventive maintenance can help operators augment vehicle lifecycle by enabling fleet providers to pro-actively service vehicles based on vehicular telemetry data such as battery consumption, coolant temperature, tire pressure, engine performance and idling status (vehicle kept idle). For instance, if a truck were to break-down due to engine failure, the fleet operator could raise a ticket and notify the nearest service station before the event occurred, cutting down idle time.

Conclusion

With 7000 cars in its current fleet, the OEM’s infrastructure is well-poised to meet a surge of more than 50,000 cars in the near future. Although the connected car and autonomous driving segment still goes through its nascent stages of adoption, it will continue to heavily draw upon the OEM’s data ingestion capabilities to deliver a seamless experience, especially when the connected car domain transcends from a single-vehicle application to a more inclusive car-to-car communication mode. Buzzwords such as two-way data/telematic exchanges, proximity-based communications and real-time feedback are likely to become part of common parlance in mobility and fleet management solutions.

As the concept of the Intelligent Transport System gathers steam, technology partners will need to look at innovative avenues to handle high volume/velocity of data and build solutions that are future-ready. To know more about how you can transform your organization’s data ingestion capability, you can consult our solution experts here.

Transforming Invoice Processing through Automation

By | AI, Automation, Blogs, Image Processing | One Comment

Written by Jeremiah Peter, Solution specialist-Advanced Services Group, Contributor: Amita PM, Associate Tech Lead at Powerupcloud Technologies.

Automation Myth

According to a recent survey by a US-based consultancy firm, organizations spend anywhere between $12 to $20 from the time they receive an invoice until they reconcile it. The statistic is a stark reminder of how organizations, in pursuit of grand cost-cutting measures, often overlook gaping loopholes in their RPA adoption policy- All or nothing!

This blog makes a compelling case for implementing RPA incrementally in strategic processes to yield satisfactory results. Streamlining the invoice management process is, undoubtedly, a judicious leap in that direction.

Unstructured invoice dilemma

In a real-world scenario, data in invoices are not standardized and the quality of submission is often diverse and unpredictable. Under these circumstances, conventional data extraction tools lack the sophistication to parse necessary parameters and, often, present organizations the short end of the stick. 

Consequently, most invoice processing solutions available today fail to reconcile the format variance within the invoices. The Powerup Invoice Processing Application is a simple Web Application (written in HTML and Python) that leverages cloud OCR (Optical Character Recognition) services, to extract text from myriad invoice formats. Powered by an intelligent algorithm, the solution uses the pattern-matching feature to extract data (e.g. Date MM-DD-YYYY) and breaks free from the limitations of traditional data extraction solutions.

A high-level peek into the solution


Picture by Google.com

Driven by a highly user-friendly interface, the Powerup Invoice Processing Application enables users to upload invoices (png, jpg) from their local workstations. The action invokes a seamless API call to Google OCR service, which returns a long string object as API response. A sample of the string is presented below:

Subsequently, the string is converted to a human-readable format through a script, which uses a Python-based Regex library to identify desirable parameters in the invoice such as date, invoice number, order number, unit price, etc. The extracted parameters are passed back to the web application after successful validation. The entire process lasts not more than 10 seconds. The video below demonstrates how Powerup has successfully deployed the complete process:

Another noteworthy feature of the solution is that it seamlessly integrates with popular ERP systems such as SAP, QuickBooks, Sage, Microsoft Dynamics, etc. Given that ERP systems stash critical accounts payable documents (purchase orders, invoices, shipping receipts), a versatile solution requires integration with the organization’s ERP software to complete the automation cycle. 

A brief look at the advantages offered by invoice processing automation can help you assess the value delivered by the solution. 

The Silver-lining

Picture by Google.com

The adoption of Powerup Invoice Processing Application helps organizations reap the following benefits:

  • Deeply optimized invoice processing TAT resulting in quicker payment cycles
  • Up to 40% cost savings in procurement and invoice processing
  • Highly scalable solution that can process multiple invoices in a few minutes
  • Fewer errors and elimination of human data-entry errors
  • Free-form parameter pattern-matching 
  • Easy integration with ERP software
  • Readily implementable solution; no change required from vendor’s end 

Conclusion 

While procurement teams in various organizations struggle to strike a trade-off between low funds dispensation and high-cost savings, measures that enable them to cut expenses and improve efficiencies in the invoicing process are a welcome respite. 

Tools such as the Powerup Invoice Processing Application can help organizations infuse automation and agility into its processes, as well as, knockdown process complexities into manageable parts. Moreover, the time and cost efficiencies achieved in these undertakings can be passed on to other functions that can significantly bolster the organization’s service offerings. To find out how your organization can be positively impacted, sign up for a free demo session here.

Running Kubernetes Workloads on AWS Spot Instances-Part 8

By | AWS, Blogs, Kubernetes | No Comments

Written by Priyanka Sharma, DevOps Architect, Powerupcloud Technologies

Till now we have practised a lot on the OnDemand Nodes of K8s Cluster. This post demonstrates how to use Spot Instances as K8s worker nodes, and shows the areas of provisioning, automatic scaling, and handling interruptions (termination) of K8s worker nodes across your cluster. Spot Instances can save you up to 70–90% cost as compared to OnDemand.Though Spot EKSInstances are cheaper, you cannot run all your worker nodes as Spot. You must have some OnDemand Instances as a backup because Spot Instances can betray you anytime with the interruptions 😉

In this article, we are discussing how you can use Spot Instances on EKS Cluster as well as the cluster you own on EC2 Servers.

Refer to our public Github Repo which contains the files/templates we have used in the implementation. This blog is covering the below-mentioned points:

Kubernetes Operations with AWS EKS

AWS EKS is a managed service that simplifies the management of Kubernetes servers. It provides a highly available and secure K8s control plane. There are two major components associated with your EKS Cluster:

  • EKS control plane which consists of control plane nodes that run the Kubernetes software, like etcd and the Kubernetes API server.
  • EKS worker nodes that are registered with the control plane.

With EKS, the need to manage the installation, scaling, or administration of master nodes is no longer required i.e. AWS will take care of the control plane and let you focus on your worker nodes and application.

Prerequisites

  • EC2 Server to provision the EKS cluster using AWSCLI commands.
  • The latest version of AWSCLI Installed on your Server
  • IAM Permissions to create the EKS Cluster. Create an IAM Instance profile with the permissions attached and assign to the EC2 Server.
  • EKS Service Role
  • Kubectl installed on the server.

Provision K8s Cluster with EKS

Execute the below command to provision an EKS Cluster:

aws eks create-cluster --name puck8s --role-arn arn:aws:iam::ACCOUNT:role/puc-eks-servicerole --resources-vpc-config subnetIds=subnet-xxxxx,subnet-xxxxx,subnet-xxxxxx,securityGroupIds=sg-xxxxx --region us-east-2

We have given private subnets available in our account to provision a private cluster.

Wait for the cluster to become available.

aws eks describe-cluster --name puck8s --query cluster.status --region us-east-2

Amazon EKS uses IAM to provide authentication to your Kubernetes cluster through the AWS IAM Authenticator for Kubernetes(Link in the References section below). Install it using the below commands:

curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator
chmod +x ./aws-iam-authenticator
cp ./aws-iam-authenticator /usr/bin/aws-iam-authenticator

Update ~/.kube/config file which will be used by kubectl to access the cluster.

aws eks update-kubeconfig --name puck8s --region us-east-2

Execute “kubectl get svc”.

Launch Spot and OnDemand Worker Nodes

We have provisioned the EKS worker nodes using a cloud formation template provided by AWS. The template is available in our Github repo as well i.e. provision-eks-worker-nodes/amazon-eks-node group-with-spot.yaml. The template will provision three Autoscaling Groups:

  • 2 ASG with Spot Instances with two different Instance types as given in the parameters
  • 1 ASG with OnDemand Instance with Instance type as given in the parameter

Create a Cloudformation stack and provide the values in the parameters. For the AMI parameter, enter the ID from the below table:

| Region                  |      AMI               | 
|-------------------------| ---------------------- |
| US East(Ohio)(us-east-2)| ami-0958a76db2d150238 |

Launch the stack and wait for the stack to be completed. Note down the Instance ARN from the Outputs.

Now get the config map from our repo.

https://github.com/powerupcloud/kubernetes-spot-webinar/blob/master/provision-eks-worker-nodes/aws-cm-auth.yaml

Open the file “aws-cm-auth.yaml ” and replace the <ARN of instance role (not instance profile)> snippet with the NodeInstanceRole value that you recorded in the previous procedure, and save the file.

kubectl apply -f aws-auth-cm.yaml
kubectl get nodes --watch

Wait for the nodes to be ready.

Kubernetes Operations with KOPS

Kops is an official Kubernetes project for managing production-grade Kubernetes clusters. It has commands for provisioning multi-node clusters, updating their settings including nodes and masters, and applying infrastructure changes to an existing cluster. Currently, Kops is actually the best tool for managing k8s cluster on AWS.

Note: You can use kops in the AWS regions which AWS EKS doesn't support.

Prerequisites:

  • Ec2 Server to provision the cluster using CLI commands.
  • Route53 domain, (for example, k8sdemo.powerupcloud.com) in the same account from where you are provisioning the cluster. Kops uses DNS for identifying the cluster. It adds the records for APIs in your Route53 Hosted Zone.
Note: For public hosted zone, you will have to add the NS records for the above domain to your actual DNS. For example, we have added an NS record for "k8sdemo.powerupcloud.com" to "powerupcloud.com". This will be used for the DNS resolution. For the private hosted zone, ensure to add the VPCs.
  • IAM Permissions to create the cluster resources and update DNS records in Route53. Create an IAM Instance profile with the permissions attached and assign to the EC2 Server.
  • S3 bucket for the state store.
  • Kubectl installed.

Install Kops

Log into the EC2 server and execute the below command to install Kops on the Server:

curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64
chmod +x kops-linux-amd64
sudo mv kops-linux-amd64 /usr/local/bin/kops

Provision K8s Cluster

kops create cluster k8sdemo.powerupcloud.com --ssh-public-key ~/.ssh/id_rsa.pub --master-zones ap-south-1a --zones ap-south-1a,ap-south-1b,ap-south-1a --master-size=t2.medium --node-count=1 --master-count 1 --node-size t2.medium --topology private --dns public --networking calico --vpc vpc-xxxx --state s3://k8sdemo-kops-state-store --subnets subnet-xxxx,subnet-xxxx --utility-subnets subnet-xxxx,subnet-xxxx --kubernetes-version 1.11.4 --admin-access xx.xxx.xxxx.xx/32 --ssh-access xx.xxx.xxx.xx/32 --cloud-labels "Environment=DEMO"

Refer to our previous blog for the explanation of the arguments in the above command.

kops update cluster --yes

Once the above command is successful, we will have a private K8s Cluster ready with Master and Nodes in the private subnets.

Use the command “kops validate cluster CLUSTER_NAME” to validate the nodes in your k8s cluster.

Create Instance Groups for Spot and OnDemand Instances

Kops Instance Group helps in the grouping of similar instances which maps to an Autoscaling Group in AWS. We can use the “kops edit” command to edit the configuration of the nodes in the editor. The “kops update” command applies the changes to the existing nodes.

Once we have provisioned the cluster, we will have two Instance groups i.e. One for master and One for Nodes. Execute the below command to get available Instance Groups:

kops get ig

Edit nodes instance group to provision spot workers. Add the below Key Values. Set the max price property to your bid. For example, “0.10” represents a spot-price bid of $0.10 (10 cents) per hour.

spec:
...
maxPrice: "1.05"
nodeLabels:
lifecycle: Ec2Spot
node-role.kubernetes.io/spot-worker: "true"

The final configuration will look like as shown in the below screenshot:

Create one more Spot Instance Group for a different instance type.

kops create ig nodes2 --subnet ap-south-1a,ap-south-1b --role Node
kops edit ig nodes2

Add maxPrice and node labels and the final configuration will look like as shown in the below screenshot:

Now, we have configured two spot worker node groups for our cluster. Create an instance group for OnDemand Worker Nodes by executing the below command:

kops create ig ondemand-nodes --subnet ap-south-1a,ap-south-1b --role Node

kops edit ig ondemand-nodes

Add node labels for the OnDemand Workers.

Also, we have added taints to avoid the pods from OnDemand Worker Nodes. Preferably, the new pods will be assigned to the Spot workers.

To apply the above configurations, execute the below command:

kops update cluster
kops update cluster --yes
kops rolling-update cluster --yes

Cluster Autoscaler

Cluster Autoscaler is an open-source tool which automatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:

  • there are pods that failed to run in the cluster due to insufficient resources
  • there are nodes in the cluster that have been underutilized for an extended period of time and their pods can be placed on other existing nodes.

CA will run as a daemonset on the Cluster OnDemand Nodes. The YAML file for daemonset is provided in our Github Repo i.e. https://github.com/powerupcloud/kubernetes-spot-webinar/tree/master/cluster-autoscaler.

Update the following variables in the cluster-autoscaler/cluster-autoscaler-ds.yaml

  • Autoscaling Group Names of On-demand and Spot Groups
  • Update minimum count of instances in the Autoscaling group
  • Update max count of instances in the Autoscaling group
  • AWS Region
  • mode selector will ensure to run the CA pods on the OnDemand nodes always.

Create ClusterAutoscaler on both of the k8s clusters EKS as well as the cluster provisioned using kops. Ensure to attach the below permissions to the IAM Role assigned to the cluster worker nodes:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeAutoScalingInstances",
"autoscaling:DescribeLaunchConfigurations",
"autoscaling:DescribeTags",
"autoscaling:SetDesiredCapacity",
"autoscaling:TerminateInstanceInAutoScalingGroup"
],
"Resource": "*"
}
]
}

The daemonset YAML file for the EKS Cluster will look like as shown in the below screenshot.

Similarly, for the cluster provisioned using Kops, the yaml file will be :


Create the DaemonSet.

kubectl create -f cluster-autoscaler/cluster-autoscaler-ds.yaml

Now create a pod disruption budget for CA which will ensure to run atleast one cluster autoscaler pod always.

kubectl create -f cluster-autoscaler/cluster-autoscaler-pdb.yaml

Verify the Cluster autoscaler pod logs in kube-system namespace:

kubectl get pods -n kube-system
kubectl logs -f pod/cluster-autoscaler-xxx-xxxx -n kube-system

Spot Termination Handler

The major fallbacks of a Spot Instance are –

  • it may take a long time to become available (or may never become available),
  • and maybe reclaimed by AWS at any time.

Amazon EC2 can interrupt your Spot Instance when the Spot price exceeds your maximum price, when the demand for Spot Instances rises, or when the supply of Spot Instances decreases. Whenever you are opting for Spot, you should always be prepared for the interruptions.

So, we are creating one interrupt handler on the clusters which will run as a daemonset on the OnSpot Worker Nodes. The workflow of the Spot Interrupt Handler can be summarized as:

  • Identify that a Spot Instance is being reclaimed.
  • Use the 2-minute notification window to gracefully prepare the node for termination.
  • Taint the node and cordon it off to prevent new pods from being placed.
  • Drain connections on the running pods.
  • To maintain desired capacity, replace the pods on remaining nodes

Create the Spot Interrupt Handler DaemonSet on both the k8s clusters using the below command:

kubectl apply -f spot-termination-handler/deploy-k8-pod/spot-interrupt-handler.yaml

Deploy Microservices with Istio

We have taken a BookInfo Sample application to deploy on our cluster which uses Istio.

Istio is an open platform to connect, manage, and secure microservices. For more info, see the link in the References section below. To deploy Istio on the k8s cluster, follow the steps below:

wget https://github.com/istio/istio/releases/download/1.0.4/istio-1.0.4-linux.tar.gz
tar -xvzf istio-1.0.4-linux.tar.gz
cd istio-1.0.4

In our case, we have provisioned the worker nodes in private subnets. For Istio to provision a publically accessible load balancer, tag the public subnets in your VPC with the below tag:

kubernetes.io/cluster/puck8s:shared

Install helm from the link below:

https://github.com/helm/helm

kubectl create -f install/kubernetes/helm/helm-service-account.yaml
helm init --service-account tiller --wait
helm install --wait --name istio --namespace istio-system install/kubernetes/helm/istio --set global.configValidation=false --set sidecarInjectorWebhook.enabled=false
kubectl get svc -n istio-system

You will get the LoadBalancer endpoint.

Create a gateway for the Bookinfo sample application.

kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

The BookInfo sample application source code, Dockerfile, and Kubernetes deployment YAML files are available in the sample-app directory in our Github repo.

Build a docker image out of provided Dockerfiles and update the IMAGE variable in k8s/deployment.yaml for all the four services. Deploy each service using:

kubectl apply -f k8s

Hit http://LB_Endpoint/productpage. you will get the frontend of your application.

AutoScaling when the Application load is High

If the number of pods increases with the application load, the cluster autoscaler will provision more worker nodes in the Autoscaling Group. If the Spot Instance is not available, it will opt for OnDemand Instances.

Initial Settings in the ASG:

Scale up the number of pods for one deployment, for example, product page. Execute:

kubectl scale --replicas=200 deployment/productpage-v1

Watch the Cluster Autoscaler manage the ASG.

Similarly, if the application load is less, CA will manage the size of the ASG.

Note: We dont recommend to run the stateful applications on Spot Nodes. Use OnDemand Nodes for your stateful services.

and that’s all..!! Hope you found it useful. Happy Savings..!!

References:

Building your first Alexa Skill — Part 1

By | AI, Alexa, Blogs, Machine Learning, ML | No Comments

Written by Tejaswee Das, Software Engineer, Powerupcloud Technologies

Technological advancement in the area of Artificial Intelligence & Machine Learning has not only helped systems to become more intelligent but has also made them more vocational. You can just speak to the phone & add items to your shopping list or just instruct your laptop to read your email. In this fast-growing era of voice-enabled automation, Amazon’s Alexa enabled devices are changing the way people go through their daily routines. In fact, it has introduced a new term in the dictionary, Intelligent Virtual Assistant (IVA).

Technopedia defines Intelligent Virtual Assistant as an engineered entity residing in software that interfaces with humans in a human way. This technology incorporates elements of interactive voice response and other modern artificial intelligence projects to deliver full-fledged “virtual identities” that converse with users.”

Some of the most commonly used IVAs are Google Assistant, Amazon Alexa, Apple Siri, Microsoft Cortana, with Samsung Bixby joining the already brimming list lately. Although IVAs seem to be technically charged, they bring enormous automation & value. Not only do they make jobs for humans easier, but they also optimize processes and reduce inefficiencies. These systems are so seamless, that just a simple voice command is required to get tasks completed.

The future of personalized customer experience is inevitably tied to “Intelligent Assistance”. –Dan Miller, Founder, Opus Research

So let’s bring our focus to Alexa, Amazon’s IVA. Alexa is Amazon’s cloud-based voice service, which can interface with multiple devices on Amazon. Alexa gives you the power to create applications, which have the capability to interact in natural language, making your systems more intuitive to interact with technology. Its capabilities mimic those of other IVAs such as Google Assistant, Apple Siri, Microsoft Cortana, and Samsung Bixby.

The Alexa Voice Service (AVS) is Amazon’s intelligent voice recognition and natural language understanding service that allows you to voice-enable any connected device that has a microphone and a speaker.

Powerupcloud has worked on multiple use-cases, where they have developed Alexa voice automation. One of the most successful & adopted use cases being one of the largest General Insurance providers.

This blog series aims at giving a high-level overview of building your first Alexa Skills. It has been divided into two parts, first, covering the required configurations for setting up the Alexa skills, while the second focuses on the approach for training the model and programming.

Before we dive in to start building our first skill, let’s have a look at some Alexa terminologies.

  • Alexa Skill — It is a robust set of actions or tasks that are accomplished by Alexa. It provides a set of built-in skills (such as playing music), and developers can use the Alexa Skills Kit to give Alexa new skills. A skill includes both the code (in the form of a cloud-based service) and the configuration provided on the developer console.
  • Alexa Skills Kit — A collection of APIs, tools, and documentation that will help us work with Alexa.
  • Utterances — The words, phrases or sentences the user says to Alexa to convey a meaning.
  • Intents — A representation of the action that fulfils the user’s spoken request.

You can find the detailed glossary at

https://developer.amazon.com/docs/ask-overviews/alexa-skills-kit-glossary.html

Following are the prerequisites to get started with your 1st Alexa skill.

  1. Amazon Developer Account (Free: It’s the same as the account you use for Amazon.in)
  2. Amazon Web Services (AWS) Account (Recommended)
  3. Basic Programming knowledge

Let’s now spend some time going through each requirement in depth.

We need to use the Amazon Developer Portal to configure our skill and build our model which is a necessity.

  • Click on Create Skill, and then select Custom Model to create your Custom Skill.

Please select your locale carefully. Alexa currently caters to English (AU), English (CA), English (IN), English (UK), German (DE), Japanese (JP), Spanish (ES), Spanish (MX), French (FR), and Italian (IT). We will use English (IN) while developing the current skill.

  • Select ‘Start from Scratch’
  • Alexa Developer Console
  • Enter an Invocation Name for your skill. Invocation name should be unique because it identifies Skills. Invocation Name is what you say Alexa to invoke or activate your skill.

There are certain requirements that your Invocation name must strictly adhere to.

  • Invocation name should be two or more words and can contain only lowercase alphabetic characters, spaces between words, possessive apostrophes (for example, “sam’s science trivia”), or periods used in abbreviations (for example, “a. b. c.”). Other characters like numbers must be spelt out. For example, “twenty-one”.
  • Invocation names cannot contain any of the Alexa skill launch phrases such as “launch”, “ask”, “tell”, “load”, “begin”, and “enable”. Wake words including “Alexa”, “Amazon”, “Echo”, “Computer”, or the words “skill” or “app” are not allowed. Learn more about invocation names for custom skills.
  • Changes to your skill’s invocation name will not take effect until you have built your skill’s interaction model. In order to successfully build, your skill’s interaction model must contain an intent with at least one sample utterance. Learn more about creating interaction models for custom skills.
  • Endpoint — The Endpoint will receive POST requests when a user interacts with your Alexa Skill. So this is basically the backend for your Alexa Skill. You can host your skill’s service endpoint either using AWS Lambda ARN, which is recommended, or a simple HTTPS endpoint. Advantages of using an AWS Lambda ARN are :
  • Sign in to AWS Management Console at https://aws.amazon.com/console/
  • Lookup for Lambda in AWS services
  • US East (N. Virginia)
  • EU (Ireland)
  • US West (Oregon)
  • Asia Pacific(Tokyo)

We are using Lambda in the N.Virginia (us-east-1) region.

  • Once we are in a supported region, we can go ahead to create a new function. There are three different options for creating your function. You can create a function from scratch or you can also use available Blueprints and Serverless Application Repositories.
  • C# / .NET
  • Go
  • Java
  • NodeJS
  • Python

We will discuss programming Alexa with different languages in the next part of this series.

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html

  • Go back to the Endpoint section in Alexa Developer Console, and add the ARN we had copied from Lambda in AWS Lambda ARN Default Region.

ARN format — arn:aws:lambda:us-east-1:XXXXX:function:function_name

In, part 2, we will discuss the training our model — adding Intents & Utterances, finding walkarounds for some interesting issues we faced, making workflows using dialog state, understanding the Alexa Request & Response JSON, and finally our programming approach in Python.