Category

Automation

Making the Connected Car ‘Real-time Data Processing’ Dream a Reality

By | AI, Analytics, Automation, AWS, Blogs | No Comments

Written by Jeremiah Peter, Solution specialist-Advanced Services Group, Contributor: Ravi Bharati, Tech Lead and Ajay Muralidhar,  Sr. Manager-Project Management at Powerupcloud Technologies

Connected car landscape

Imagine driving your car on a busy dirt road in the monsoon, dodging unscrupulous bikers, jaywalking pedestrians and menacing potholes. Suddenly, a fellow driver makes wild gestures to inform you that the rear door is unlocked, averting an imminent disaster.

In a connected car system, these events are tracked in near real-time and pushed to the driver’s cell phone within seconds. Although the business relevance of real-time car notifications is apparent, the conception of the underlying technology and infrastructure hardly is. The blog attempts to demystify the inner workings of handling data at scale for an Indian automobile behemoth and equips you with a baseline understanding of storing and processing vast troves of data for IoT enabled vehicles.

The paradigm of shared, electric and connected mobility, which seemed a distant reality a few years ago, is made possible through IoT sensors. Laced with tiny data transmitting devices, vehicles can send valuable information such as Battery Percentage, Distance to Empty (DTE), AC On/Off, Door Locked/Unlocked, etc. to the OEM. The service providers use this information to send near real-time alerts to consumers, weaving an intelligent and connected car experience. Timely analysis and availability of data, thus, becomes the most critical success component in the connected car ecosystem.

Before reaching the OEM’s notification system, data is churned through various phases such as data collection, data transformation, data labeling, and data aggregation. With the goal of making data consumable, manufacturers often struggle to set up a robust data pipeline that can process, orchestrate and analyze information at scale.

The data conundrum

According to Industry Consortium 5GAA, connected vehicles ecosystem can generate up to 100 terabytes of data each day. The interplay of certain key factors in the data transmission process will help you foster a deeper understanding of the mechanics behind IoT-enabled cars. As IoT sensors send data to a TCP/IP server, parsers embedded within the servers push all the time series data to a database. The parsing activity converts machine data (hexadecimal) into a human-readable format (Json) and subsequently triggers a call to a notification service. The service enables OEM’s to send key notifications over the app or through SMS to the end-consumer.

Given the scale and frequency of data exchange, the OEM’s earlier set up was constrained by the slow TCP/IP data transfer rate (Sensor data size: TCP/IP- 360 bytes; MQTT- 440 bytes). The slow transfer rate has far-reaching implications over the user experience, delaying notifications by 6-7 minutes. As part of a solution-driven approach, Powerup experts replaced the existing TCP/IP servers with MQTT servers to enhance the data transfer rate. The change affected a significant drop in notification send-time, which is presently calibrated at around 32-40 seconds.

Furthermore, the OEM’s infrastructure presented another unique challenge in that only 8 out of 21 services were containerized. The rest of the services ran on plain Azure VM’s. To optimize costs, automate scalability and reduce operational overhead, all services are deployed on Docker Containers. Containers provide a comprehensive runtime environment that includes dependencies, libraries, framework and configuration files for applications to run. However, containers require extensive orchestration activities to aid scalability and optimal resource management. AWS Fargate is leveraged to rid the OEM’s infrastructure management team of routine container maintenance chores such as provisioning, patching, cluster and capacity management

Moreover, MQTT and TCP IP brokers were also containerized and deployed on Fargate to ensure that all IoT sensor data is sent to the AWS environment. Once inside the AWS environment, sensor data is pushed to Kinesis Stream and Lambda to identify critical data and to call the AWS notification service-SNS. However, the AWS solution could not be readily implemented since the first generation of electric vehicles operated on 2G sim cards, which did not allow change of IP whitelisting configuration. To overcome the IP whitelisting impediment, we set up an MQTT bridge and configured TCP port forwarding to proxy the request from Azure to AWS. Once the first generation vehicles are called back, the new firmware will be updated over-the-air, enabling whitelisting of new AWS IP addresses. The back-handed approach will help the OEM to fully cut-over to the AWS environment without downtime or loss of sensor data.

On the Database front, the OEM’s new infrastructure hinges on the dynamic capabilities of Cassandra DB and PostgreSQL. Cassandra is used for storing Time Series data from IoT sensors. PostgreSQL database contains customer profile/vehicle data and is mostly used by the Payment Microservice. Transactional data is stored in PostgreSQL, which is frequently called upon by various services. While PostgreSQL holds a modest volume of 150 MB Total, the database size of Cassandra is close to 120 GB.

Reaping the benefits

While consumers will deeply benefit from the IoT led service notifications, fleet management operators can also adopt innovative measures to reduce operational inefficiencies and enhance cost savings. Most fleet management services today spend a significant proportion on administrative activities such as maintaining oversight on route optimization, tracking driver and vehicle safety, monitoring fuel utilization, etc. A modern fleet management system empowers operators to automate most of these tasks.

Additionally, preventive maintenance can help operators augment vehicle lifecycle by enabling fleet providers to pro-actively service vehicles based on vehicular telemetry data such as battery consumption, coolant temperature, tire pressure, engine performance and idling status (vehicle kept idle). For instance, if a truck were to break-down due to engine failure, the fleet operator could raise a ticket and notify the nearest service station before the event occurred, cutting down idle time.

Conclusion

With 7000 cars in its current fleet, the OEM’s infrastructure is well-poised to meet a surge of more than 50,000 cars in the near future. Although the connected car and autonomous driving segment still goes through its nascent stages of adoption, it will continue to heavily draw upon the OEM’s data ingestion capabilities to deliver a seamless experience, especially when the connected car domain transcends from a single-vehicle application to a more inclusive car-to-car communication mode. Buzzwords such as two-way data/telematic exchanges, proximity-based communications and real-time feedback are likely to become part of common parlance in mobility and fleet management solutions.

As the concept of the Intelligent Transport System gathers steam, technology partners will need to look at innovative avenues to handle high volume/velocity of data and build solutions that are future-ready. To know more about how you can transform your organization’s data ingestion capability, you can consult our solution experts here.

Transforming Invoice Processing through Automation

By | AI, Automation, Blogs, Image Processing | 2 Comments

Written by Jeremiah Peter, Solution specialist-Advanced Services Group, Contributor: Amita PM, Associate Tech Lead at Powerupcloud Technologies.

Automation Myth

According to a recent survey by a US-based consultancy firm, organizations spend anywhere between $12 to $20 from the time they receive an invoice until they reconcile it. The statistic is a stark reminder of how organizations, in pursuit of grand cost-cutting measures, often overlook gaping loopholes in their RPA adoption policy- All or nothing!

This blog makes a compelling case for implementing RPA incrementally in strategic processes to yield satisfactory results. Streamlining the invoice management process is, undoubtedly, a judicious leap in that direction.

Unstructured invoice dilemma

In a real-world scenario, data in invoices are not standardized and the quality of submission is often diverse and unpredictable. Under these circumstances, conventional data extraction tools lack the sophistication to parse necessary parameters and, often, present organizations the short end of the stick. 

Consequently, most invoice processing solutions available today fail to reconcile the format variance within the invoices. The Powerup Invoice Processing Application is a simple Web Application (written in HTML and Python) that leverages cloud OCR (Optical Character Recognition) services, to extract text from myriad invoice formats. Powered by an intelligent algorithm, the solution uses the pattern-matching feature to extract data (e.g. Date MM-DD-YYYY) and breaks free from the limitations of traditional data extraction solutions.

A high-level peek into the solution


Picture by Google.com

Driven by a highly user-friendly interface, the Powerup Invoice Processing Application enables users to upload invoices (png, jpg) from their local workstations. The action invokes a seamless API call to Google OCR service, which returns a long string object as API response. A sample of the string is presented below:

Subsequently, the string is converted to a human-readable format through a script, which uses a Python-based Regex library to identify desirable parameters in the invoice such as date, invoice number, order number, unit price, etc. The extracted parameters are passed back to the web application after successful validation. The entire process lasts not more than 10 seconds. The video below demonstrates how Powerup has successfully deployed the complete process:

Another noteworthy feature of the solution is that it seamlessly integrates with popular ERP systems such as SAP, QuickBooks, Sage, Microsoft Dynamics, etc. Given that ERP systems stash critical accounts payable documents (purchase orders, invoices, shipping receipts), a versatile solution requires integration with the organization’s ERP software to complete the automation cycle. 

A brief look at the advantages offered by invoice processing automation can help you assess the value delivered by the solution. 

The Silver-lining

Picture by Google.com

The adoption of Powerup Invoice Processing Application helps organizations reap the following benefits:

  • Deeply optimized invoice processing TAT resulting in quicker payment cycles
  • Up to 40% cost savings in procurement and invoice processing
  • Highly scalable solution that can process multiple invoices in a few minutes
  • Fewer errors and elimination of human data-entry errors
  • Free-form parameter pattern-matching 
  • Easy integration with ERP software
  • Readily implementable solution; no change required from vendor’s end 

Conclusion 

While procurement teams in various organizations struggle to strike a trade-off between low funds dispensation and high-cost savings, measures that enable them to cut expenses and improve efficiencies in the invoicing process are a welcome respite. 

Tools such as the Powerup Invoice Processing Application can help organizations infuse automation and agility into its processes, as well as, knockdown process complexities into manageable parts. Moreover, the time and cost efficiencies achieved in these undertakings can be passed on to other functions that can significantly bolster the organization’s service offerings. To find out how your organization can be positively impacted, sign up for a free demo session here.

Azure Inventory Details to CSV — Azure Automation

By | Automation, Azure, Blogs, Cloud, Cloud Assessment | No Comments

Written By: Jijo Varghese, Former Cloud Engineer, Powerupcloud Technologies

Below is a step by step by using Azure Automation Hybrid Runbook Worker and a SendGrid account to trigger daily inventory emails. We chose Hybrid runbook instead of a plain Azure automation runbook because we needed local storage to write the CSV file with inventory details before sending it out as an email. So in order to follow this step by step, you need the following resources:

  1. Azure Automation
  2. Operations Management Suite ( OMS )
  3. Windows Machine
  4. Sendgrid Account

Hybrid Runbook Worker feature allows you to run runbooks on machines located in your data center in order to manage local resources. The runbooks are stored and managed in Azure Automation and then delivered to one or more on-premises machines

Installing Hybrid Runbook Worker

Create Operations Management Suite workspace and add Automation solution to Operations Management Suite workspace

  • Open OMS Workspace
  • Click on Solution Gallery
  • Select Automation Hybrid Worker from All Solutions Blade
  • Click Add Solution
  • This will add Hybrid Worker to OMS Workspace

Next, log in to your windows machine that can be used to run this hybrid runbook and Install the runbook environment and connect to Azure Automation.

Open PowerShell session in Administrator mode and run the following commands to import the module

cd "C:\Program Files\Microsoft Monitoring Agent\Agent\AzureAutomation\<version>\HybridRegistration"Import-Module HybridRegistration.psd1
Add-HybridRunbookWorker –Name <String> -EndPoint <Url> -Token <String>

Then run the Add-HybridRunbookWorker cmdlet using the following command. Name is the name of the Hybrid Runbook Worker Group, EndPoint is the URL field in the Azure Automation Account and Token is the Primary Access Key in the Azure Automation account

Create a SendGrid Account

You need SendGrid so that you can send emails. Go to Azure console and look for all services and search for SendGrid.

Give a Name and password for the sendgrid account. Select an existing Subscription and Resource Group. Once provisioning is done, you might want to note down the Username and server name from properties.

Create Azure Automation Account

  • On the hub menu, click New -> Automation Account –> select Automation Account here.
  • Now in Create Automation Account blade, enter the following things: Give a name to Automation Account.
  • Select your Azure subscription from the dropdown box.
  • Select option use existing and then select the resource group from the drop-down box or Create a new resource group
  • Then select the location from where it has to be hosted.
  • Be sure to Create Azure Run as Account. Click Yes to Create an Azure run as Account.

And then click create to launch the Automation Account.

Add Automation Credentials

  • On the Automation Account Blade, Click on Assets -> Credentials -> Add Credentials
  • Give a friendly name. By Default it as Default Azure Credential.
  • Give the username and password of your Azure Credential. Then click Create

Create Automation Runbook

  • On the Automation Account Blade Click On Runbooks-Add Runbook
  • Give a name and select Runbook type as Powershell. Then Click Ok
  • Open the created runbook.
  • Click on Edit and use the following Powershell script to get all running azure resources using Powershell

Before running the Powershell script Select Run Setting as Hybrid Worker and select the Hybrid Worker

Then run the Powershell Script. You can schedule it to Particular day and time. This will automatically run the script at the scheduled time.

This code will generate a .csv file contain all running resources in azure and stores in our Server where our Hybrid worker running. Then Using SMTP it will send the created .csv file to the specified email address.

Apart from inventory, if you would like to track the list of all open ports on a network security group, Powershell below:

That’s it. Hope you found this useful!

Daily AutoScaling Activity Report

By | Automation, AWS, Blogs | No Comments

Written By: Azhagiri Panneerselvam, Cloud Engineer, Powerupcloud Technologies

Ever ran into a situation where you have to figure out how many times autoscaling occurred and triggered new instances since you went offline yesterday night? Well, that’s a good problem to have – that means you have a lot of traffic and you are scaling out often. We ran into this situation for one of our customers and what follows is a way to get a summary of what happened with autoscaling.

Prerequisites

  1. AWS Access Key and Secret key (Read-Only).
  2. Need AWS CLI on your local machine or Server
  3. Tag your AutoScaling Group based on your application like Name: Application Name in the following script I’m getting the AutoScaling name from the AutoScaling group tag.

Configuring AWS CLI

We are going to use CLI to get this done. Configure the AWS CLI with the following command

aws configure

The above command will ask you the access key, Secret key, region and output format for your AWS CLI command to give the appropriate values based on your requirement.

Preparing the list of AutoScaling group Name in your account

Use the following command to get the list of AutoScaling group name.

auto-scaling-details# aws autoscaling describe-auto-scaling-groups --query 'AutoScalingGroups[].[AutoScalingGroupName]'  --output text > /path/for/list-of-autoscaling-group-name.txt

Example

auto-scaling-details# aws autoscaling describe-auto-scaling-groups --query 'AutoScalingGroups[].[AutoScalingGroupName]'  --output text  > /root/auto-scaling-details/list-of-autoscaling-group-name.txt

The above AWS CLI command will get all autoscaling group names in the list-of-autoscaling-group-name.txt file. Will use this file later to get the auto-scaling group activity.

AutoScaling Group Activity Script

Now on to the interesting part. The following Shell Script will get the daily AutoScaling activity in auto-scaling details.csv

In the above Shell Script, you might want to change the path to where you need to save your CSV file and where your AutoScaling group name is located.

Sample output

The above sample output having three data points – application name, instance launched and instance terminated per day. See, I told you its that simple.

sampleapplication,4,2
example.com,2,1

Hope you found this useful. Happy scaling! 🙂

AWS environment automation

By | Automation, AWS, Uncategorized | No Comments

Customer: WeInvest India

Problem Statement

Whenever WeInvest on-boarded a new customer, they had to deploy their
application infra to their AWS account. It consumed a lot of time which delayed the
application deployment. customer was also looking for centralized log
management.

Proposed Solution

Powerup helped WeInvest in reducing the infrastructure provisioning time by
creating the cloud formation templates (infrastructure as code) for both
monolithic and micro services architecture. we helped customer sort centralized
log management by implementing ELK stack.

Cloud platform

AWS.

Technologies used

Cloud Formation template, ECS, RDS, SQS, ELB, Autoscaling.

Automate Updating Thousands of Record Sets in Route53 using Nodejs

By | Automation, AWS, Blogs, Cloud, Cloud Assessment | No Comments

Written By: Priyanka Sharma, Sr. Cloud Engineer at Powerupcloud Technologies.

In our previous post, we have automated the multiple hosted zones creation and importing the record sets into them.

In this blog, we are showing how you can automate updating the resource record set which is already available in Route53. We are going to update the A record IP value to ‘A’ Alias value pointing to the ELB Endpoint.

So, we are already having multiple hosted zones available in our account and one zone has more than 100 records. How you will update the A record if more than 100 records are available? Here’s is the solution:

Listing all the Hosted Zones in a file

The snippet of the script is as follows:

So this script will get the Zone Id of all the hosted zones available and save it in a text file zone.txt. So, after executing this script the zone.txt will have the zone ids in each line as shown below:


/hostedzone/Z1F9TRA0OIUSXK
/hostedzone/Z1QTRDVXPRRM0Q
/hostedzone/Z2DZKSDACI5Q1H
/hostedzone/Z3PPNK2BKN9YSM
……………….

Updating the Record Sets

In the first section of the code, we are reading the zone.txt file which contains the Hosted Zone id of all the zones available. We have used the “line-reader” module of Nodejs which will read each line of the text file one by one and loop through all the zones with storing each line in a variable called “HostedZoneId”.

lineReader.eachLine(‘zones.txt’, function(line, last) {
console.log(line);
var recordparams = {
HostedZoneId: line,
}

Next, for each of the hosted zone available, we have used listResourceRecordSets() API to list all the record sets available for that hosted zone.

route53.listResourceRecordSets(recordparams, function(err, data) {
if (err) console.log(err, err.stack);
else {

Next, we have applied a loop on all the recordsets. So, it will loop through all the record sets available and check for a specific IP. If the IP matches, it will get the recordset name with the matching IP and store it in a variable called “name”.

Once we get the record name for which we need to do the required change, we are passing this name to the params used by the changeResourceRecordSets() API.

So we need to pass the following params for the change:

  • DNS Name in Alias Target: The ELB Endpoint to be passed as the A Alias Record Value.
  • HostedZoneId in Alias Target: This is the ZoneId of the ELB. (You can get the HostedZoneId of the ELB in the EC2 Console)
  • HostedZoneId in ChangeBatch: The hosted zone available in Route53 for which we are going to do that change.

Since there is some limit on changeResourceRecordSets() API (Five requests per second per AWS account. If you submit more than five requests per second, Amazon Route 53 returns an HTTP 400 error), so we are adding a timeout of 2 seconds. Hence, after updating one record, the script will wait for 2 seconds and then proceed to update the next value.

Updating the next 100 Records

changeResourceRecordSets() will update the first 100 records only. So, for the next 100 records, you need to pass the “NextRecordName” in the “StartRecordName” of the params and apply the same logic again. So, it will start looping through the next 100 records and updates the value. The HostedZoneId in params will be the Hosted Zone Id of the zone which has more than 100 records.

Since we have approx 800 records in one hosted zone, so we have applied the same logic 8 times.

The full scripts for a listing hosted zone (listhostedzone.js) and updating recordsets (updaterecordset.js) are available here

I hope it will reduce the manual efforts you put in so far. Happy Automation..!! 🙂

Wish you all a very Happy New Year ..!! 🙂