Category

Data Case Study

Data Analytics helping real-time decision making

By | Data Case Study | 2 Comments

Customer: The fastest-growing cinema business in the Middle East

Summary

The customer is the fastest-growing cinema business in the Middle East wanted to manage the logs from multiple environments by setting up centralized logging and visualization, this was done by implementing the EKK(Amazon Elasticsearch, Amazon Kinesis and Kibana) solution in their AWS environment.

About Customer

The customer is a cinema arm of a leading shopping mall, retail and leisure pioneer across the Middle East and North Africa. They are the Middle East’s most innovative and customer-focused exhibitor, and the fastest and rapidly growing cinema business in the MENA region.

Problem Statement

The customer’s applications generate huge amounts of logs from multiple servers, if any error occurs in the application it is difficult for the development team to get the logs or view the logs in real-time to troubleshoot the issue. They do not have a centralized location to visualize logs and get notified if any errors occur.

In the ticket booking scenario, by analyzing the logs that are generated by the application, an organization can enable valuable features, such as notifying the developers that error occurred in the application server while customers are booking the ticket. If the application logs can be analyzed and monitored in real-time, developers can be notified immediately to investigate and fix the issues.

Proposed Solution

Powerup built a log analytics solution on AWS using ElasticSearch as the real-time analytics engine. AWS Kinesis firehose pushes the data to ElasticSearch. In some scenarios, the Customer wanted to transform or enhance data streaming before it is delivered to ElasticSearch. Since all the application logs are in an unstructured format in the server, the customer wanted to filter the unstructured data and transform it into JSON before delivering it to Amazon Elasticsearch Service. Logs from Web, App and DB were pushed to Elasticsearch for all the six applications.

Amazon Kinesis Agent

  • The Amazon Kinesis Agent is a stand-alone Java software application that offers an easy way to collect and send data to Kinesis Streams and Kinesis Firehose.
  • AWS Kinesis Firehose Agent – daemon installed on each EC2 instance that pipes logs to Amazon Kinesis Firehose.
  • The agent continuously monitors a set of files and sends new data to your delivery stream. It handles file rotation, checkpointing, and retry upon failures. It delivers all of your data in a reliable, timely, and simple manner.

Amazon Kinesis Firehose

  • Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture, transform, and load streaming data into Amazon Kinesis Analytics, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools and dashboards that you’re already using today.
  • Kinesis Data Firehose Stream – endpoint that accepts the incoming log data and forwards to ElasticSearch

Data Transformation

Kinesis Data Firehose can invoke your Lambda function to transform incoming source data and deliver the transformed data to destinations. When you enable Kinesis Data Firehose data transformation, Kinesis Data Firehose buffers incoming data up to 3 MB by default. Kinesis Data Firehose then invokes the specified Lambda function asynchronously with each buffered batch using the AWS Lambda synchronous invocation model. The transformed data is sent from Lambda to Kinesis Data Firehose. Kinesis Data Firehose then sends it to the destination when the specified destination buffering size or buffering interval is reached, whichever happens first.

ElasticSearch

  • Elasticsearch is a search engine based on the Lucene It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
  • Store, analyze, and correlate application and infrastructure log data to find and fix issues faster and improve application performance. You can receive automated alerts if your application is underperforming, enabling you to proactively address any issues.
  • Provide a fast, personalized search experience for your applications, websites, and data lake catalogs, allowing users to quickly find relevant data.
  • Collect logs and metrics from your servers, routers, switches, and virtualized machines to get comprehensive visibility into your infrastructure, reducing mean time to detect (MTTD) and resolve (MTTR) issues and lowering system downtime.

Kibana

Kibana is an open-source data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features such as histograms, line graphs, pie charts, heat maps, and built-in geospatial support. Also, it provides tight integration with Elasticsearch, a popular analytics and search engine, which makes Kibana the default choice for visualizing data stored in Elasticsearch.

  • Using Kibana’s pre-built aggregations and filters, you can run a variety of analytics like histograms, top-N queries, and trends with just a few clicks.
  • You can easily set up dashboards and reports and share them with others. All you need is a browser to view and explore the data.
  • Kibana comes with powerful geospatial capabilities so you can seamlessly layer in geographical information on top of your data and visualize results on maps.

Ingesting data to ElasticSearch using Amazon Kinesis Firehose.

Kinesis Data Firehose is part of the Kinesis streaming data platform, along with Kinesis Data Streams, Kinesis Video Streams, and Amazon Kinesis Data Analytics. With Kinesis Data Firehose, you don’t need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Kinesis Data Firehose to transform your data before delivering it.

Record

The data of interest that your data producer sends to a Kinesis Data Firehose delivery stream. A record can be as large as 1000 KB.

Data producer

Producers send records to Kinesis Data Firehose delivery streams. For example, a web server that sends log data to a delivery stream is a data producer. You can also configure your Kinesis Data Firehose delivery stream to automatically read data from an existing Kinesis data stream, and load it into destinations.

Writing Logs to Kinesis Data Firehose Using Kinesis Agent

  • Amazon Kinesis Agent is a standalone Java software application that offers an easy way to collect and send data to Kinesis Data Firehose. The agent continuously monitors a set of files and sends new data to your Kinesis Data Firehose delivery stream.
  • The agent handles file rotation, checkpointing, and retry upon failures. It delivers all of your data in a reliable, timely, and simple manner. It also emits Amazon CloudWatch metrics to help you better monitor and troubleshoot the streaming process.
  • The Kinesis Agent has been installed on all the production server environments such as web servers, log servers, and application servers. After installing the agent, we need to configure it by specifying the log files to monitor and the delivery stream for the data. After the agent is configured, it durably collects data from those log files and reliably sends it to the delivery stream.
  • Since the data in the servers are unstructured and the customer wanted to send the specific format of data to ElasticSearch and visualize it on Kibana. So we configured an agent to preprocess the data and deliver the preprocessed data to AWS Kinesis Firehose. Preprocessed configuration used in the Kinesis Agent

MatchPattern

  • Since the data in the logs are unstructured and needed to filter some specific records from the data. So we used the match pattern to send the record to filter the data and send it to Kinesis Firehose.
  • The agent has configured in a way to capture the unstructured data using regular expression and send it to the AWS Kinesis Firehose.

An Example How we filtered the data and sent it to the kinesis firehose.

  • LOGTOJSON configuration with Match Pattern

Sample Kinesis agent configuration:

{

    "optionName": "LOGTOJSON",

    "logFormat": "COMMONAPACHELOG",

    "matchPattern": "^([\\d.]+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3})",

    "customFieldNames": ["host", "ident", "authuser", "datetime", "request", "response"]

}

The record in the server before conversion:


100.189.189.89 - - [27/Oct/2000:09:27:09 -0400] "GET /java/javaResources.html HTTP/1.0" 200

After conversion:

{

"Host":"100.189.189.89",

"Ident":null,

"Authuser":null,

"datetime":"27/Oct/2000:09:27:09 -0400",

"request":"GET /java/javaResources.html HTTP/1.0",

"Response":"200"

}

The record in the server has been converted to JSON format. The Match pattern only captures the data in the data according to regular expression and sends the data to AWS Kinesis Firehose. AWS Kinesis Firehose sends the data to Elasticsearch and can be visualized on the Kibana.

Business Benefits

  • Powerup Team successfully implemented the real-time centralized log analytics solution using AWS kinesis firehose and ElasticSearch.
    • Kinesis agent was used to filtering the applications and kinesis firehose streams the logs to Elasticsearch.
    • Separate indexes were created for all 6 applications in  Elasticsearch based on access log and error log.
    • A Total of 20 dashboards were created in Kibana based on error types, for example, 4xx error, 5xx error, cron failure, auth failure.
    • Created Alerts were sent to the developers using AWS SNS. when the configured thresholds, so that developers can take immediate actions on the errors generated on the application and server.
    • Developer log analysis time has greatly decreased from a couple of hours to a few minutes.
  • The EKK setup implemented for the customer is a total log-analysis platform for search, analyses and visualization of log-generated data from different machines and perform centralized logging to help identify any server and application-related issues across multiple servers in the customer environment and correlate the logs in a particular time frame.
  • The data analysis and visualization of EKK setup have benefited the management and respective stakeholders to view the business reports from various application streams which led to easy business decision making.

Cloud platform

AWS.

Technologies used

Lambda, Kibana, EC2, Kinesis.

Data lake setup aiding rapid insights with regulatory compliance

By | Data Case Study | No Comments

Summary

The customer is a leading US-based medical equipment company catering mainly to cloud-connected medical devices that transform care for people with sleep apnea, COPD and other chronic diseases. They are looking at integrating their MyApp application’s data to MosaIQ Data Lake platform on AWS cloud. MyApp is a self-monitoring sleep therapy progress application used extensively by medical representatives and caregivers.

About Customer

The customer is one of the top medical equipment companies based in San Diego, California. They primarily provide cloud-connectable medical devices for the treatment of sleep apnea, chronic obstructive pulmonary disease (COPD) and other respiratory conditions. It employs more than 7,500 employees worldwide with a presence in more than 120 countries globally that have manufacturing facilities in Australia, France, Singapore and the United States.

Problem Statement

MyApp is the customer’s patient self-monitoring application that helps track patient’s sleep therapy progress both online as well as on smartphones. MyApp facilitates tailored coaching and handy tips to make therapy more comfortable. The Customer wanted to,

  • To integrate MyApp application data to MosaIQ Data Lake platform on AWS.
  • Reuse and replicate data flow of AirView, inclusive of policy, pseudo rules, de-identification, Protected Health Information (PHI) and non-PHI.
  • Build code for data staging, data transformations for regulatory adherence and storage on AWS Simple Storage Service (S3).

Proposed Solution

Powerup to analyze and define the scope of integration. Obtain complete access to AWS development, system integration test and production setups and create AWS services catering to Virtual Private Network (VPC)s, subnets, route tables and Internet gateways. Define fixed and incremental S3 buckets for PHI as well as non-PHI accounts.

Ensure that a detailed definition of MyApp S3 policies including source connections and scheduling is made available before coding in the development environment. Also, freeze all policies and pseudo rules for PHI and non-PHI data encryption until coding completion and migration to test environment.

 Implement Data Migration Service (DMS) to migrate data from on-prem to AWS cloud storage S3. Data with all the files to be pushed inside a single folder per table in the S3 bucket via lambda functions. CDC to be implemented for incremental data transfer to S3 event which in turn will trigger and push the requests to Amazon Simple Queue Service (SQS).

Leverage Fargate containers to run scripts in order to check data against the IDs. Run Electronic Medical Records (EMR) cluster by applying masking logic to this data which is sent for further analytics. Identify and save the same in S3 buckets. The next step is to create a test strategy for unit and integration tests.

Powerup DevOps to configure Complement Fixation Test (CFT) and implement continuous integration and continuous deployment (CI/CD) process for MyApp migration. Create integration test scripts, test CI/CD process before the actual system integration migration (SIT), prepare migration to development and UAT environments and devise automation.

The next task is to migrate to SIT through Ci/CD to validate all the resources and execute full load and schedule trigger for CDC load before moving to production deployment. Repeat the process in the production environment and perform UAT.

Post the integration, Powerup took up the responsibility of architectural assessment and went ahead with the Well-Architected Review (WAR) framework. WAR is an architectural assessment based on AWS framework that is built on five pillars – operational efficiency, reliability, security, performance efficiency and cost optimization.

Powerup identified the workload to be reviewed and once relevant data were identified, reviews were arranged with the stakeholders at the company. Review could be conducted onsite or remotely. A report aligning with AWS best practices, categorized as critical, needs improvement or meets best practices were generated for the selected workload. The report highlights the priority with which remediation should be carried out.

 

Benefits

MyApp application data has been integrated to MosaIQ on AWS cloud successfully. This platform can now provide capabilities to wider business team communities as MosaIQ is a data lake platform built on top of AWS stack and stores structure and unstructured data in raw format. It assists in the rapid discovery of actionable insights to improve patient care and business outcomes while maintaining security and regulatory compliance.

MosaIQ platform allows analytics, engineers, and data scientists to respond more efficiently and provide timely information to support better business decisions. This is mainly because data segregation is more organized and bifurcated for PHI and non-PHI data.

Reusable design from MyApp integration can be utilized for similar use cases across the company. A significant improvement in performance was noticed due to features like scalability and reduction of in-memory processing.

Cloud platform

AWS.

Technologies used

AWS S3, Lambda, AWS Glue, AWS EMR, AWS DynamoDB, AWS Step Function, AWS CloudFormation, AWS DMS + CDC.

Managed Data Lake on Cloud Improved Driver Notification by 95%

By | Data Case Study, Data Lake | No Comments

Customer: The pioneer of Electric vehicles and related technologies in India.

 

Summary

The customer is the pioneer of Electric vehicles and related technologies in India, involved in designing and manufacturing of compact electric vehicles. The moving a fully managed & scalable infrastructure and configuration shift, resulted in a cost saving of 30%.

About Customer

The customer is the pioneer of Electric Vehicle technology in India. Their mission is to bring tomorrow’s movement today. They have a wide variety of electric vehicles and will be increasing this range even further with products spanning across personal and commercial segments. Their design supports the new paradigm of shared, electric, and connected mobility. Currently, the total number of connected cars is at 7000 and is further expected to grow to 50,000.

Problem Statement

The customer was looking at,

  • A fully managed and scalable Infrastructure set up and configuration on AWS.
  • Application and services migration from current Azure set up to AWS.
  • Setting up of an Extract, Transform, Load (ETL) pipeline for analytics along with managed data lake.
  • Availability and structure of historical data similar to live data for analytics and
  • Framework for near real-time notification.

They were also eyeing at maintaining the reliability of data in Postgres and Cassandra as well as on it’s back up server.

Proposed Solution

All application microservices and MQTT/TCP IoT brokers will be containerized and deployed on AWS Fargate. All latest IoT sensor data will be sent to the AWS environment. IoT Sensor data will be pushed to a Kinesis stream and a Lambda function to query the stream to find the critical data (low battery, door open, etc) and call the notification microservice. Old sensor data to be sent to the Azure environment initially due to existing public IP whitelisting. MQTT bridge and TCP port forwarding to be done to proxy the request from Azure to AWS. Once the old sensors are updated fully cut-over to AWS.

Identity Access Management (IAM) roles to be created to access different AWS services. The network is to be setup using the Virtual Private Cloud (VPC) service with appropriate CIDR range, subnets, and route tables created. Network Address Translation (NAT) gateway is setup to enable internet access for servers in the private subnet and all Docker Images will be stored in Elastic Container Registry (ECR).

AWS Elastic Container Service (ECS), Fargate, is used to run the docker containers to deploy all the container images on the worker nodes. ECS task definitions are configured for each container to be run. In Fargate the control plain and worker nodes are managed by AWS. The scaling, highly available (HA) services, and patching is handled by AWS as well.

Application Load Balancer (ALB) will be deployed as the front end to all the application microservices. ALB will forward the request to the Kong API gateway which in turn will route the requests to the microservices. Service level scaling will be configured in Fargate for more containers to spin up based on load. AWS Elasticache, a managed service with Redis Engine will be deployed across multiple Availability Zone (AZ) for HA, patching, and updates.

Aurora PostgreSQL will be used to host the PostgreSQL database. SQL dump will be taken from Azure PostgreSQL Virtual Machine (VM) and then restored on Aurora. 3 Node Cassandra cluster, of which 2 will be running in one AZ and the remaining ones in the second AZ will be setup for HA. A 3-node ElasticSearch cluster will also be setup using the AWS managed services.

 

 

In the bi-directional notification workflow, TCP and MQTT gateways will be running on EC2 machines and Parser application on a different EC2 instance. AWS Public IP addresses will be whitelisted on the IoT Sensor during manufacturing for the device to securely connect to AWS. The Gateway Server will push the raw data coming from the sensors to a Kinesis Stream. The Parser server will push the converted and processed data to the same or another Kinesis stream.

Lambda function will query the data in the Kinesis stream to find the fault or notification type data and will invoke the notification Microservice/ SNS to notify the customer team. This reduces the current notification time from 6-8 minutes to almost near real-time. The plan is to have Kinesis Firehose as a consumer reading from the Kinesis streams to push processed data to a different S3 bucket. Another Firehose will push the processed data to Cassandra Database and a different S3 bucket.

AWS Glue will be used for data aggregation previously done using Spark jobs to push the data to a separate S3 bucket. Athena will be used to query on the S3 buckets and standard SQL queries work with Athena. Dashboards will be created using Tableau.

 

 

Cloud platform

AWS.

Technologies used

Cassandra, Amazon Kinesis, Amazon Redshift, Amazon Athena, Tableau.

Business benefit

  • The customer can send and receive notifications in real-time & time taken to send notifications to the driver is reduced by 95%. Using AWS, applications can scale on a secure, fault-tolerant, and low-latency global cloud. With the implementation of the CI/CD pipeline, the customer team is no longer spending its valuable time on mundane administrative tasks. Powerup helped the customer achieve its goal of securing data while lowering cloud bills and simplifying compliance.
  • API Gateway proved to be one of the most beneficial services offered by AWS with its wide range of functionalities as it helped Powerup to address customer’s issues.
  • There was a parallel and collaborative effort from the customer and the Powerup team on containerization of the microservices.
  • Data-driven business decisions taken by the customer team helped in easier movement of data and eliminated the repetitive process.
  • 30% – Cost savings with new architecture

Data lake implementation improved processing time by 4X for India’s largest media company

By | Alexa, Case Study, Data Case Study | No Comments

Customer: India’s largest media company

Summary

The customer is one of India’s leading media and entertainment companies. They were looking to improve their ad placements across channels for improved conversion along with taking other parameters like social media feedback into consideration. With the push towards digital content, shifting the on-premise infrastructure set up to cloud was necessary to optimize costs and manage high volumes of data. Powerup was also to support and maintain the deployed AWS services for them.

About Customer

The customer is one of the substantial News network houses in India with rights to more than 3,818 movie titles, it entertains over 1 billion viewers across 172+ countries offering 80+ channels. The customer has always provided quality entertainment across the globe and is committed to achieving maximum efficiency in their ad conversion rates through strategic advertisement placements across their multiple channels.

Problem Statement

The customer’s current on-premise infrastructure was proving expensive due to the volume of data being generated and the shift to the cloud was the need of the hour. Softwares provided TRP information on a weekly basis whereas some reports needed to be generated every 6 to 12 minutes from the source to destination. All the players in the business generated these reports to aid critical

decisions and the customer, with some major failure in its existing scheduled-based processes were facing further delays. The time is taken to generate reports while making changes in promos and ad placements were proving to be highly critical.

The customer is looking at a fully managed and scalable Infrastructure setup and configuration on AWS. We proposed to create a data lake on AWS with all the source data getting pushed to a commonplace. All data warehouse objects are to be created as in the existing system. Migrate existing SQL Server Integration Services (SSIS) extract, transform, load (ETL) jobs on Talend for the new data to start moving to data warehouse along with the movement of Tableau dashboards on AWS and point to Redshift, Redshift spectrum.

Proposed Solution

A plan was drafted to make the shift from a tight-knit synchronous architecture to an event-based loosely coupled asynchronous architecture in order to ensure accurate and on-time report generation as per the user’s requirement.

Their entire process transformation involved a cloud-first approach where the client gathered on-premise data from multiple sources like SAP, Chrome feeds, Twitter feeds, social media feedback in excel files and then piped it to cloud. This gave birth to data warehouses where data got extracted and moved from physical to digital form.

AWS landing zone to be setup with the following – organization account, centralized logging, shared services, security and production accounts. The shared service account used to deploy common applications like Bastion and Tableau server whereas the security account is created only for audit purposes.

Appropriate users, groups and permissions created using Identity and Access Management (IAM) service to access different AWS service along with Multi-factor authentication (MFA) activation. The network is setup using Virtual Private Cloud (VPC) with appropriate Classless inter-domain routing (CIDR) range, subnets and route table creation.

VPN tunnel is setup between AWS and customer location. One-time data transfer will be done directly to Amazon Simple Storage Service (S3) after which, the backup file on S3 will be restored in Amazon Relational Database Service (RDS) and then the same will be moved to Redshift entirely. Ultimately S3 will have the entire dump, which will be deployed on Amazon Elastic Compute Cloud (EC2) on AWS.

Once collated as one single repository, the data could be easily transformed from raw to columnar format using Lambda functions that can then be smoothly pushed and visualized on Tableau.

An extract, transform, load (ETL) tool like Talend can be used to transfer incremental data to S3. An SSH File Transfer Protocol (SFTP) server can be used to upload the excel files to S3. Alternatively, Talend can be used to extract the data from excel files and load it into S3. Active Directory Federation Services (ADFS) is configured to provide federated access to Tableau server as on-premise AD has employees as well as third party vendors added. Glue Crawlers will run periodically and scan the S3 data lake to automatically populate structured as well as unstructured data in S3, which in turn can be connected with Amazon Redshift and all other data warehouses being used.

CloudWatch service will be used for monitoring and Amazon Simple Notification Service (SNS) will be used to notify the users in case of alarms or metrics crossing thresholds. All snapshot backups will be regularly taken and automated based on best practices.

Security and Logging

It was ensured that the system also had a built-in centralized log system which, kept a check on various parameters like time taken for each process, success/failure of a process, the reason for the failure of the process and so on. Data will be secured and security groups will be used to control traffic at the Virtual Machine (VM) level.

Network Access Control Lists (NACLs) are used to control traffic at the subnet level and VPC flow logs will be enabled to capture the entire network traffic. CloudTrail will be enabled to capture all the Application Program Interface (API) activities. All the logs will be sent to AWS Guard Duty for threat detection and identifying malicious activities in the account and AWS Config will be enabled.

To ease the process, even an EC2 auto-recovery feature is enabled to address failures, if any, so that data is not lost.

The solution was designed in a modular manner keeping in mind the possibility of the addition of new channels and scalability in future, where components could be added or removed without any code changes.

The solution architecture

Application Support

Our support involved components that were developed or modified as a part of the project implementation process.

  • Broadcast Audience Research Council (BARC) sequences related to ETL (extract, transform, load) pipeline functionality support.
  • Data lake support on S3.
  • For Amazon Redshift data warehouse, support on data issues on the stored procedures migrated as a part of the project.
  • Tableau Dashboard support on data links to Redshift.

The customer was to submit requests, prioritize defects and provide technical support to Powerup before planned releases. Based on these inputs, we were able to maintain a backlog of defects reported by client stakeholders. Powerup cloud resources were to plan and schedule each release jointly with the customer’s team. Conduct unit tests and provides support acceptance tests by the customer while fixing business-critical issues if any. Deploy releases to production and warranty support for any defects found in production releases. Conduct weekly project status meetings with client stakeholders to review work progress, planned activities, risks and issues, dependencies and action items, if any.

Opportunities:

Post the shift to cloud, the customer was able to derive Sentiment Analysis through TRP. These TRPs were based on Social Media data on new as well as existing shows. It also paved the way for them to conduct GAP Analysis in order to understand and compare the current Infrastructure and process improvements with potential or expected performance, which helped enhance their efficiency.

Cloud platform

AWS.

Technologies used

Tableau, Redshift, DMS, Glue, Athena.

Business Benefits

  • The customer enjoyed a fully managed and scalable Infrastructure set up and configuration on AWS.
  • 120 dashboards created and data processing time reduced from 2-hr to 30-min.
  • The immediate business impact recorded was the modular solution resulting in the management being able to take improved and timely business-critical decisions.
  • Migration to cloud-enabled a swift generation of critical reports in an end-to-end time span of 3-min from 6-12min which improved the decision-making capability of business leaders to a significant extent. Going forward, it is anticipated that TRP is also to increase further due to this digital shift.

Data Lake on Cloud

By | AWS, Case Study, Data Case Study | No Comments

Customer: One of India’s largest media companies

Problem Statement

One of India’s largest media companies uses various SaaS platforms to run their media streaming application. Hence all of the customers’ data was residing in these SaaS applications. The customer wanted to build a Data Lake to bring all their customers’ and operations’ data at one place to understand their business better

Proposed Solution

Powerup built real-time and batch ETL jobs to bring the data from varied data sources to S3. The raw data was stored in S3. The data was then populated in Redshift for further reporting while advanced analytics was run using Hadoop based ML engines on EMR. Reporting was done using QuickSight.

Cloud platform

AWS.

Technologies used

S3, DynamoDB, AWS ElasticSearch, Kibana, EMR Clusters, RedShift, QuickSight,
Lambda, Cognito, API gateway, Athena, MongoDB, Kinesis.

Optimization of AWS platform, DB consulting services, managed services and DevOps support

By | AWS, Case Study, Cloud Case Study, Data Case Study, DevOps, Managed Services | No Comments

A global logistics group

Problem Statement

A global logistics group with operations in over 14 countries including Singapore,
India, Australia, the US, China, Brazil, Africa, and APAC has its data center on AWS with
more than 70 servers and databases powering close to 20 applications. With global
users from more than 14 countries using these applications, the availability of the
applications are critical to ensure the smooth operations and freight movement.
The customer was seeking an able partner to come in and effectively manage their
cloud-based data center including the databases.

Proposed Solution

Powerup helped the customer to continuously optimize their AWS environment
and provided Database Consulting services. Powerup managed the data center
running on AWS which hosts some of their high critical enterprise workloads like
Oracle Financials, MS SQLServer Enterprise and more.
Powerup also provided 24*7 cloud managed services and DevOps support for the
customer and acts as the integration point for 6 different application development
vendors.

Cloud platform

AWS.

Technologies used

Windows Server, RHEL, Oracle, MS SQLServer, RDP.

Data Warehouse on Cloud

By | Azure, Case Study, Data Case Study | No Comments

A fin-tech company

Problem Statement

The fin-tech customer processes millions of transactions and API calls in a month in
their on-prem setup and this reduced their flexibility in using some advanced cloud
services. The customer was looking to migrate their massive Data Warehouse and
Reporting modules to cloud to reduce the overall time taken for generating insights
and reports on a daily basis.

Proposed Solution

Powerup helped the customer move their 5-year old legacy data and 100+ complex
SSIS jobs to Azure SQL Data Warehouse. Powerup also helped the customer in
optimizing most of their long-running queries & data model, optimizing SSIS jobs &
Tableau jobs to help them run the whole DWH set up effectively in the cloud.

Cloud Platform

Azure

Technologies used

Azure SQL Data Warehouse, SSIS, Tableau.

POS Automation

By | AI Case Study, AWS, Case Study, Data Case Study | No Comments

Joint venture of world’s largest banking and financial services groups

Problem Statement

The customer has a user base of 6 million across India & have a major presence in tier 2
& 3 cities. They have a sales team of more than 8000 field agents that visit
customers for background verification, new policy registrations, etc. Most of their
tasks involve capturing user photo ID details for verification/registration. This
happens either via manual entry or by clicking an image of the photo ID.

Proposed Solution

Powerup developed a POS automation, as an Android application, which allows
sales agents to capture real-time images, performs image pre-processing to improve
the image quality & extracts data real-time for their perusal & sends the data to
backend system for user storage.
The solution helped increase sales team efficiency by more than 15%.

Cloud Platform

AWS.

Technologies used

AWS Rekognition, Python.

Taxi operations reporting

By | AI Case Study, AWS, Case Study, Data Case Study | No Comments

Customer: A taxi aggregator platform

Problem Statement

A taxi aggregator platform in India wanted to build a real-time reporting platform
which will help them track their multi-city taxi operations and more importantly
help them track fraudulent activities in real-time.

Proposed Solution

Powerup built a real-time streaming analytics platform on AWS. The real-time event data was ingested using Kinesis and processed on the fly using Kinesis Streams. The resultant data was persisted to S3 from where it was parallelly loaded to RedShift clusters. Visualization tools talk to RedShift for dashboard rendering.

Cloud platform

AWS.

Technologies used

MongoDB, Athena, RedShift, EMR Clusters, Kinesis, QuickSight.

Restroom feedback reporting

By | Azure, Case Study, Data Case Study | No Comments

One of the world’s largest IT park builders

Problem Statement

One of the world’s largest IT park builders planned to build a comprehensive
Restroom Feedback Reporting Platform to help them assess the conditions of
restrooms across all the assets they manage, understand sentiments of the
customer feedback and improve overall operations.

Proposed Solution

Powerup’s Data Analytics team started with a strong data model, built batch ETL
jobs and Data Warehouse on Azure SQL Data Warehouse. The jobs were automated
to load data on a daily basis and aggregated tables were populated. Reports were
developed on the PowerBI platform with interactive drill-down features.

PowerBI Dashboard

Cloud Platform

Microsoft Azure.

Technologies used

Azure Data Warehouse, PowerBI, SSIS, Azure Data Gateway.