All Posts By

powerupcloud

40% faster deployments post migrating to cloud along with DevOps transformation

By | Cloud Case Study | No Comments

Customer: The fastest-growing cinema business in the Middle East

Summary

The customer is the fastest-growing cinema business in the Middle East who is intending to migrate their core movie ticket management and booking application on to AWS. Currently, it is a colocation data center that needs to be migrated for better scalability and availability followed by implementation of DevOps optimisation, disaster recovery, log analysis, and managed services.

  • Case – Migration
  • Type –  RePlatform
  • Number of VM’s – 50
  • Number of applications migrated – 5
  • Approximate size of DB – 250 GB

About Customer

The customer is the fastest-growing cinema business in the Middle East, which is the leading shopping mall, communities, retail, and leisure pioneer across the Middle East, Africa, and Asia. They operate over 300 screens across UAE, Oman, Bahrain, Qatar, Lebanon and Egypt and will expand to own and manage 600 screens by the year 2020.

Problem Statement

The customer is planning to migrate their core movie ticket management and booking application from their colocation data center (DC) to AWS for better availability and scalability. Migration to AWS was to facilitate the customer to move their production workload as-is without many architectural changes and then focus on DevOps optimisation.

Powerup proposed a 2 phased approach in migrating the customer applications to AWS.

  • Phase 1 – As-is migration of applications from on-premise set up to AWS cloud along with the migration of databases for better scalability and availability.
  • Phase 2 – Implementation of DevOps optimisation.

Post this, Powerup’s scope of work for the customer also included Disaster Recovery (DR) implementation, setting up EKK for log analysis and AWS managed services.

Proposed Solution

Migration

Powerup will study the customer’s environment and prepare a blueprint of the overall architecture. They will identify potential servers and database failure points and will accordingly fix the automation of backups.

Powerup to work in coordination with the customer team to,

  • export application and data from on-premise architecture to AWS using either Amazon Import/Export functionality or over the internet.
  • Restore data on the cloud and enable database replication between on-premise and Amazon data stores to identify differential data.
  • Implement the monitoring agents and configure backups.
  • Conduct load testing if required as well as system and user acceptance accessibility tests to identify and rectify vulnerabilities.

Post-deployment and stabilization, Powerup completed the automation of the infrastructure using AWS Cloud formation and code deployment automation to save operational time and effort.

Automation

Post phase 1 as-is migration of the customer’s applications, Powerup’s DevOps team will perform weekly manual and automated audits and share reports with the customer team.
Weekly reports on consolidated uptime of applications, total tickets logged, issue resolution details and actionable plans will be shared with the customer team. Powerup will also run a Vulnerability Assessment & Penetration Testing (VAPT) on cloud coupled with quarterly Disaster recovery (DR) drills for one pre-selected application in the cloud platform to ensure governance is in place.
DevOps is also responsible for seamless continuous integration (CI) typically handled by managing a standard single-source repository, automating the build, track the build changes/progress and finally automating the deployment.

Disaster Recovery (DR)

Powerup to understand customer’s compliance, Recovery Point Objective (RPO) and Recovery Time Objective (RTO) expectations before designing the DR strategy.

  • Configure staging VPC, subnets and the entire network set up as per current architecture.
  • Set up network access controls, create NAT gateway and provision firewall for the DR region.
  • Initiate CloudEndure console, enable replication to AWS staging server, create failover replication to DR site from CloudENdure dashboard to conduct DR drill.
  • Verify and analyze RTO and RPO requirements.

EKK set up

The AWS EKK stack (Amazon Elasticsearch Service, Amazon Kinesis and Kibana) act as an alternative to ELK, an open-source tool by Amazon to engage in and visualize data logs. Powerup’s scope involved gathering information on environment and providing access to relevant users to create an AWS ElasticSearch service and AWS Kinesis service. The intent was to install and configure the Kinesis agent on the application servers in order to push the data logs and validate the log stream to the Kibana dashboard. This set up would help configure error and failure logs, alerts, anti-virus logs and transition failures.  The EKK solution is known to provision for analyzing logs and debugging applications. Overall, it helps in managing a log aggregation system. Know more on EKK implementation here.

Managed services

The first step is to study the customer’s cloud environment to identify potential failure points and loopholes if any.

Powerup DevOps team will continuously monitor the customer’s cloud infrastructure health by keeping a check on CPU, memory and storage utilization as well as URL uptime and application performance.

OpenVPN will be configured for the secure exchange of data between the customer’s production setup on AWS and the individual cinema locations. The web, application and database servers are implemented in High Availability (HA) mode. Databases are implemented on Amazon EC2 with replication enabled to a standby server. Relational database service (RDS) may be considered if there are no dependencies from the application end.

Security in the cloud is a shared responsibility between the customer, cloud platform provider and Powerup. Powerup will continuously analyze and assist the customer with best practices around application-level security.

The security monitoring scope includes creating an AWS organization account and proxy accounts with multi-factor authorization (MFA) for controlled access on AWS. Powerup to also set up an Identity and Access Management (IAM), security groups as well as network components on customer’s behalf.

The powerup team has helped setting up the VPN tunnel from AWS to different customer theatre locations [31 different locations].

Enable server-side encryption and manage Secure Sockets Layer (SSL) certificates for the website. Monitor logs for security analysis, resource change tracking, and compliance auditing. Powerup DevOps team to track and monitor firewall for the customer’s environment and additionally mitigate distributed denial-of-service (DDoS) attacks on their portals and websites.

Anti-virus tools and intrusion detection/prevention to be set up by Powerup along with data encryption at the server as well as storage level. Powerup DevOps team will continuously monitor the status of automated and manual backups and record the events in a tracker. In case of missed automatic backups, a manual backup will be taken as a corrective step. Alerts to also be configured for all metrics monitored at the cloud infrastructure level and application level.

Business Benefits

  • Migration helped the customer achieve better scalability and availability.
  • DevOps tool helped automate manual tasks and facilitated seamless continuous delivery while AWS cloud managed services provisioned the customer to reduce operational costs and achieve maximum optimization of workload efficiency.

Cloud platform

AWS.

Technologies used

EC2, S3, ALB, Autoscaling, CodeDeploy, CloudFormation, MS SQL, Jboss, Infinispan Cluster,  Windows, AWS Export/Import.

Data Analytics helping real-time decision making

By | Data Case Study | 2 Comments

Customer: The fastest-growing cinema business in the Middle East

Summary

The customer is the fastest-growing cinema business in the Middle East wanted to manage the logs from multiple environments by setting up centralized logging and visualization, this was done by implementing the EKK(Amazon Elasticsearch, Amazon Kinesis and Kibana) solution in their AWS environment.

About Customer

The customer is a cinema arm of a leading shopping mall, retail and leisure pioneer across the Middle East and North Africa. They are the Middle East’s most innovative and customer-focused exhibitor, and the fastest and rapidly growing cinema business in the MENA region.

Problem Statement

The customer’s applications generate huge amounts of logs from multiple servers, if any error occurs in the application it is difficult for the development team to get the logs or view the logs in real-time to troubleshoot the issue. They do not have a centralized location to visualize logs and get notified if any errors occur.

In the ticket booking scenario, by analyzing the logs that are generated by the application, an organization can enable valuable features, such as notifying the developers that error occurred in the application server while customers are booking the ticket. If the application logs can be analyzed and monitored in real-time, developers can be notified immediately to investigate and fix the issues.

Proposed Solution

Powerup built a log analytics solution on AWS using ElasticSearch as the real-time analytics engine. AWS Kinesis firehose pushes the data to ElasticSearch. In some scenarios, the Customer wanted to transform or enhance data streaming before it is delivered to ElasticSearch. Since all the application logs are in an unstructured format in the server, the customer wanted to filter the unstructured data and transform it into JSON before delivering it to Amazon Elasticsearch Service. Logs from Web, App and DB were pushed to Elasticsearch for all the six applications.

Amazon Kinesis Agent

  • The Amazon Kinesis Agent is a stand-alone Java software application that offers an easy way to collect and send data to Kinesis Streams and Kinesis Firehose.
  • AWS Kinesis Firehose Agent – daemon installed on each EC2 instance that pipes logs to Amazon Kinesis Firehose.
  • The agent continuously monitors a set of files and sends new data to your delivery stream. It handles file rotation, checkpointing, and retry upon failures. It delivers all of your data in a reliable, timely, and simple manner.

Amazon Kinesis Firehose

  • Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture, transform, and load streaming data into Amazon Kinesis Analytics, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools and dashboards that you’re already using today.
  • Kinesis Data Firehose Stream – endpoint that accepts the incoming log data and forwards to ElasticSearch

Data Transformation

Kinesis Data Firehose can invoke your Lambda function to transform incoming source data and deliver the transformed data to destinations. When you enable Kinesis Data Firehose data transformation, Kinesis Data Firehose buffers incoming data up to 3 MB by default. Kinesis Data Firehose then invokes the specified Lambda function asynchronously with each buffered batch using the AWS Lambda synchronous invocation model. The transformed data is sent from Lambda to Kinesis Data Firehose. Kinesis Data Firehose then sends it to the destination when the specified destination buffering size or buffering interval is reached, whichever happens first.

ElasticSearch

  • Elasticsearch is a search engine based on the Lucene It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
  • Store, analyze, and correlate application and infrastructure log data to find and fix issues faster and improve application performance. You can receive automated alerts if your application is underperforming, enabling you to proactively address any issues.
  • Provide a fast, personalized search experience for your applications, websites, and data lake catalogs, allowing users to quickly find relevant data.
  • Collect logs and metrics from your servers, routers, switches, and virtualized machines to get comprehensive visibility into your infrastructure, reducing mean time to detect (MTTD) and resolve (MTTR) issues and lowering system downtime.

Kibana

Kibana is an open-source data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features such as histograms, line graphs, pie charts, heat maps, and built-in geospatial support. Also, it provides tight integration with Elasticsearch, a popular analytics and search engine, which makes Kibana the default choice for visualizing data stored in Elasticsearch.

  • Using Kibana’s pre-built aggregations and filters, you can run a variety of analytics like histograms, top-N queries, and trends with just a few clicks.
  • You can easily set up dashboards and reports and share them with others. All you need is a browser to view and explore the data.
  • Kibana comes with powerful geospatial capabilities so you can seamlessly layer in geographical information on top of your data and visualize results on maps.

Ingesting data to ElasticSearch using Amazon Kinesis Firehose.

Kinesis Data Firehose is part of the Kinesis streaming data platform, along with Kinesis Data Streams, Kinesis Video Streams, and Amazon Kinesis Data Analytics. With Kinesis Data Firehose, you don’t need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Kinesis Data Firehose to transform your data before delivering it.

Record

The data of interest that your data producer sends to a Kinesis Data Firehose delivery stream. A record can be as large as 1000 KB.

Data producer

Producers send records to Kinesis Data Firehose delivery streams. For example, a web server that sends log data to a delivery stream is a data producer. You can also configure your Kinesis Data Firehose delivery stream to automatically read data from an existing Kinesis data stream, and load it into destinations.

Writing Logs to Kinesis Data Firehose Using Kinesis Agent

  • Amazon Kinesis Agent is a standalone Java software application that offers an easy way to collect and send data to Kinesis Data Firehose. The agent continuously monitors a set of files and sends new data to your Kinesis Data Firehose delivery stream.
  • The agent handles file rotation, checkpointing, and retry upon failures. It delivers all of your data in a reliable, timely, and simple manner. It also emits Amazon CloudWatch metrics to help you better monitor and troubleshoot the streaming process.
  • The Kinesis Agent has been installed on all the production server environments such as web servers, log servers, and application servers. After installing the agent, we need to configure it by specifying the log files to monitor and the delivery stream for the data. After the agent is configured, it durably collects data from those log files and reliably sends it to the delivery stream.
  • Since the data in the servers are unstructured and the customer wanted to send the specific format of data to ElasticSearch and visualize it on Kibana. So we configured an agent to preprocess the data and deliver the preprocessed data to AWS Kinesis Firehose. Preprocessed configuration used in the Kinesis Agent

MatchPattern

  • Since the data in the logs are unstructured and needed to filter some specific records from the data. So we used the match pattern to send the record to filter the data and send it to Kinesis Firehose.
  • The agent has configured in a way to capture the unstructured data using regular expression and send it to the AWS Kinesis Firehose.

An Example How we filtered the data and sent it to the kinesis firehose.

  • LOGTOJSON configuration with Match Pattern

Sample Kinesis agent configuration:

{

    "optionName": "LOGTOJSON",

    "logFormat": "COMMONAPACHELOG",

    "matchPattern": "^([\\d.]+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3})",

    "customFieldNames": ["host", "ident", "authuser", "datetime", "request", "response"]

}

The record in the server before conversion:


100.189.189.89 - - [27/Oct/2000:09:27:09 -0400] "GET /java/javaResources.html HTTP/1.0" 200

After conversion:

{

"Host":"100.189.189.89",

"Ident":null,

"Authuser":null,

"datetime":"27/Oct/2000:09:27:09 -0400",

"request":"GET /java/javaResources.html HTTP/1.0",

"Response":"200"

}

The record in the server has been converted to JSON format. The Match pattern only captures the data in the data according to regular expression and sends the data to AWS Kinesis Firehose. AWS Kinesis Firehose sends the data to Elasticsearch and can be visualized on the Kibana.

Business Benefits

  • Powerup Team successfully implemented the real-time centralized log analytics solution using AWS kinesis firehose and ElasticSearch.
    • Kinesis agent was used to filtering the applications and kinesis firehose streams the logs to Elasticsearch.
    • Separate indexes were created for all 6 applications in  Elasticsearch based on access log and error log.
    • A Total of 20 dashboards were created in Kibana based on error types, for example, 4xx error, 5xx error, cron failure, auth failure.
    • Created Alerts were sent to the developers using AWS SNS. when the configured thresholds, so that developers can take immediate actions on the errors generated on the application and server.
    • Developer log analysis time has greatly decreased from a couple of hours to a few minutes.
  • The EKK setup implemented for the customer is a total log-analysis platform for search, analyses and visualization of log-generated data from different machines and perform centralized logging to help identify any server and application-related issues across multiple servers in the customer environment and correlate the logs in a particular time frame.
  • The data analysis and visualization of EKK setup have benefited the management and respective stakeholders to view the business reports from various application streams which led to easy business decision making.

Cloud platform

AWS.

Technologies used

Lambda, Kibana, EC2, Kinesis.

Using AI to make roller coasters safer

By | AI Case Study, Artificial Intelligence | No Comments

Customer: One of the leading integrated resorts

Summary

The customer an integrated resort on the island in Singapore. They offer some world-class attractions one of which is the Battlestar Galactica, the most enticing duel rollercoaster ride at the resort. They decided to cater to preventive maintenance of the wheels of this ride to ensure top-class safety. They planned to adopt Machine Learning (ML) based solution via Google cloud platform (GCP).

Problem Statement

  • The Customer’s Battlestar Galactica ride is financially quite demanding and requires high maintenance.
  • The wheel detection process is time-consuming and a high maintenance manual job.
  • Decision making on the good versus the bad wheel is based on human judgement and expert’s experience.

The ultimate goal was to remove human intervention and automate the decision making on the identification of a bad wheel using machine learning. The machine learning model needed to be trained on currently available data and ingest real-time data over a period of time to help identify patterns of range and intensity values of wheels. This would in turn help in identifying the wheel as good or bad at the end of every run.

Proposed Solution

Project pre-requisites

Ordering of .dat files generated by SICK cameras to be maintained in a single date-time format for appropriate Radio-frequency identification (RFID) wheel mapping. Bad wheel data should be stored in the same format as a good wheel (.dat files) in order to train the classifier. The dashboard to contain the trend of intensity and height values. Single folder to be maintained for Cam_01 and another folder for Cam_02, folder name or location should not change.

Solution

  • Data ingestion and storage

An image capturing software tool named Ranger Studio was used to absorb the complete information on wheels. The Ranger Studio onsite machine generates .dat files for wheels post every run and stores in a local machine. An upload service picks these .dat files from the storage location at pre-defined intervals and runs C# code on it to provide CSV output with range and intensity values.

CSV files are pushed to Google Cloud Services (GCS) using Google Pub/Sub real-time messaging service. The publisher is used to publish files from the local machine using two separate python scripts for Cam01 and Cam02. The subscriber is then used to subscribe to the published files for Cam01 and Cam02.

  • Data Processing

Powerup is responsible to ingest the data into cloud storage or cloud SQL based on the defined format. Processing of data would include the timestamp and wheel run count. There is a pub tracker and a sub tracker maintained to track the files for both cameras so that the subscribed files can be stored on GCS for both the cameras separately. After CSV data is processed, it is removed from the local machine via a custom script to avoid memory issues.

  • Data modelling Cloud SQL

Once data is processed, Powerup to design the data model in cloud SQL where all the data points will be stored in relational format.

The CSV files of individual wheels are then used to train the classifier model. The classifier model is built with an application programming interface named Keras. The trained classifier helps generate a prediction model (.pkl file) to identify good and bad wheels. The prediction model resides on a GCP VM. The generated CSV files are passed through the prediction model and are classified as good or bad based on an accuracy value.

  • Big Query and ML Model

Once the prediction for a wheel is done, the predicted accuracy score, timestamp and wheel information is stored into the Big Query tables. The average wheel accuracy for wheels is then displayed on Google Data Studio.

Powerup to ensure optimization of data performance via tuning and build the ML model. This would enable the customer to obtain large volumes of height and intensity data, post which, they score the ML model with new data.

Current accuracy threshold for SMS trigger is set at 70. Accuracy of prediction is set to improve over a period 6 months when the model has enough bad wheel data reported for training the ML classifier model. SMS will be triggered if the accuracy value is below 70.

SMS will also be triggered if a file is not received from the local machine to Google Cloud Storage via Google Pub/Sub. The reason for file not being received needs to be checked by the client’s SICK team as it may be due to multiple reasons like source file not generated due to camera malfunction, system shutdown or maintenance and so on. Powerup team to be informed about the same as the restart of instances may be required in such cases. Twilio is the service used for SMS whereas SendGrid is used for email notifications.

  • Security and Deployment

Powerup to build a secure environment for all third party integrations. Deploy User Acceptance Test (UAT) environments, conduct regression tests and provide

Go Live Support to off-site applications. The number of servers and services supported with the production was 10 where support included server management in terms of security, network, DevOps, backup DR and audit. Support also included adjusting ML models to improvise training.

 

Limitations

Since the request payload size was higher, Google ML / Online Predictor could not be used. A custom prediction model was built with Keras to overcome this.

Artificial Intelligence

Cloud platform

Google Cloud Platform.

Technologies used

Cloud Storage, Bog Query, Data Studio, Compute Engine.

Business Benefits

Powerup has successfully been able to train the classifier model with a limited set of good and bad wheel real-time data. The accuracy of the model is expected to improve over time. With current data, the accuracy of the model stands at 60% ensuring cost-effectiveness and world-class safety.

Managing the big ‘R’ in cloud migrations

By | Powerlearnings | No Comments

Compiled by Mitu Kumari Senior Executive, Marketing at Powerupcloud Technologies.

Cloud adoption, with migRation at the core, is possibly one of the biggest technology-based organizational changes undertaken. It has a profound impact on a business’s ability to innovate and fully transform the way they operate.

Moving the business applications and data to the cloud is a great strategic move that gives a competitive edge by reducing IT costs. It helps lighten the budget and promotes greater collaboration. It also has the added benefit of disaster risk management and supports continuity of business operations.

However, in spite of the obvious benefits, migRation to the cloud can be a daunting process; and needs to be done in the right way to ensure it supports business needs and delivers the value that it promises.

According to the seventh annual Cisco Global Cloud Index (2016-2021), which focuses on data centre virtualization and cloud computing, both consumer and business applications are contributing to the growing dominance of cloud services over the Internet.

Key highlights & projections of cloud computing:

  • The study forecasts global cloud data center traffic to reach 19.5 zettabytes (ZB) per year by 2021, up from 6.0 ZB per year in 2016.
  • Cloud data center traffic will represent 95 percent of total data center traffic by 2021, compared to 88 percent in 2016.
  • It is expected that by 2021 there will be 628 hyperscale data centers globally, compared to 338 in 2016, 1.9-fold growth or near doubling over the forecast period. 

The key to a successful cloud migRation is in the planning. Creating a migRation strategy is vital and involves identifying the potential options, understanding the interdependencies between applications, and deciding upon what layers of an application would benefit from migRation. Each of these steps provides more opportunities to take advantage of cloud-native features.

Having established a basic understanding of the importance of migRation, let’s try to understand the primary steps to be taken by  organizations and businesses to migrate applications to the cloud.

But, prior to that, an organisation/ business needs to analyze its current applications and systems.

What are the parameters to analyze in order to select the right migRation approach?

There are broadly two considerations – business and technology; for migrating an application to the cloud, which can also affect the choice of a migRation strategy. Using this knowledge, organizations can outline a plan on how they’ll approach migrating each of the applications in their portfolio and in what order.

The 3 main parameters for the business considerations are:

  • Business Fit: When the current application is not a suitable fit for business requirements.
  • Business Value: When the current application does not provide adequate value to the users in terms of information quality and support.
  • Agility: The existing application failing to keep up with the current pace, creating financial and operational risks in the future.

The 3 main parameters for the technical considerations are:

  • Cost: The total cost of ownership for maintenance of the application is higher than its business value. 
  • Complexity: Complexity within current application causes various problems which can be a major factor in maintainability and time, cost and risk of change.
  • Risk: With regards to older applications various level risk exists within the application tech stack or functionality. 

Before an organization begins moving to the cloud, it is vital that IT executives and business leaders take a step back and carefully craft a strategic plan for cloud migRation and application transformation.

Identifying issues in the application layer:

The underlying cause of the issue and its location must be identified within the application. The source of the issue can exist within 3 main aspects of software component: 

  •  Technology 
  • Functionality
  •  Architecture 

Functionality = Likely source for fit and value issues. 

Technology = Common Source for Cost, Complexity and Risk issues Architecture.

 Architecture = Contributor for both and has an impact on complexity and agility.

After having identified and confirmed the application layer issues, the next key step would be to choose the right and most suitable migRation strategy.

How do you choose a migRation strategy, the big ‘R’:

There are 6 primary approaches to choose the most suitable migRation strategy. They are listed and defined as below.

  1. Rehost
  2. Replatform
  3. Repurchase
  4. Refactor/Rearchitect
  5. Retire
  6. Retain

1. Rehosting (lift-and-shift)

The simplest path is the lift and shift, which works just how it sounds.  It’s about simply taking the existing data applications and redeploying them on the cloud servers. This is the most common path for companies new to cloud computing, who are not yet accustomed to a cloud environment & can benefit from the speed of deployment.

Gartner refers to this as rehosting, because it involves moving your stack to a new host without making extensive changes. This enables a rapid, cost-effective migRation, minimal disruption and quick ROI, minimizing the business disruption that could occur if the application was to be refactored.

 However, the lack of modification to your system also prevents you from harnessing certain cloud migRation benefits in the short term.

You shouldn’t treat lift and shift as the end of the migRation story. Very often, applications can be migrated with lift and shift and then, once in the cloud, re-architected to take advantage of the cloud computing platform.

When to choose Rehost approach for cloud migRation:

  • Avoiding expensive investments in hardware: For example, if setting up a data operation center is costing twice compared to the cloud, it is advisable to move the application to the cloud with minor or no modification.
  •  Some applications can be easily optimized once migrated to the cloud: For those applications, it is a good strategy to first move them to the cloud by adopting the rehost approach as it is and then optimize.
  •  In the case of commercially off-the-shelf applications: It is impossible to do code changes on those applications. In this case, it is a better idea to adopt rehost.
  •  MigRation of applications that need to keep running: For organizations choosing to move to the cloud and having some applications that just need to keep running without disruption or modification, rehost is a good

2. Re-platforming (lift-tinker-and-shift)

 This is a good strategy for organizations that aren’t ready for expansion or configuration, or those that want to build trust in the cloud before making a commitment.

Re-platforming is really a variation of lift and shift, here you might make a few cloud (or other) optimizations in order to achieve some tangible benefit, you aren’t otherwise changing the core architecture of the application, but use cloud-based frameworks and tools that allow developers to take advantage of the cloud’s potential.

While this and migRation has some cost associated, it is sometimes a significant savings when compared to the cost of rebuilding the existing legacy system. 

When to choose Re-platform approach for cloud migRation:

  •  Organizations willing to automate tasks that are essential to operations, but are not the business priorities. 
  • If for moving an application to cloud, the source environment is not supported, then a slight modification is required.
  •  Organizations looking to leverage more cloud benefits other than just moving the application to the cloud. 
  • Organizations that are sure that minor changes won’t affect the application functioning.

3. Re-purchase (Drop & Shop)

Repurchasing is another fast way to access cloud technologies. Software as a service (SaaS) can take existing data and applications and translate them into a comparable cloud-based product. This can help make significant progress with operations such as customer relationship management and content management. 

Repurchasing involves a licensing change where the on-premise version is being swapped for the same application but as a cloud service. Dropping on-premise applications and switching to the cloud can offer improved feature sets, without the complexities of rehosting the existing application. The approach is a common choice for legacy applications that are incompatible with the cloud. 

When to choose Re-purchase approach for cloud migRation:

  • Use this approach for legacy applications incompatible with the cloud: If you find existing applications that could benefit from the cloud but would be difficult to migrate using “lift and shift”, “drop and shop” could be a good option.
  • Many commercial off the shelf (COTS) applications are now available as Software as a Service (SaaS): Repurchasing, an excellent and fast way to access cloud-based SaaS that is tailored to your business needs by the cloud provider.

4. Re- architecting (Re-factoring)

This strategy calls for a complete overhaul of an application to adapt it to the cloud. It is valuable when you have a strong business need for cloud-native features, such as improved development agility, scalability or performance.

Highly customized applications that provide a key business differentiator should be re-architected to take advantage of cloud-native capabilities. 

Re-architecting means a rebuild of your applications from scratch to leverage cloud-native capabilities you couldn’t otherwise, such as auto-scaling or serverless computing.  It is  the most future-proof for companies that want to take advantage of more advanced cloud features.

When to choose Re-architect approach for cloud migRation:

  • When restructuring of the code is required to take full advantage of cloud capability.
  • When there is a strong business need for adding features and performance to the application and that is not possible within the existing framework.
  • When an organization is looking to boost agility or improve business continuity, the re-architecting strategy is a better
  • When an organization is willing to move to a service-oriented architecture, this approach can be used.

5. Retire

In today’s data centers there are oftentimes several workloads that are no longer used but have been kept running.  This can have lots of causes, but in some cases, the best thing to do to a workload is to simply turn it off.

Care should be taken to ensure that the service is decommissioned in a fashion that is in line with your current procedure of retiring a platform, but oftentimes migRation is a great time to remove deprecated technology from your service catalog.

When to choose Retire approach as part of cloud migRation:

  •  In many cases, during a migRation project – identify applications that are redundant, and shutting them down can represent a cost-saving.
  •  There may already be existing plans to decommission the application or consolidate it with other applications.

6. Retain

In some cases when a server or IT service is still required and cannot be migrated to the cloud it makes the most sense to retain that server or service in its current position. This Retain methodology is used in a hybrid cloud deployment that uses some on-premises IT servers and services combined with cloud technologies to offer a seamless user experience.  

While it might at times make sense to retain a technology, doing so is only advisable in select circumstances.  The impact of retaining some technologies on-premises is usually an increased demand for hybrid connectivity.

When to choose Retain approach as part of cloud migRation:

  • The business is heavily invested in the on-premise application and may have currently active development projects.
  • Legacy operating systems and applications are not supported by cloud environments.
  • The application is working well-no business case for the cost and disruption of migRation.
  • For industries that must adhere to strict compliance regulations that require that data is on-premise.
  • For applications that require very high performance, the on-premise option may prove the better choice.

Managing the big R:

So in conclusion, which is the best approach to cloud migRation?

Different use-cases have different requirements, so there is no “one size, fits all”. Selecting one among the six migRation approaches is  finding the best that suits your specific needs. That said, there is a way to determine whether one of these three cloud migRation approaches will suit you better than the others.

While choosing an approach for cloud migRation, to improve the technology, architecture, & functionality of the IT infrastructure, one must always keep in consideration the cost and risk associated with the chosen approach.

Start by checking if the app can be moved to a cloud environment in its entirety while maintaining running costs and keeping operational efficiency in check. If the answer is yes, starting with a rehost is a good idea.

If rehosting is not an option or if cost-efficiency is at a level that needs to be improved, you can also consider replatforming as the approach to take. Remember that not all apps can be migrated this way, so you may end up having to find other solutions entirely.

For workloads that can easily be upgraded to newer versions, the repurchase model might allow a feature set upgrade as you move to the cloud.

The same can be said for refactoring. When there is a strong business need for refactoring to take full advantage of cloud capability, which is not possible with the existing applications. The time and resources to complete the full refactoring should be taken into consideration.

Some applications simply won’t be required anymore, so it is important to identify these prior to migrating to the cloud & retire, so that you do not end up paying for application infrastructure that is not delivering any business benefit.

Retain portions of your IT infrastructure, if there are some applications that are not ready to be migrated, would produce more benefit when kept On-prem or it was a recent upgrade.

One can certainly take (most of) the hassle out of moving to the cloud with the right cloud migRation strategy. You will be left with the exciting part: finding new resources to use, better flexibility to benefit from, and a more effective environment for your apps.

How CTX used AWS Well-Architected to save infrastructure cost by 70%

By | Cloud Case Study | No Comments

Customer: One of the largest digital asset trading companies

Summary

Cyberdyne Tech Exchange (CTX) is one of the largest digital asset exchange companies who publish and trade-in asset-backed security tokens like artwork, real estate, diamonds, etc. This trading is carried out by qualified issuers and investors through their newly developed applications which they plan to migrate to AWS cloud. The deployed applications must provision for high availability and scalability, automation and security along with a well-architected framework.

About the customer

CTX is one of the largest digital asset exchange companies where qualified issuers and investors publish and trade asset-backed security tokens. These security tokens are backed by curated investment-grade assets such as artwork, diamonds, real estate and equity securities.

Their platform offers a complete suite of services that include primary issuance, trading and settlement as well as custody services.

Global institutional investors trade 24/7 on their trading architecture that is powered by Nasdaq’s matching and market surveillance (SMARTS) engines. Clients have the assurance that they are trading on an institutional-grade platform with fair and transparent price discovery.

Problem Statement

CTX intends to deploy their newly developed application to AWS. The deployed application should adhere to the following:

  • Highly Available & Scalable environment
  • AWS Well-Architected Framework
  • MAS & PDPA compliance
  • Automated Infrastructure Provisioning
  • Automated Application deployment

Proposed Solution

Architecture Description

  • The AWS accounts are provisioned based on the landing zone concept, with a centralized logging account, security account, shared service account and separate accounts for different applications.
  • Master Account hosts AWS organization Cost, SCP for member accounts and consolidated billing.
  • Shared service account hosts all common services like Transit Gateway, ECR, Jenkins, Bastion Route 53 etc.
  • Security Account hosts GuardDuty master and all other accounts will be added as members.
  • All security services like IPS & IDS, Cyberark and other future security-related services will be deployed in Security accounts.
  • Centralized Logging Account host all logs like VPC flow logs, ELB logs, CloudTrail, Cloudwatch and Elasticsearch streaming live application logs from all member accounts.
  • DEV Account / UAT Account / Staging Account / Production Account is provisioned to host the application.
  • All the infrastructure provisioning happens through CloudFormation Template. [VPC, EC2, EKS, S3, RDS, TGW, TGW connections]
  • CTX has two major applications – Business System[BS] and Business Application[BA].
  • BS & BA are provisioned in separate VPCs for all the environments.
  • BS is a monolithic application and the applications are deployed on EC2 instances.
  • BA is a service layer and talks to BS systems through API. BA deployed in the EKS cluster.
  • Around 20 microservices are deployed in the EKS cluster.
  • ALB ingress controller has been deployed along with AWS WAF for the application communication from External users to microservices.
  • The application deployment lifecycle is completely automated using JenkinsFile.
  • PostgreSQL RDS is used as a Database.
  • CloudWatch service will be used for monitoring and SNS will be used to notify the users in case of alarms, metrics crossing thresholds etc.
  • All snapshot backups will be regularly taken and automated based on best practices.

Security & Logging

  • AWS SSO is created and appropriate IAM accounts have been formed with least permissible access provided to the accounts.
  • MFA has been enabled on both root and IAM accounts.
  • Except for bastion, all servers will be placed in private subnets.
  • Security groups are used to control traffic at the VM level. Only the required ports will be opened, and access allowed from required IP addresses.
  • Network Access Control Lists (NACLs) are used to control traffic at the subnet level.
  • CloudTrail has been enabled to capture all the API activities occurring in the account.
  • VPC flow logs enabled to capture all network traffic.
  • AWS Guard Duty enabled for threat detection and identifying malicious activities in the account like account compromise.
  • AWS Config enabled, and all the AWS recommended config rules are created.
  • All servers will be encrypted using KMS. KMS keys are stored in Security accounts and admin access to keys are restricted only to Security Admins.

AWS WAF

  • WAF is mandatory for Compliance requirements[MAS].
  • AWS WAF has been used as a Web Application Firewall for the external-facing applications.
  • WAF Managed rules are created to mitigate top 10 OWASP’s web application vulnerabilities.
  • AWS CloudFormation has been created to deploy the WAF rules in all the required environments.
  • The following rules are created using the CloudFormation template.
    • Generic-detect-admin-access
    • Generic-detect-bad-auth-tokens
    • Generic-detect-blacklisted-ips
    • Generic-detect-php-insecure
    • Generic-detect-rfi-lfi-traversal
    • Generic-detect-ssi
    • Generic-enforce-csrf
    • Generic-mitigate-sqli
    • Generic-mitigate-xss
    • Generic-restrict-sizes
  • Web ACL logging has been enabled to capture information about all incoming requests.
  • AWS Cloudwatch has been used to monitor and alert based on WAF rules.
  • AWS WAF has been integrated with ALB. ALB has been provisioned by ALB ingress controller which is deployed in EKS cluster.

Benefits

  • Successfully provisioned the entire application in AWS using CloudFormation with all the necessary security measures as per MAS compliance specifications.
  • Spot instances are used for scalable instances. This saves AWS infrastructure cost by 60% to 70%.
  • Application deployment is completely automated using Jenkins.
  • The highly secured environment has been provisioned with the help of AWS services like AWS WAF, Guard Duty and other third-party solutions like Trend Micro Deep Security Manager.

Cloud platform

AWS.

Technologies/Services used

EC2, RDS, S3, EKS, ECR, TGW, Guard Duty, Route53, ALB, AWS WAF, IAM, KMS.

AWS Lambda Java: Creating Deployment Package for Java 8/11 using Maven in Eclipse – Part 1

By | Powerlearnings | No Comments

Written by Tejaswee Das, Software Engineer at Powerupcloud Technologies Collaborator: Neenu Jose, Senior Software Engineer

Introduction

When we talk about being serveless, AWS Lambda is definitely one that we connect with. It was never that simple before, AWS Lambda has made life easy for Developers and Data Engineers alike. You will hardly find any use-cases involving AWS without Lambda. It’s a nightmare to think about AWS without a Lambda. To know more about my AWS Lambda and some simple S3 events use cases around it,

you should have a look at one of my posts on AWS Lambda

This post is part of a two-part blog series. This part 1 blog will guide you through the steps to create a Java (Java 8/Java 11) deployment package for AWS Lambda in Eclipse using Maven and use S3 Event triggers. 

In Part 2, we will discuss steps on using SES with S3 Event triggers to Lambda.

Use Case

One of our clients had their workloads running on Azure Cloud. They had few serverless applications in Java 8 in Azure Functions. They wanted to upgrade Java from Java 8 to Java 11. Since Java 11 was not supported (Java 11 for Azure Functions has recently been released in Preview), they wanted to try out other cloud services – that’s when AWS Lambda was the one to come into the picture. We did a POC feasibility check for Java 11 applications running on AWS Lambda.

Deployment Steps

Step 1: Install AWS Toolkit in Eclipse

1.1 Open Eclipse → Go to Help → Install New Software

1.2 Enter https://aws.amazon.com/eclipse in Work with and select AWS Toolkit for Eclipse Core(Required)

1.3 Click Next and Install

Note: The toolkit requires Eclipse 4.4 (Luna) or higher.

Step 2: Create New AWS Lambda Function

You might need to restart Eclipse for the installation to reflect.  Add your AWS Access Keys when asked. This step is optional, you can anytime add/remove AWS credentials/accounts from Preferences Menu.

2.1 Go to File → New → Other…

2.2 Select AWS → AWS Lambda Java Project → Next

2.3 Fill in your Project Name and other details

Class Name is your Lambda Handler. In Lambda terms – it’s like the main function of your Project.  You can have anything here – we are using the default name.

For our demo we are using in-built S3 Events. There are a lot of other events to use from like – DynamoDB Event, Stream Request Handler, SNS Event, Kinesis Event, Cognito Event or even Custom if you want to build from scratch.

2.4 Click Finish

For our demonstration & test purposes, you can go with the default code. We are using us-east-1.  Make sure you add the region, you might encounter an error if not added.

Sample Code

package com.amazonaws.lambda.demo;

import com.amazonaws.regions.Regions;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.S3Object;

public class LambdaFunctionHandler implements RequestHandler<S3Event, String> {

	private AmazonS3 s3 = AmazonS3ClientBuilder.standard()
    		.withRegion(Regions.US_EAST_1)
    		.build();

    public LambdaFunctionHandler() {}

    // Test purpose only.
    LambdaFunctionHandler(AmazonS3 s3) {
        this.s3 = s3;
    }

    @Override
    public String handleRequest(S3Event event, Context context) {
        context.getLogger().log("Received event: " + event);

        // Get the object from the event and show its content type
        String bucket = event.getRecords().get(0).getS3().getBucket().getName();
        String key = event.getRecords().get(0).getS3().getObject().getKey();
        try {
            S3Object response = s3.getObject(new GetObjectRequest(bucket, key));
            String contentType = response.getObjectMetadata().getContentType();
            context.getLogger().log("CONTENT TYPE: " + contentType);
            return contentType;
        } catch (Exception e) {
            e.printStackTrace();
            context.getLogger().log(String.format(
                "Error getting object %s from bucket %s. Make sure they exist and"
                + " your bucket is in the same region as this function.", key, bucket));
            throw e;
        }
    }
}

Step 4: Java Runtime Environment (JRE)

4.1 Go to Windows → Preferences → Java → Installed JREs

Step 5: Maven Build

Now to the final step where we will build our deployment package.

5.1 Right click on your project in the Project Explorer → Run As → Maven Build

5.2 Edit Configuration & Launch

Enter ‘package’in Goals. Select your JRE, can leave else as default.

5.3 Run

Your build should happen without any errors for Java 8, but with Java 11, you might run into few errors. Make sure you add updated mockito-core.

In pom.xml of  the generated project change the version of mockito-core

<dependency>
      <groupId>org.mockito</groupId>
      <artifactId>mockito-core</artifactId>
      <version>2.7.22</version> //Change to 3.3.3
      <scope>test</scope>
    </dependency>

This version change is necessary for java 11 build to work.

On successful build, you should see something similar

Sample build

[INFO] Scanning for projects...
[INFO] 
[INFO] ------< com.amazonaws.lambda:demo >----------------------
[INFO] Building demo 1.0.0
[INFO] -----------------[ jar ]---------------------------------
[INFO] 
[INFO] -- maven-resources-plugin:2.6:resources (default-resources) @ demo ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 0 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.6.0:compile (default-compile) @ demo ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ demo ---
[WARNING] Using platform encoding (Cp1252 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-compiler-plugin:3.6.0:testCompile (default-testCompile) @ demo ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ demo ---
[INFO] Surefire report directory: D:\Eclipse Workspace\Maven\demo-blog-s3\target\surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.amazonaws.lambda.demo.LambdaFunctionHandlerTest
Received event: com.amazonaws.services.lambda.runtime.events.S3Event@124ac145
CONTENT TYPE: image/jpeg
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.117 sec

[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ demo ---
[INFO] Downloading from : https://repo.maven.apache.org/maven2/org/apache/maven/maven-archiver/2.5/maven-archiver-2.5.pom
.
.
.
.
.
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing D:\Eclipse Workspace\Maven\demo-blog-s3\target\demo-1.0.0.jar with D:\Eclipse Workspace\Maven\demo-blog-s3\target\demo-1.0.0-shaded.jar
[INFO] Dependency-reduced POM written at: D:\Eclipse Workspace\Maven\demo-blog-s3\dependency-reduced-pom.xml
[INFO] ----------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ----------------------------------------------------------------
[INFO] Total time:  03:10 min
[INFO] Finished at: 2020-08-03T14:53:09+05:30
[INFO] ---------------------------------------------------------------

Look at the highlighted path to locate your .jar file. This is as per your Workspace directory configuration.

My Eclipse Workspace directory here is : D:\Eclipse Workspace\Maven\

Step 6: Create Test S3 Bucket

Create S3 bucket,  for putting the files, from the AWS console.

Make sure this bucket is in the same region where you are planning to create the Lambda Function.

Step 7: Creating Lambda Function in AWS Console

There are a couple of ways of creating Lambda functions. The easiest way is through the AWS Console. Choose Language, create Function and get going. You get a lot of runtimes to choose from. Start writing code on the Go. Works great for interpreted languages – Python, Node, Ruby others.

But for compiler based languages like Java, Go, .NET, you will need to upload the deployment package and do not allow in-line editing.

There are other ways to directly upload Lambda functions from Eclipse itself. We faced issues with that, so to get our task done, we created a deployment package (.jar in Java) and uploaded it to Lambda. Works great.

7.1 S3 Event Triggers & IAM

Please refer to one of my previous posts

Follow solution steps 1&2. Create required Execution roles and attach policies to provide required permissions.

Step 8: Deploying Lambda Function

8.1 All setup done, you just need to upload the maven built jar file in Step 5.3 here.

You can either directly upload if file size is less than 10MB, else you can upload large files using Amazon S3.

Great! Your code is deployed successfully. Time to test it now.

Step 9: Testing

9.1 To test your deployment, Go to S3

9.2 Go to your Bucket that you created in Step 6 and configured trigger in 7.1

Upload a test file

9.3 Go back to your Lambda Function. Click on Monitoring → View logs in CloudWatch

You can see the S3 trigger events log here. When the file was uploaded to that bucket, it triggered Lambda.

Conclusion

This was a very simple proof of concept that we have demonstrated here. This was mainly to get AWS Lambda working with Java 11 for our client. In the next part of this series, we will try to demonstrate some more stuff we can try with AWS Lambda using Java 8/11 – using AWS Java SDK to send emails & notifications on file upload to S3.

Hope this was informative. We had a tough time figuring the correct resources to use, so planned to write this to help folks out there looking for help with different Java versions & AWS Lambda.

References

https://aws.amazon.com/eclipse/

Data lake setup aiding rapid insights with regulatory compliance

By | Data Case Study | No Comments

Summary

The customer is a leading US-based medical equipment company catering mainly to cloud-connected medical devices that transform care for people with sleep apnea, COPD and other chronic diseases. They are looking at integrating their MyApp application’s data to MosaIQ Data Lake platform on AWS cloud. MyApp is a self-monitoring sleep therapy progress application used extensively by medical representatives and caregivers.

About Customer

The customer is one of the top medical equipment companies based in San Diego, California. They primarily provide cloud-connectable medical devices for the treatment of sleep apnea, chronic obstructive pulmonary disease (COPD) and other respiratory conditions. It employs more than 7,500 employees worldwide with a presence in more than 120 countries globally that have manufacturing facilities in Australia, France, Singapore and the United States.

Problem Statement

MyApp is the customer’s patient self-monitoring application that helps track patient’s sleep therapy progress both online as well as on smartphones. MyApp facilitates tailored coaching and handy tips to make therapy more comfortable. The Customer wanted to,

  • To integrate MyApp application data to MosaIQ Data Lake platform on AWS.
  • Reuse and replicate data flow of AirView, inclusive of policy, pseudo rules, de-identification, Protected Health Information (PHI) and non-PHI.
  • Build code for data staging, data transformations for regulatory adherence and storage on AWS Simple Storage Service (S3).

Proposed Solution

Powerup to analyze and define the scope of integration. Obtain complete access to AWS development, system integration test and production setups and create AWS services catering to Virtual Private Network (VPC)s, subnets, route tables and Internet gateways. Define fixed and incremental S3 buckets for PHI as well as non-PHI accounts.

Ensure that a detailed definition of MyApp S3 policies including source connections and scheduling is made available before coding in the development environment. Also, freeze all policies and pseudo rules for PHI and non-PHI data encryption until coding completion and migration to test environment.

 Implement Data Migration Service (DMS) to migrate data from on-prem to AWS cloud storage S3. Data with all the files to be pushed inside a single folder per table in the S3 bucket via lambda functions. CDC to be implemented for incremental data transfer to S3 event which in turn will trigger and push the requests to Amazon Simple Queue Service (SQS).

Leverage Fargate containers to run scripts in order to check data against the IDs. Run Electronic Medical Records (EMR) cluster by applying masking logic to this data which is sent for further analytics. Identify and save the same in S3 buckets. The next step is to create a test strategy for unit and integration tests.

Powerup DevOps to configure Complement Fixation Test (CFT) and implement continuous integration and continuous deployment (CI/CD) process for MyApp migration. Create integration test scripts, test CI/CD process before the actual system integration migration (SIT), prepare migration to development and UAT environments and devise automation.

The next task is to migrate to SIT through Ci/CD to validate all the resources and execute full load and schedule trigger for CDC load before moving to production deployment. Repeat the process in the production environment and perform UAT.

Post the integration, Powerup took up the responsibility of architectural assessment and went ahead with the Well-Architected Review (WAR) framework. WAR is an architectural assessment based on AWS framework that is built on five pillars – operational efficiency, reliability, security, performance efficiency and cost optimization.

Powerup identified the workload to be reviewed and once relevant data were identified, reviews were arranged with the stakeholders at the company. Review could be conducted onsite or remotely. A report aligning with AWS best practices, categorized as critical, needs improvement or meets best practices were generated for the selected workload. The report highlights the priority with which remediation should be carried out.

 

Benefits

MyApp application data has been integrated to MosaIQ on AWS cloud successfully. This platform can now provide capabilities to wider business team communities as MosaIQ is a data lake platform built on top of AWS stack and stores structure and unstructured data in raw format. It assists in the rapid discovery of actionable insights to improve patient care and business outcomes while maintaining security and regulatory compliance.

MosaIQ platform allows analytics, engineers, and data scientists to respond more efficiently and provide timely information to support better business decisions. This is mainly because data segregation is more organized and bifurcated for PHI and non-PHI data.

Reusable design from MyApp integration can be utilized for similar use cases across the company. A significant improvement in performance was noticed due to features like scalability and reduction of in-memory processing.

Cloud platform

AWS.

Technologies used

AWS S3, Lambda, AWS Glue, AWS EMR, AWS DynamoDB, AWS Step Function, AWS CloudFormation, AWS DMS + CDC.

Greenfield Deployment for One of the top biopharmaceutical companies

By | Cloud Case Study | No Comments

Summary

The customer is an international clinical-stage biopharmaceutical company focusing on cellular immunotherapy treatments for cancer is looking at adopting cloud services for the very first time. They plan to structure their database on Google cloud platform. The intention is to enhance performance and have efficient research outputs from their applications especially since they handle large volumes of data. They were also looking at the ability to scale at any point of time during peak loads along with complete automation of continuous integration and continuous deployment (CI/CD) process for easier deployments and better auditing, monitoring and log management.

About Customer

The customer is a clinical-stage biopharmaceutical organization with the scientific vision of revolutionizing the treatment of cancer. They specialize in the research, clinical development and commercialization of cancer immunotherapy treatments. The combination of technologies from its academic, clinical and commercial research partners have enabled the company to create a fully integrated approach to the treatment of cancer with immunotherapy. They plan to work with Powerup to use Google Cloud Platform (GCP) as its cloud platform for their Cancer Research program.

Problem Statement

The customer plans to use Google Cloud Platform (GCP) as its cloud platform for their Cancer Research program. Data scientists will be using a Secure File Transfer Protocol (SFTP) server to upload data on an average of one to two times a month with an estimated data volume of 2-6 TB per month.

The data transferred to GCP has to undergo a two-step cleansing process before uploading it on a database. The first step is to do a checksum to match the data schema against the sample database. The second step is transcoding and transformation of data after which the data is stored on a raw database.

Proposed Solution

Greenfield setup on GCP

Understanding customer needs while also understanding the current python models and workflows to be created were the first steps in initiating this project. Post these preliminary studies and sign-off, a detailed plan and solution architecture document formed a part of the greenfield project deliverables.

The set up included shared services, logging UAT and production accounts. The Cloud Deployment Manager (CDM) was configured to manage their servers, networks, infrastructure and web applications. Cloud Identity and Access Management (IAM) roles were created to access different GCP services as per customer specification, which helped in securely accessing another service.

On-premise connectivity is established via VPN tunnels.

The data scientists team have built nearly 50+ python/R models that help in the data processing. All the models are stored in GitHub currently. Python model will meet performance expectations when deployed and CI/CD pipelines to be created for 48 python models.

Once the data arrives on the database, the customer wants the python code to process the data and store the results on an intermediate database.

Multiple folders were created to deploy production, UAT and management applications. Cloud NAT was set up to enable internet access, Virtual Private Cloud (VPC) peering done for inter-connectivity of required VPCs and SFTP server was deployed on Google Compute Engine.

Once data gets uploaded on the raw GCS, checksum function will be triggered to initiate data cleansing. In the first phase, the data schema will be verified against a sample database after which the data will be pushed to transcoding and transformation jobs. Processed data will be stored to GCS.

All the python/R models will be deployed as a docker image on a Kubernetes cluster that is managed by Google ensuring that GCP is taking care of high availability and scaling.

The customer will have multiple workflows created to process data that in turn would be able to define all the workflows for python model executions.

The customer team will view the current data through a web application.

The processed data also has to be synced back to the on-premise server. An opensource antivirus tool is used to scan and verify data before migrating to Google Cloud Storage (GCS).

Monitoring and Logging

Monitoring tools such as stackdriver for infrastructure and application monitoring as well as log analytics was used as it supported features like tracing, debugging and profiling to monitor the overall performance of the application.

Additional tools such as Sensu to monitor infrastructure, Cloud Audit logging that checks Application Program Interface (API) activities, VPC flow logs to capture network logs and FluxDB as well as Grafana to store data on the database and visualize and create dashboards respectively were utilised.

Stackdriver logging module ensures centralized logging and monitoring of the entire system.

Security and Compliance

IAM with least permissible access and Multi-Factor Authentication (MFA) be enabled as an additional layer of security for account access. The databases won’t have direct access to critical servers like database and app servers. Firewall rules will be configured at the virtual networking level for effective protection and traffic control regardless of the operating system used. Only the required ports will be opened to give access to the necessary IP addresses.

Both data in transit and at rest are by default encrypted in GCP along with provisions for static code analysis and container image-level scanning.

CI/CD pipeline

Setup CI/CD pipeline using Jenkins which is an open-source tool that facilitates modern DevOps environment. It bridges the gap between development and operations by automating building, testing and deployment of applications.

Benefits

After the successful deployment of code, code integration and log auditing got simpler. The customer was able to handle large blocks of data efficiently and auto-scaling at any point of time during new product launches and marketing events became effortless. This improved their performance as well.

The customer was also able to scale up without worrying about storage and compute requirements. They could move into an Opex model on the cloud by paying as per usage.

Moving to GCP enabled the customer to save 20% of their total costs as they could adopt various pricing models and intelligent data tiering.

Cloud platform

GCP.

Technologies used

Shared VPC, Cloud VPN, Compute Engine, Kubernetes Engine, Cloud Storage, Cloud Security Scanner, Cloud IAM, Cloud Security Command Center, Cloud Registry.

Key evaluating factors in deciding the right Cloud Service Provider

By | Powerlearnings | No Comments

Compiled by Kiran Kumar 

Since August 9, 2006, when the then Google CEO Eric Schmidt introduced the term to an industry conference, cloud computing has been driving the IT industry over the past decade and a half through its outright performance, ease of use, and its industry adaptability. Broadly segmented in Iaas, PaaS, SaaS, BPaaS, and Management & Security with the line’s blurring between each it, makes it challenging to find the right fit for your computing needs. So we have listed down a few factors you should consider while evaluating, and what are the core considerations for each. 

  1. Infrastructure setup
  2. The learning curve
  3. The relevance of the service catalog
  4. Data governance and Security
  5. Partner relationships
  6. SLAs
  7. Consistency and Reliability
  8. Back-Up and Support
  9. Cost
  10. Flexibility and Exit strategy

Infrastructure Setup

Infrastructure setup can have a huge impact on your latency, network speeds, data transfer rate, and so on, a diversified infrastructure setup would require a data center (also referred to as Availability Zones) that is closest to your preferred location. 

There are 4 standards of data centers as defined by uptime institute – globally accepted standards for data center planning, the exact guidelines and protocols are not clearly out in public but some of these metrics include redundant electrical path for power, uptime guarantee, cooling capacity, and concurrent maintainability, etc. Tier 4 is the highest standard for data centers. Min requirements for Tier 4 data centers are 

  • 99.995 % uptime in a  year. 
  • 2N+1 infrastructure (two times the amount required for operation plus a backup). 
  • Maximum allowed downtime per year of 26.3 minutes. 
  • 96-hour power outage protection.

Data centers can be easily affected by power outages, earthquakes, tornadoes, lightning strikes, etc. hence require careful planning, try and get a sense of the key design parameters adopted in setting up their data centers to counter such occurrences. Also, make sure to evaluate the cloud provider’s crisis management processes and guidelines as it showcases their ability and how well equipped they are to quickly resolve an ongoing crisis.

If your enterprise is into IoT and edge computing check for highly redundant network connectivity (5g) and low latency services to improve response times and save bandwidth. They are key factors in supporting the edge computing environments like real-time securities market forecasting, autonomous vehicles, and transportation traffic routing, etc. 

Key considerations

  • Location (Availability Zone’s)
  • Datacenter tier
  • Crisis management guidelines and protocols
  • Roadmap for upcoming technology support

The learning curve 

Despite its popularity among enterprises, with almost all Fortune 500 companies having one provider, there is still a shortage of cloud understanding. Cloud transformation can be challenging and demands to be considered as a separate project of its own, the lack of necessary cloud skills can cause inefficient migration leading to unintended consequences and unwanted costs. 

Every organization has its strengths and weaknesses, identify your strengths, and try to build your Cloud infrastructure around it. Start by assessing each cloud provider and the type of offerings available. It is imperative to be cognizant of all skills required for the general operation, governance, compliance amongst others. The storage of data is often an afterthought, stemming from a general absence of protocol-related knowledge. This needs to be addressed through various upskilling programs and strategic talent acquisitions. So it is important to list down for each of those providers how steep the learning curve would be before and after the migration. As a remark, most companies preferred to outsource these functions to specialist managed services providers.

Key considerations

  • Ease of learning
  • Upskilling support from the cloud provider
  • Active communities and partnership 

The relevance of the service catalog

Each cloud provider offers different products and services but it is important to make sure that your cloud goals align with the provider’s vision for improvement. Do your preferences match with the provider’s standards, SLA’s and your security needs, how much re-coding or customization is required at the architectural level to suit your workloads, and what are the associated costs? 

Especially if you are looking for SaaS-based services with a high dependency on a particular application, understanding the service development roadmap, or how they continue to innovate, grow, and support their product over time. Does their roadmap fit your needs in the long term? 

Cloud providers also offer a lot of services to assist you in your cloud transformation, assessment, and planning or in case of a large scale public cloud provider they provide a lot of offers custom made for your organization. In support, they also have a well-established partner community that can help you with all your cloud requirements. 

Key considerations

  • Services in-line with your needs
  • Rich and broad marketplace and an active developer community
  • Compatibility over the long term

Data Governance and Security

Storing data can be tricky simply because of the diversities in the data law across the world. Every organization needs to be aware of the local data regulations and prevailing privacy laws.

Your choice will depend on the cloud provider offering most flexibility and most compliant, also be aware of the provider’s data center location, and verify it.

Data encryption is another factor that needs your attention. Assess what are the different modes of encryption available for both data transit and during storage. Check the provider’s history for any major incidents that have occurred and understand what processes they have in place to quickly resolve the issues. High risk, highly sensitive data can be stored using much more secure and encrypted data storage solutions to and cheaper storage solutions to store some of the less sensitive data like (inventory information, daily logs, etc). 

On the information-security side check what are the compliance standards followed and any recognizable certifications they hold. However, even with all this in place, it is important for the cloud provider to offer the flexibility to support your own security practices and your commitments to your clients.   

Key considerations

  • Flexible data access and management
  • Data Compliance & Security
  • Wide range of data services  

Partner relationships

It is common practice to have a partner ecosystem to guide and facilitate these transitions to the cloud. It is, therefore, important to assess the provider’s relationship with key vendors, their accreditation levels, technical capabilities, number of projects completed, staff certifications, and the overall expertise they bring to the table. Important to note here, expertise in multi-cloud is a bonus. Powerup, for example, is a top-tier partner with the big three cloud service providers.

If you are largely reliant on SaaS-based services, check for the overall compatibility of the product across the platforms, as some of the SaaS-based services are platform-specific. Look for an active marketplace to buy complementary services that are super compatible.

In some of the regions, cloud services are made available through a subcontractor mainly due to the local laws like in the case of China, such interdependencies have to be uncovered and guarantee the primary SLAs stated across all parts of the service.

Key factors

  • Check for accreditations
  • Level of expertise across platforms
  • Relationship with the cloud provider
  • Service compatibility across platforms 

SLAs

Cloud agreements seem complex simply because of the lack of industry standards which defines how these contracts should be constructed. However, ISO/IEC 19086-1:2016 tries to an extent to facilitate a common understanding between cloud service providers and cloud service customers. Usually, agreements are a mixture of commonly agreed general terms and conditions and some negotiated terms.  

Service Levels

Make sure each service objectives like accessibility, service availability or uptime in percentages, service capacity and what is the upper limit in terms of users, connections, resources response time, and deliverables are defined. Be clearly aware of your roles and responsibilities related to delivery, provisioning, service management, monitoring, support, escalations, etc and how responsibilities are split between you and the provider. 

Other scenarios include during an outage or natural disaster the min and max accepted downtime, data loss, or recovery times have to be clearly analyzed against your requirements. Control over data access, data location, confidentiality usage, and ownership rights are crucial check for standards and commitments under data resilience and data backup needs to be in line with your requirements and necessary provisions for a safe exit strategy.

Some of the key business considerations: 

  • Contractual and service governance includes to what extent the provider can unilaterally change the terms of service or contract
  • What are the policies on contract renewals and exits and what are the notice periods?
  • What insurance policies, guarantees, and penalties are included, and some exceptions.

Key considerations

  • Key SLAs
  • How compatible are the terms and conditions with your organization’s goals
  • Are they equipped to support their claims?
  • Are they negotiable?
  • Are the liabilities and responsibilities equally shared?  

Consistency and Reliability 

High availability and reliability are essential for both CSP and the client in maintaining customer’s confidence and preventing revenue losses due to service level agreement (SLA) violation penalties. Cloud computing has appealed to a larger audience in recent years for supporting critical mission systems. However, the lack of consistency in cloud services is quickly becoming a major issue. According to 2018 research reports, about $285 million have been lost yearly due to cloud service failures and offering availability of about 99.91%.

No cloud platform is perfect and downtime may occur more often than not, so try to measure it against their SLAs for the last 6-12 months, this data is mostly available online if not on request. Check what learnings they take back from such occurrences and how consistent they have been in their recovery times as stated in SLAs. Also, ensure monitoring and reporting tools on offer are sufficient and can neatly integrate into your overall management and reporting systems.

Key factors

  • Check for consistency in delivery through past performance.
  • Fault management and reporting systems. 

Back-Up and Support

Check what back-up provisions and processes are in place and understand the limits of their ability to support your data preservation expectations. Roles, responsibilities, escalation processes, and who has the burden of proof, all must be clearly documented in the service agreement. Taking into consideration the increasing levels in criticalness of data, data sources, scheduling, backup, restore, integrity checks, etc. Consider purchasing additional risk insurance if the costs associated with recovery are not covered by the provider’s terms and conditions.

Cloud providers offer a wide range of support services to help their customers out on each step migration, managed services, etc. The support delivery medium offered is also important, where you want them to be available – a phone call, chat, or email? And if they are offered through a partner, the expertise they bring to you. Here staff certification is a good barometer of the quality of the support on offer. 

Key considerations

  • Well equipped multi-channel support services (24/7)
  • Clear documentation around the roles and responsibilities
  • Insurance coverage

Cost

Don’t just go by the list price, as providers might offer services at low cost but may not offer you the optimum level of performance. While that does not mean more expensive is better. The correct way to approach it is by comparing all of them against your core requirements as illustrated above. It is not uncommon to ask for offers, so do so with the ultimate goal to incorporate all the services you need in the desired price range. Studies express that just basic optimization can save about 10% of your cloud cost. Some of the cloud providers have tried and tested ways through which you can save costs. Also through a multi-cloud approach, you can leverage far more value and flexibility, read more about it here

Key considerations

  • Price to value comparison
  • Look for offers
  • Predefined guidelines to save cost 

Flexibility & Exit Strategy

Vendor lock-in is an important factor in most considerations, however, some things cannot be avoided and it’s best that we check if the provider has minimal use of such services. Here is where adopting open source services can be an effective workaround. Also, stay aware of the updates which drastically change the working model policies and technologies of a product to support a particular platform and make sure to have policies in place to counter such situations.

Exiting a CSP can be tricky; it boils down to the exit provisions provided by the service provider and the services levels agreed upon by both parties. All your digital assets starting from your data products and services need to have their own exit strategy which needs to be integrated deeply into your cloud transformation plan. Most organizations don’t include an exit strategy in their cloud adoption roadmap which leads to a lack of preparation, waste of effort, and penalties due to exceeding the exit duration.

Key considerations

  • Ease of transition
  • Support for open source
  • Exit provisions

Azure Data Factory – Self Hosted Integration Runtime Sharing

By | Blogs, data | No Comments

Written by Tejaswee Das, Software Engineer, Powerupcloud Technologies

Contributors: Sagar Gupta, Senior Software Engineer | Amruta Despande, Azure Associate Architect | Suraj R, Azure Cloud Engineer

Introduction

Continuing our discussion on Azure Data Factory(ADF) from our previous blogs. In the past we have discussed ADF and configuration steps for a high availability self hosted integration runtime (IR). You can read more about that here: Azure Data Factory – Setting up Self-Hosted IR HA enabled

This is a quick short post on IR sharing in ADFs for better cost optimization and resource utilization also covers common shortcomings while creating ADF using Terraform and/or SDKs.

Use Case

This is again part of a major data migration assignment from AWS to Azure. We are extensively using ADF to setup ETL pipelines and migrate data effectively – both historical and incremental data.

Problem Statement

 Since the data migration activity involves different types of databases and complex data operations, we are using multiple ADFs to achieve this. Handling private production data required self-hosted IRs to be configured to connect to the production environment. The general best practices for self-hosted IR is a high-availability architecture. An IR can have max 4 nodes – a minimum of 2 nodes for high availability. So here arises the problem – for multiple ADFs how many such self-hosted IRs would one use to power this?

Solution

This is where IR sharing comes into the picture. ADF has this brilliant feature of IR sharing wherein many ADFs can share the same IR. The advantage of this will be price & resource reduction. Suppose you had to run 2 ADFs – one to perform various heavy migrations for AWS RDS MySQL to Azure, and the other one for AWS RDS PostgreSQL. Ideally we would have created 2 different IRs one each able to connect to MySQL & PostgreSQL separately. For a production level implementation, this would mean 2X4 = 8 nodes (Windows VMs). Using IR sharing, we can create one self-hosted IR with 4 nodes and share this IR with both ADFs cutting cost on 4 extra nodes. Please note – The IR node sizing depends on your workloads. That’s a separate calculation. This is only from a high level consideration.

Steps to enable IR sharing between ADFs

Step1: Login to the Azure Portal.

Step 2: Search forData Factories in the main search bar.

Step3: Select your Data Factory. Click on Author & Monitor.

Click on Pencil icon to edit.

Step 4: Click on Connections. Open Management Hub.

Step 5: Click on Integration runtimes to view all your IRs. Select your self-hosted IR for which you want to enable sharing.

Refer to https://www.powerupcloud.com/azure-data-factory-setting-up-self-hosted-ir-ha-enabled/ for detailed information on creating self-hosted IRs.

Step 6: This opens the Edit integration runtime tab on the right side. Go to Sharing and Click on + Grant permission to another Data Factory.

Copy the Resource ID from this step. We will use it in Step 9.

This will list down all ADFs with which you can share this IR.

Step 7: You can either search your ADF or manually enter service identity application ID. Click on Add

Note: You may sometimes be unable to find the ADF from this dropdown list. Even though your ADF lists in the Data Factory page, it does not show up in this list. That will leave you puzzled. Not to worry – such a case might arise when you are creating ADFs using the Azure APIs programmatically or through Terraform. Don’t forget to add the optional identity parameter while creating. This assigns a system generated Identity to the resource.

Sample Terraform for ADF

provider "azurerm" {
    version = "~>2.0"
  features {}
}

resource "azurerm_data_factory" "adf-demo" {
  name                = "adf-terraform-demo"
  location            = "East US 2"
  resource_group_name = "DEMO-ADF-RG"
  identity {
    type = "SystemAssigned"
  }
}

To locate the service identity id of ADF. Go to Data Factories page, select the ADF and click on Properties.

Step 8: Click on Apply for the changes to effect.

Incase you do not have required permissions, you might get the following error

Error occurred when grant permission to xxxxxxxx-xxxx-xxxx-xxx-xxxxxxxxx. Error: {"error":{"code":"AuthorizationFailed","message":"The client 'xxxxxxxx@powerupcloud.com' with object id 'xxxxxxxx-xxxx-xxxx-xxx-xxxxxxxxx' does not have authorization to perform action 'Microsoft.Authorization/roleAssignments/write' over scope '/subscriptions/xxxxxxxx-xxxx-xxxx-xxx-xxxxxxxxx/resourcegroups/DEMO-ADF-RG/providers/Microsoft.DataFactory/factories/adf-terraform-demo-Postgres-to-MySQL/integrationRuntimes/integrationRuntime3/providers/Microsoft.Authorization/roleAssignments/xxxxxxxx-xxxx-xxxx-xxx-xxxxxxxxx' or the scope is invalid. If access was recently granted, please refresh your credentials."}}

Step 9:

Now go to the ADF where this has to be shared (one added in the sharing list – adf-terraform-demo). Go to Connections → Integration runtimes → +New  →  Azure, Self Hosted

Here you will find Type as  Self-Hosted (Linked). Enter the Resource ID from Step 6 and Create.

After successful creation, you can find the new IR with sub-type Linked

The IR sharing setup is complete. Be seamless with your ADF pipelines now.

Conclusion

Sharing IRs between ADFs will save greatly on the infrastructure costs. Sharing is simple & effective. We will come up with more ADF use cases and share our problem statements, approaches and solutions.

Hope this was informative. Do leave your comments below for any questions.

Read the series here