Making the Connected Car ‘Real-time Data Processing’ Dream a Reality

By October 16, 2019 October 23rd, 2019 Analytics, Automation, AWS, Blogs

Written by Jeremiah Peter, Solution specialist-Advanced Services Group, Contributor: Ravi Bharati, Tech Lead and Ajay Muralidhar,  Sr. Manager-Project Management at Powerupcloud Technologies

Connected car landscape

Imagine driving your car on a busy dirt road in the monsoon, dodging unscrupulous bikers, jaywalking pedestrians and menacing potholes. Suddenly, a fellow driver makes wild gestures to inform you that the rear door is unlocked, averting an imminent disaster.

In a connected car system, these events are tracked in near real-time and pushed to the driver’s cell phone within seconds. Although the business relevance of real-time car notifications is apparent, the conception of the underlying technology and infrastructure hardly is. The blog attempts to demystify the inner workings of handling data at scale for an Indian automobile behemoth and equips you with a baseline understanding of storing and processing vast troves of data for IoT enabled vehicles.

The paradigm of shared, electric and connected mobility, which seemed a distant reality a few years ago, is made possible through IoT sensors. Laced with tiny data transmitting devices, vehicles can send valuable information such as Battery Percentage, Distance to Empty (DTE), AC On/Off, Door Locked/Unlocked, etc. to the OEM. The service providers use this information to send near real-time alerts to consumers, weaving an intelligent and connected car experience. Timely analysis and availability of data, thus, becomes the most critical success component in the connected car ecosystem.

Before reaching the OEM’s notification system, data is churned through various phases such as data collection, data transformation, data labeling, and data aggregation. With the goal of making data consumable, manufacturers often struggle to set up a robust data pipeline that can process, orchestrate and analyze information at scale.

The data conundrum

According to Industry Consortium 5GAA, connected vehicles ecosystem can generate up to 100 terabytes of data each day. The interplay of certain key factors in the data transmission process will help you foster a deeper understanding of the mechanics behind IoT-enabled cars. As IoT sensors send data to a TCP/IP server, parsers embedded within the servers push all the time series data to a database. The parsing activity converts machine data (hexadecimal) into a human-readable format (Json) and subsequently triggers a call to a notification service. The service enables OEM’s to send key notifications over the app or through SMS to the end-consumer.

Given the scale and frequency of data exchange, the OEM’s earlier set up was constrained by the slow TCP/IP data transfer rate (Sensor data size: TCP/IP- 360 bytes; MQTT- 440 bytes). The slow transfer rate has far-reaching implications over the user experience, delaying notifications by 6-7 minutes. As part of a solution-driven approach, Powerup experts replaced the existing TCP/IP servers with MQTT servers to enhance the data transfer rate. The change affected a significant drop in notification send-time, which is presently calibrated at around 32-40 seconds.

Furthermore, the OEM’s infrastructure presented another unique challenge in that only 8 out of 21 services were containerized. The rest of the services ran on plain Azure VM’s. To optimize costs, automate scalability and reduce operational overhead, all services are deployed on Docker Containers. Containers provide a comprehensive runtime environment that includes dependencies, libraries, framework and configuration files for applications to run. However, containers require extensive orchestration activities to aid scalability and optimal resource management. AWS Fargate is leveraged to rid the OEM’s infrastructure management team of routine container maintenance chores such as provisioning, patching, cluster and capacity management

Moreover, MQTT and TCP IP brokers were also containerized and deployed on Fargate to ensure that all IoT sensor data is sent to the AWS environment. Once inside the AWS environment, sensor data is pushed to Kinesis Stream and Lambda to identify critical data and to call the AWS notification service-SNS. However, the AWS solution could not be readily implemented since the first generation of electric vehicles operated on 2G sim cards, which did not allow change of IP whitelisting configuration. To overcome the IP whitelisting impediment, we set up an MQTT bridge and configured TCP port forwarding to proxy the request from Azure to AWS. Once the first generation vehicles are called back, the new firmware will be updated over-the-air, enabling whitelisting of new AWS IP addresses. The back-handed approach will help the OEM to fully cut-over to the AWS environment without downtime or loss of sensor data.

On the Database front, the OEM’s new infrastructure hinges on the dynamic capabilities of Cassandra DB and PostgreSQL. Cassandra is used for storing Time Series data from IoT sensors. PostgreSQL database contains customer profile/vehicle data and is mostly used by the Payment Microservice. Transactional data is stored in PostgreSQL, which is frequently called upon by various services. While PostgreSQL holds a modest volume of 150 MB Total, the database size of Cassandra is close to 120 GB.

Reaping the benefits

While consumers will deeply benefit from the IoT led service notifications, fleet management operators can also adopt innovative measures to reduce operational inefficiencies and enhance cost savings. Most fleet management services today spend a significant proportion on administrative activities such as maintaining oversight on route optimization, tracking driver and vehicle safety, monitoring fuel utilization, etc. A modern fleet management system empowers operators to automate most of these tasks.

Additionally, preventive maintenance can help operators augment vehicle lifecycle by enabling fleet providers to pro-actively service vehicles based on vehicular telemetry data such as battery consumption, coolant temperature, tire pressure, engine performance and idling status (vehicle kept idle). For instance, if a truck were to break-down due to engine failure, the fleet operator could raise a ticket and notify the nearest service station before the event occurred, cutting down idle time.

Conclusion

With 7000 cars in its current fleet, the OEM’s infrastructure is well-poised to meet a surge of more than 50,000 cars in the near future. Although the connected car and autonomous driving segment still goes through its nascent stages of adoption, it will continue to heavily draw upon the OEM’s data ingestion capabilities to deliver a seamless experience, especially when the connected car domain transcends from a single-vehicle application to a more inclusive car-to-car communication mode. Buzzwords such as two-way data/telematic exchanges, proximity-based communications and real-time feedback are likely to become part of common parlance in mobility and fleet management solutions.

As the concept of the Intelligent Transport System gathers steam, technology partners will need to look at innovative avenues to handle high volume/velocity of data and build solutions that are future-ready. To know more about how you can transform your organization’s data ingestion capability, you can consult our solution experts here.

Leave a Reply