Case Study

Modernizing Massive Log Ingestion

LG U+ operates a hybrid cloud environment (Public AWS & Private Cloud) processing over 2.2 billion log messages daily. Their existing architecture utilized standard Kafka brokers to pipe logs from EKS/OKD clusters to OpenSearch and Sumo Logic.

The "Stateful" Bottleneck

The team aimed to make their infrastructure truly cloud-native—prioritizing stateless operations, fast rolling updates, and elastic scaling. However, traditional Kafka's coupled storage and compute architecture made it heavyweight to manage and difficult to scale dynamically.

They needed a solution that could:

• Run efficiently on AWS ECS (Elastic Container Service) within a strict Private VPC environment
• Eliminate expensive local block storage requirements for long-term retention
• Support stateless broker operations for true cloud-native agility
• Maintain 100% compatibility with existing observability tooling

AutoMQ's cloud-native architecture was the perfect fit for LG U+'s modernization goals:

Perfect Fit for AWS ECS (Stateless Architecture)

AutoMQ's stateless broker design was the key enabler for deploying on AWS ECS. Unlike traditional Kafka, which requires complex state management, AutoMQ allows brokers to be treated as interchangeable compute tasks. This enabled LG U+ to utilize ECS's rolling update capabilities and Circuit Breaker deployment strategies to ensure zero downtime during maintenance.

Cost-Efficiency via S3 Offloading

By adopting AutoMQ, LG U+ replaced local disk reliance with Amazon S3 for persistent storage. This "S3Stream" architecture meant that data durability was handled by S3, allowing the brokers to remain lightweight. This significantly reduced storage costs for long-term log retention (up to 3 days) compared to provisioning high-performance EBS volumes.

100% Drop-in Compatibility

A strict requirement was preserving the existing toolchain. AutoMQ provided 100% Kafka protocol compatibility, allowing LG U+ to keep their existing Fluentd, Sumo Logic, and OpenSearch integrations running without changing a single line of configuration or client code.

Since migrating to AutoMQ, LG U+ has achieved significant improvements in their log infrastructure:

Standardized Cloud-Native Operations

LG U+ successfully established a standardized, Terraform-managed log pipeline on AWS ECS. The migration to AutoMQ eliminated the operational burden of managing storage replication at the broker level. The team can now perform safe, automated rolling deployments and handle infrastructure updates with the agility typical of stateless microservices.

Secure & Scalable Private Deployment

Despite the challenges of a strictly isolated network (Private VPC, internal DNS, and custom security groups), the AutoMQ cluster was successfully validated to handle high-throughput log ingestion. The architecture now supports seamless horizontal scaling to accommodate traffic spikes, with a roadmap to implement metric-based auto-scaling for further efficiency.

Key operational improvements include:

• Brokers treated as stateless compute resources, enabling true cloud-native operations
• Rolling updates with zero downtime using ECS Circuit Breaker patterns
• Terraform-managed infrastructure as code for consistent deployments
• Seamless integration with existing Fluentd, OpenSearch, and Sumo Logic pipelines
• Foundation for metric-based auto-scaling to optimize resource utilization