Case Study

The Gap in Cloud-Native Elasticity

Tencent Cloud Elastic MapReduce (EMR) provides a highly elastic environment for big data engines like Spark, Flink, and Trino. However, the traditional Apache Kafka architecture—relying heavily on local disks and stateful brokers—became a bottleneck. Users struggled to scale their messaging layer at the same speed as their compute layer. The complexity of rebalancing partitions and managing physical storage prevented true cloud-native agility for the streaming infrastructure.

Complexity in Stream-to-Lake Pipelines

For EMR users, moving real-time data from Kafka into data lakes (like Iceberg or Hudi) for analysis typically required maintaining complex ETL pipelines or heavy connector clusters. This added significant latency and operational overhead, hindering the ability to perform rapid real-time analytics on incoming data streams.

Diskless Architecture for EMR

AutoMQ's architecture perfectly complements the EMR ecosystem by decoupling storage from compute. By offloading all data to Tencent Cloud Object Storage (COS), AutoMQ Brokers become stateless. This aligns with EMR's design philosophy, allowing the streaming layer to scale in and out rapidly without the need for time-consuming data replication or manual partition rebalancing.

Zero-ETL Stream Analytics with Table Topics

AutoMQ bridges the gap between streaming and data lakes. Its unique capability to expose Kafka topics directly as Iceberg tables (Table Topics) allows EMR engines like Spark and Trino to query real-time data streams immediately. This seamless integration eliminates the need for heavy ETL jobs, enabling faster insights and simplifying the data architecture for EMR customers.

Native Security and Operational Integration

AutoMQ is deeply integrated with Tencent Cloud's security framework. It utilizes EMR role-based services to dynamically acquire temporary security keys (SecretId/SecretKey), ensuring secure, imperceptible access to object storage. Additionally, it leverages Tencent Cloud's VPC isolation and self-developed TencentOS hardening, providing enterprise-grade security out of the box.

Native Integration as a First-Party Product

AutoMQ has been successfully integrated as a core component of the Tencent Cloud EMR product suite. It is now available on both the Tencent Cloud International and China stations. Users can select AutoMQ directly from the EMR purchasing page, enabling a unified billing and management experience alongside other big data components.

"Out-of-the-Box" Experience & One-Click Scaling

The deployment complexity has been drastically reduced. Users can now provision a fully configured, storage-compute separated AutoMQ cluster in under two minutes via the EMR console. Operational tasks such as cluster scaling, configuration updates, and service monitoring are now one-click actions, allowing data teams to focus on analytics rather than infrastructure maintenance.