Intel IT is building a modern, scalable cyber intelligence platform (CIP), using an Apache Kafka data pipeline based on Confluent Platform.

Advanced cyber threats continue to increase in frequency and sophistication, threatening computing environments and impacting businesses’ ability to grow. More than ever, large enterprises must invest in effective information security, using technologies that improve detection and response times. Apache Kafka is the foundation of our CIP architecture. We achieve economies of scale as we acquire data once and consume it many times. Simplified connection of data sources helps reduce our technical debt, while filtering data helps reduce costs to downstream systems. We selected Confluent as our Kafka partner to help with the typical pain points and risks that accompany pure open source software deployments. We immediately started to realize benefits of Kafka by achieving economies of scale. We can now acquire data once and consume it many times. Other benefits like reduction of technical debt by eliminating legacy point-to-point and custom connectors soon followed. And through data filtering, we have reduced the cost of ingest and storage in a variety of downstream systems. Confluent provides enterprise capabilities beyond open source Apache Kafka that make it more powerful, manageable, maintainable, and easy to use. Some of these key capabilities are:

• Confluent Control Center for monitoring our Kafka clusters
• Replicator and Multi-Region Clusters for multiple data center design and fault tolerance
• Auto Data Balancer for cluster rebalancing and easy scale out, when needed
• Enterprise-grade security including Role-based Access Control and audit logs
• 100+ pre-built connectors to common sources and sinks
• Technical Account Manager for feature prioritization and guidance
• Architecture and design partnership via a Confluent Resident
• Architect and “health check” engagements
• Ongoing training available in real time (for example, webinars) and on demand
• Availability of 24x7x365 support
• Partnership for collaborating on new capabilities, such as security features

Our CIP is based on Intel® Xeon® Scalable processors and Intel® solid state drives (SSDs). Intel vice president and Chief Information Security Officer, Brent Conran, explains, “Kafka helps us produce contextually rich data for both IT and our business units. Kafka also enables us to deploy more advanced techniques in-stream, such as machine-learning models that analyze data and produce new insights. This helps us reduce mean time to detect and respond; it also helps decrease the need for human touch. The most transformational capability of Kafka for cybersecurity is in-stream processing. The ability to operate on data as it is produced helps security responders improve detection techniques and response times. It also enables us to develop and deploy more advanced techniques, such as machine-learning models that perform in-stream processing, which can identify threats in near real time. These advances are helping us achieve our mission to “make it safe for Intel to go fast.” Using industry-leading technologies that provide a modern scalable architecture enables us to continue to transform well into the future.