The Faster You Can Access Analytics, the More Market Advantage You Have
Today, quantitative traders and data analytics teams need to see all of today’s data in nanoseconds. They also require access to as much historical data as possible for accurate trend analysis and insights. That can mean data from last week, last month, or ten years ago.
Traditionally, the most regularly-accessed data, also known as hot and warm data, would be accessed from memory or fast storage, while historical data and analysis meant accessing colder storage via slow disk drives or—even worse—going out to the even slower network. This is not acceptable for today’s trading requirements.
On top of these increasing historical queries, the onslaught of data continues. Data rates have doubled since last year, and now data from IDC reports that by 2025, the global datasphere will be 175 ZB, with nearly 30 percent of it real time.1 This completely changes how to approach IT infrastructure.
Five years of trading data can add up to petabytes that need to be rapidly accessed and then processed affordably, reliably and with minimal latency. Latency-to-result is the key metric.
In the past, latency problems have been solved by adding more servers or investing in expensive high-performance memory and network storage. With today’s data volumes, neither are realistic. Yet knowledge is power in the world of financial services, and the challenge to harness data and analytics and turn them into actionable insights is getting increasingly difficult. A new infrastructure is required, one that keeps as much data close to compute as possible, for lightning-fast access.
Intel® Optane™ technology solves this. The first new memory and storage technology in 25 years comes in two formats: Intel® Optane™ persistent memory (Intel® Optane™ PMem) and Intel® Optane™ Solid State Drives (Intel® Optane™ SSDs.) These technologies essentially add two layers of memory and storage to close the capacity, cost, and performance gaps between DRAM (nanoseconds) and NAND SSDs (microseconds). If we think of this in the view of a memory storage pyramid—Intel Optane PMem sits right below DRAM in memory, providing memory-like performance at a lower cost but with NAND-like persistence. Intel Optane SSDs sit above NAND SSDs and hard drives, delivering faster data reads and writes, erasing bottlenecks even under heavy loads.
For financial services, both Intel Optane PMem and Intel Optane SSDs effectively pair with three new innovative technologies: Oracle Exadata*, Aerospike Database*, and KX Streaming Analytics*. Here’s a look at the benefits of each.
Oracle* Exadata
Nine of the world's ten largest financial services firms use Oracle* Exadata to run databases for core banking applications and customer analytics.2 Exadata's industry-leading performance, scale, availability, and security allows them to accelerate their business, reduce risks, and cut costs. Several firms for financial services have used Oracle Exadata to run diverse workloads faster and increase customer insights for over ten years. The Intel® technologies in Exadata solutions deliver capabilities that provide a host of benefits.
This is all possible because Oracle Exadata is an Intel® Optane™ PMem-enhanced scalable architecture. It includes smart system software that eliminates performance bottlenecks in crucial customer transaction processing and analytics workloads.
This results in extremely low latency, which is critical for transaction processing. Data is not simply in-memory, it's a tier of data that enables massive throughput and parallelism in operations. This unique architecture enables impressive scalability and in-memory speeds without the limitations of in-memory design.
Huge advancements have been made in latency-sensitive database operations. This includes operations such as reading blocks from storage, writing to the transaction log, and transferring blocks across the cluster. Whereas Exadata storage servers once used flash as their fastest tier of storage, they now can use persistent memory, which provides almost an order-of-magnitude reduction in latency. As previously mentioned, there are multiple operations inside the Oracle* Database that benefit from the performance gains it brings.
Through using Intel Optane PMem combined with Remote Direct Memory Access (RDMA), caching data into persistent memory allows the Oracle Database to access those contents much faster compared to any storage technology that used a request/response processing model. The Oracle Database does not request a block from storage. Rather, the database simply reads the data it needs directly from Intel Optane PMem. Exadata Storage Server software is responsible for identifying the hot blocks of data and placing them into persistent memory.
“Exadata is the only platform in the world to use Intel Optane PMem in shared storage. The combination of Remote Direct Memory Access over Converged Ethernet (RoCE), persistent memory, and Exadata’s unique storage server software makes the Exadata X9M generation the fastest, most reliable, scalable, and secure platform for running Oracle Database,” said Subban Raghunathan, Oracle vice president of Exadata product management.
Intel® Optane™ Technology Benefits for Oracle* Exadata
- Greater scale and availability accelerate back-office and front-office applications.
- Built-in cloud automation increases availability and simplifies IT operations.
- Online consumption scaling simultaneously optimizes performance and cost.
- Oracle* Exadata, Oracle Database, and Oracle Cloud Infrastructure (OCI) offer several built-in features that increase security and help meet regulatory requirements.
- Consistent data sovereignty, security, and latency requirements.
- Industry-leading auto-provisioning, autoscaling, and auto-securing capabilities.
Aerospike* Database
Aerospike is the global leader in next-generation, real-time NoSQL data solutions for any scale. Aerospike captures streaming data at the digital edge, consolidates it with your core data, feeds the data into AI and ML systems for optimized training and analysis, then pushes instructions and transaction outcomes back to the edge in real time. For finance, it provides hyperscale performance to process massive datasets quickly and at extremely low latency to ensure the integrity of financial transactions, data, and analysis. Financial analysts now have increased accuracy, with the ability to access market, customer, and transactional data more frequently from multiple feeds simultaneously. And Aerospike can handle millions of transactions per second while scaling to meet petabyte-range data volumes.3
“We excel at large databases, with our microsecond performance—it’s essentially zero delay,” explains Jeff Ballard, Aerospike director of partner marketing. Aerospike’s Hybrid-Memory Architecture* enables both database indexes and data to be stored in Intel® Optane™ PMem. Aerospike’s tiered internal architecture has been optimized to use Intel Optane PMem natively. This offers capabilities far beyond the implementation of an in-memory database and simply inserting persistent memory. Intel Optane PMem supercharges the Aerospike Database, delivering DRAM-like performance with low predictable latency, and enables “RAM index” at a fraction of the cost, as well as up to 10x scale improvement for data in NAND-based storage.4
PayPal, the world’s largest online payments system, chose Aerospike with Intel Optane PMem to deliver a 5x throughput increase that delivered a 30x reduction in fraud transactions missed.3 These transformative performance benefits enabled game-changing improvements in fraud management and prevention. In addition, this solution was achieved with a 15x reduction in server footprint and a $9M hardware cost savings projected.3 Overall, this combination of hardware and software empowered PayPal to achieve better business results, improve customer satisfaction, and reduce cost and complexity.
The speed, reliability, and consistency of Aerospike with Intel Optane PMem enables multiple use cases.
The bottom line? Forrester notes that “with Aerospike’s NoSQL Data Platform, organizations saw profit growth improvements, improved conversion rates, TCO reductions, server footprint reduction, and full-time equivalent (FTE) support reduction.”5
Intel® Optane™ Technology Benefits for Aerospike* Database
- Fraud prevention and passive identity validation are powered by the ability to process millions of transactions, billions of data points, and petabytes of historical information in milliseconds.
- This in turn enables companies to incorporate the most advanced AI and neural net/deep learning algorithms into their solutions.
- Hyper-personalization offers a connected 360-degree profile of customers based on AI/ML data sets, delivering immersive insights enabling customized cross-selling and upselling opportunities.
- Digital payments—secured with online user authentication—use real-time data and AI-powered analytics for friction-free payment processing and powerful fraud prevention.
- Brokerages can enhance office operations in many areas: optimizing and automating portfolios with AI/ML; improving risk, compliance, liquidity, and margin lending capabilities; and automating clearing/settlement and reporting functions.
- Bankers can analyze massive volumes of datasets across retail, corporate, and investment banking to hyper-personalize digital content and ensure the integrity of every transaction.
KX* Streaming Analytics
KX has been at the forefront of helping financial services firms, process, analyze, and deliver actionable insights on massive amounts of both real-time and historical data. KX* Streaming Analytics reduces costs and improves productivity through anomaly detection, analytics, predictions, corrective action, and automation.
“Financial Services companies can be looking at millions of transactions per second,” notes Ferenc Bodon, KX solutions architect. “The quants require automated trade analytics, so they develop a strategy and need it codified for training in the system. Then they need to backtest their assumptions on historical data, and finally encapsulate the algorithm and transfer it to the trading engine. In each of these steps they do not want to wait.”6 This desire for fast access and action drives a need for high performance hardware to maximize software value—this is where Intel® Optane™ technology comes in.
Daily trading data can easily exceed a terabyte—or several terabytes if you include derivatives. To handle these volumes without amassing huge amounts of prohibitively expensive DRAM, Intel Optane technology delivered a game changing solution. For example, a top-tier financial institution achieved nearly 10x speed improvement when implementing Intel Optane Persistent Memory with KX kdb+ v4.0 compared to high-performance NAND SSDs.7 In addition, use of Intel Optane Persistent Memory as a block-storage device with KX Streaming Analytics delivers 4x to 12x improved analytics performance over high performance NVMe storage—similar performance to DRAM for query workloads.7
For key data-processing workloads, DRAM requirements were significantly reduced. This allows organizations to support more demanding analytic workloads on more data with less infrastructure. Now analysts and quants have access to weeks or months of data instantly. This concept is relevant everywhere across the financial services landscape. Big banks, trading houses, exchanges, digital payment providers—they all benefit.