Blackout Power Trading, a private capital commodity trading fund specializing in North American power markets, leverages Alluxio's low-latency distributed caching platform to achieve multi-join double-digit millisecond latency offline feature store performance while maintaining the cost and durability benefits of Amazon S3 for persistent data storage.

CHALLENGE

S3 latency bottlenecks were limiting the scalability and performance of ML platform

Blackout Power Trading executes day-ahead electricity trades across thousands of U.S. power grid locations. Each morning, within a critical 15-minute market window, they run thousands of ML models using the latest weather and renewable generation forecasts. Their ML pipeline processes 500+ GB of feature data stored as Parquet files on Amazon S3.

As trading operations scaled, feature retrieval from S3-hosted Parquet files introduced unacceptable I/O latency and reduced compute resource utilization, capping their ability to scale. A typical feature retrieval query, joining 20+ tables and returning 80+ columns, took several seconds to complete, far too long for their aggressive performance requirements.

Attempts to reduce latency proved unsuccessful. For example, the team attempted a pre-caching solution using Ray's object store. While it provided some improvement, the solution quickly hit S3 bandwidth limitations and memory capacity ceilings on cost-effective spot instances. These workarounds increased operational complexity without providing the reliability needed for mission-critical trading operations.

Key Challenges and Needs:

• High availability: The platform must tolerate node failures during the inference window without service disruption
• Low-latency access: Feature store queries and model artifact loads must meet predictable latency targets
• Offline feature store only: The company requires an offline feature store, not an online one, to reduce system complexity and memory costs
• Elastic scaling: The cluster should scale up/down to match workload, ensuring they only pay for resources when in use
• 10x+ growth capacity: Architecture must support scaling beyond 100,000 models without redesign
• Strong consistency: Concurrent read/writes against plain Parquet files stored on S3 should remain consistent while sustaining high throughput
• Platform integration: The solution must integrate cleanly across their existing tools and infrastructure

SOLUTION

A low-latency distributed feature store on top of Alluxio that powers ML training and inference

Blackout Power Trading implemented Alluxio as a transparent caching layer between their compute infrastructure and Amazon S3 storage. This created a dynamically scalable, distributed feature store that eliminated S3 as a performance bottleneck while maintaining the cost and durability benefits of object storage.

Their architecture leverages Alluxio's distributed caching on NVMe SSDs, with a Rust-based query engine integrating via Alluxio's POSIX (FUSE) interface. This enables their Python dataframe-first pipelines to read S3 data as if it were on a local file system, providing predictable, low-latency access for both model artifacts and feature inputs.

Alluxio operates transparently, caching frequently accessed data on local NVMe drives within its compute cluster while utilizing write-through persistence to reliably store models and outputs back to S3.

‍

Why Alluxio:

• Double-digit millisecond latency data reads from Parquet files with Alluxio’s low-latency cache (multi-join training and inference data)
• API translation layer unifying S3 access through local file system semantics
• Lightweight deployment with local disk caching
• Seamless integration with existing platform and tooling
• On-demand scalability supports future growth

RESULTS

With Alluxio deployed, Blackout Power Trading achieved significant performance improvements across its entire ML pipeline, enabling the company to scale from 5,000 to over 100,000 models within the same 15-minute trading window.

Impact:

• 22 to 37× reduction in large-join query latency for training.
• 37 to 83× reduction in large-join query latency for inference.
• 10× increase in model capacity, scaling from 5,000 to 100,000+ models in the same 15-minute window.

Detailed Performance Benchmark Results:

‍

"Alluxio has been a key enabler in delivering the low-latency feature store required for our ML trading models. By providing double-digit millisecond latency for multi-join queries in our offline feature store, we can now scale beyond 100,000 models within our 15-minute trading window. Its lightweight deployment aligns perfectly with our lean infrastructure strategy, and seamless integration with our existing platform and tooling made adoption straightforward. Most importantly, the time saved through Alluxio’s caching layer translates directly into more time for risk analysis, and ultimately, better trading decisions."

— Greg Lindstrom, Vice President - ML Trading, Blackout Power Trading

ABOUT BLACKOUT POWER TRADING

Blackout Power Trading Inc. is a private capital commodity trading fund based in Calgary, Alberta, Canada, specializing in North American power markets. The traders at Blackout Power Trading participate in daily electricity auctions throughout the year, operating independently using their own technology stacks. Focused on day-ahead virtual power and congestion markets, the traders leverage advanced machine learning strategies to inform trading decisions and create more efficient electricity markets through their speculative trading activities.

Learn more about Blackout Power Trading’s architecture in this article written by Greg Lindstrom, Vice President of ML Trading at Blackout Power Trading.