I. Introduction

Previous works

There have been numerous articles and online webinars dealing with the benefits of using Alluxio as an intermediate storage layer between the S3 data storage and the data processing system used for ingestion or retrieval of data (i.e. Spark, Presto), as depicted in the picture below:

To name a few use cases:

• Alluxio-Presto use cases:
  • Ryte
  • ING
• Alluxio-Spark use cases:
  • Bazaarvoice

The main conclusion from these use cases is that using Alluxio as a data orchestration layer has the following benefits:

• lower latency in data processing pipelines:
  • Co-locating data and computation reduces network traffic
• horizontal scalability for usage concurrency:
  • S3 API has limitations on the request rate for a given prefix

All these benefits are critical when deciding a production-grade data processing architecture, however one important benefit has so far not been sufficiently highlighted when choosing such architecture – cost reduction on the usage of the S3 API. This will be the focus of this article.

datasapiens

datasapiens is an international startup, based in Prague, that focuses primarily on helping companies to transform their business through data. We offer 3 key products:

• Business insights
  • our business intelligence platform with an embedded insights framework to help clients to become data-driven companies.
• Brands insights
  • commercialize clients' data assets by driving cooperation with their suppliers.
• Personalized offers
  • increase customer engagement, sales and margin through a simple plan-do-review process that continuously measures and improves the loyalty lifecycle of clients' customers.

Our infrastructure journey

The first generation of our infrastructure used AWS Redshift as the main data warehouse. However, when we started to acquire clients with larger data volumes (i.e. billions of records per year) we started to struggle with the following:

• storage-compute coupled architecture
• large infrastructure costs
• long query compilation times
• vendor-locking

It was at this point that we decided to move to the Hadoop ecosystem.

The first version of our second generation was the common software stack for cloud-based data lake architectures:

• S3 for data persistence
• Spark for ETL
• Hive for data warehousing
• Presto as a distributed query engine

With this storage-compute separated architecture, we achieved faster performance and reduced compilation times. However, the infrastructure costs were above expectations due to S3 API cost increases, making up a large proportion of the overall cost (described in detail below).

The second version of our second generation includes Alluxio as a data orchestration layer. We co-locate our data with our compute for in-memory data access. This architecture achieves even faster performance more consistently. In addition, we saw a drop in infrastructure costs due to S3 API costs dropping to a negligible level. 

II. Our observations

A drastic increase in S3 API costs

During the fine-tuning of our first Presto cluster, we conducted several performance tests of the cluster under various levels of user concurrency.

The setup of the performance tests was the following:

• dataset setup:
  • internal client data (1 billion rows of retail transactional data)
  • persisted in Parquet
• workload setup:
  • 23 different queries used in client’s reports
  • concurrency levels from 1 to 150
• cluster hardware setup:
  • worker nodes:
    • count: 10
    • type: c5.4xlarge
  • master nodes:
    • count: 1
    • type: c5.4xlarge
• cluster software deployment setup:
  • PrestoDB 0.227
  • Apache Hive v2.3.5

For the design of our DWH, we wanted to avoid using the traditional star/snowflake schemas. Instead, we could pre-join the table entities in our ETL into a few OLAP cubes to avoid expensive table joins in Presto.

Towards the end of the testing phase, we detected a dramatic increase in costs for the S3 API service. The following image depicts the incurred S3 API costs for each day of the performance tests:

The following table summarizes the total number of executed queries with the S3 API costs and S3 API request counts per each day of performance tests:

DateTotal query countTotal S3 costs ($)Total S3 request countAvg. costs per query ($)Avg. S3 requests per query2019-08-2815,086172.73197,549,8600.0114513,0952019-08-2929,997124.87138,210,9690.004164,6072019-08-306,39426.2722,638,5800.004113,5412019-08-3136,81580.4766,406,7980.002191,8042019-09-0113,93962.5456,195,9760.004494,0322019-09-037524.723,331,2280.006284,4302019-09-066,80628.5918,005,2070.004202,6452019-09-0975,687231.74175,443,2690.003062,318Total185,476731.93677,781,8870.003953,654

The next table shows the percentage of the S3 API costs out of the total costs:

DateTotal S3 costs ($)Total EC2 costs ($)S3 API costs - % of total costs2019-08-28172.7345.2579.24%2019-08-29124.87107.2153.80%2019-08-3026.2727.1849.15%2019-08-3180.4729.4273.23%2019-09-0162.5423.6972.53%2019-09-034.7211.4729.15%2019-09-0628.5973.0528.13%2019-09-09231.74102.9769.24%Total731.94420.2463.53%

During this testing period, we tested several configuration variations of the Presto cluster in order to achieve a satisfying query speed at a specific concurrency level. In the first table above, we can observe the variance in the S3 API costs/requests per day. We attribute this variance to the varying Presto cluster deployments for each day in the testing period.

After reviewing the incurred S3 API costs under the given workloads, we came to the conclusion that running a Presto cluster that read directly from S3 was not feasible, as it would impact our profit margin.

Alluxio as a remedy

We considered deploying Hadoop as an intermediate layer between Presto and S3 and synchronizing through simple jobs.  

We also looked at Alluxio, which provided similar functionality without the need for manual synchronization. Furthermore, Alluxio has additional benefits such as connections to various cloud providers & storage devices and having tiered storage functionality which we thought would improve query speed.

After deploying a co-located Alluxio-Presto cluster, the performance of queries improved (especially for those with a higher number of execution stages), and the S3 API costs dropped to below $1/day. We have since implemented co-located Alluxio-Presto clusters in production. The infrastructure is successfully serving hundreds of users per client.

We also learned that although manual synchronization is not needed. To prevent any performance degradation of the Presto clusters during startup and after ETL jobs it’s best to “hydrate” the cluster using “warm-up” jobs. 

III. A reproducible example using TPC-DS

To get a better quantitative understanding of the S3 API costs and the cost reduction when using Alluxio, we conducted a simple experiment by querying data from S3 in two separate clusters:

• a co-located Alluxio-Presto cluster using PrestoSQL deployed in a Docker Swarm cluster
• an EMR cluster with PrestoDB without Alluxio

In both clusters, we measured the API request count to Alluxio per each query.

The following are the setup schemas for each cluster:

Dataset setup:

• TPC-DS dataset with scale factor 100
• stored in a S3 bucket

Alluxio-Presto cluster setup:

• hardware setup:

• worker nodes:
  • count: 6
  • type: c5.4xlarge
• master nodes:
  • count: 1
  • type: m5.2xlarge
• software deployment setup:
  • versions of used systems:
    • Alluxio 2.2.0
    • PrestoSQL 337

EMR-Presto cluster setup:

• hardware setup:

• worker nodes:
  • count: 6
  • type: c5.4xlarge
• master nodes:
  • count: 1
  • type: m5.2xlarge
• software deployment setup:
  • PrestoDB 0.227

Query execution setup:

• set of queries:
  • TPC-DS suite excluding Query no. 72 (due to longer query times)
• query executions:
  • number of repeats: 10
  • concurrency level: 1 (no parallel executions)

Measurements setup:

• Alluxio:
  • logical Operations: ‘File Infos Got’
  • RPC Invocations: ‘GetFileInfo Operations’
• S3:
  • total request counts per request type
  • total request costs per request type

The deployment configuration for each of the clusters, as well as the complete test results, can be found in our GitHub repository.

Here are the results from the conducted tests on the Alluxio-Presto cluster:

•  10 most API request-expensive queries:

Query nameFile Infos Got - avgGetFileInfo Operations - avgq14_1159,200.1127,576.9q09137,031.0109,669.0q14_2110,933.888,732.6q75101,468.481,166.3q6475,148.360,099.4q8873,224.058,584.0q23_161,313.649,054.3q23_260,566.248,457.6q9556,518.045,212.0q2854,810.043,866.0

• cumulative request counts:

Operation typeCumulative countFile Infos Got24,089,740GetFileInfo Operations19,287,627

• S3 API costs for caching the dataset into Alluxio:

Request typeCumulative countCumulative cost ($)ListBucket28,3240.14GetObject24,0330.01HeadObject44,5810.02Total96,9380.17

Here are the results from the conducted tests on the EMR-Presto cluster:

Request typeCumulative countCumulative cost ($)ListBucket5,771,21928.86GetObject29,254,28011.70HeadObject133,8880.05Total35,159,38740.61

As we can observe, the cumulative count for the `GetObject` request type and the `File Infos Got` request type are more-less compatible. The different values for the storage API calls in both cases can be attributed to:

• different nature of the Alluxio and S3 file system API
• different versions of used Presto clusters (PrestoSQL 337 vs PrestoDB 0.227)

In the case of the S3 API, we also have an additional cost for the `ListBucket` request type which is more costly then the `GetObject` type.

Now, we will try to estimate the per-query S3 API costs for the EMR-Presto cluster. We are limited by the following facts:

• we cannot tag individual requests to S3
• we do not know the exact number of each API request type incurred by individual query

To estimate the per-query S3 API costs, we will take a naïve approach and redistribute the costs to each query proportionally to the number of requests sent to Alluxio.

This gives us the following table with the top 10 request-expensive queries: 

Query nameS3 API cost ($)q14_10.2684q090.2310q14_20.1870q750.1711q640.1267q880.1234q23_10.1034q23_20.1021q950.0953q280.0924

As can be seen, we can have quite significant costs for a single executed query. Let's do a simple comparison of the cluster infrastructure and S3 API costs for both cases with and without Alluxio:

ClusterInfrastructure costs ($)S3 API costs ($) Alluxio+Presto cluster29.020.17EMR+Presto cluster42.5540.61

It is apparent that the S3 API costs for the cluster with Alluxio form a negligible (0.58%) part of the total costs, whereas for the cluster without Alluxio, they form almost half (48.83%) of the total costs in our simple test case.

IV. General implications for cloud data lake architectures

Looking at the pricing models for the storage services of three largest cloud providers (AWS S3, Azure Blob Store, and Google Cloud Storage), we see that they are very similar to each other, hence one can expect similar storage API costs when running analytical workloads with or without Alluxio. However, this claim still needs to be verified.

Our test case was a simple setup of a small cluster to which we were sending queries in a sequential manner. However, real-life production environments cope with workloads with far higher compute and storage demands. Frequently, we have:

• a large number of processes and users frequently and concurrently running queries
• workloads running real-time data streaming, ingesting and retrieving large amounts of data

For these types of cloud-based production environments, using no intermediate data storage layer between the cloud storage service and the data processing system can lead to tremendous costs. 

Therefore we would be inclined to recommend that when running analytical workloads in a cloud-based environment, to have a data orchestration layer such as Alluxio.