Originally published on vmblog.com: https://vmblog.com/archive/2022/12/27/alluxio-2023-predictions-what-s-next-for-data-analytics-ai-and-cloud-in-2023.aspx

As we enter 2023, the world of analytics, AI, and cloud is entering an exciting new phase, with a wide range of innovations and developments set to reshape the landscape. Below are some trends that will have the most impact in the coming year.

Trend 1: Cloud cost optimization is becoming increasingly important

In 2023, as global economic uncertainty continues, enterprises with data-intensive workloads in the cloud will need to review their cloud strategies with a greater focus on cost optimization. Cloud spending will be more closely scrutinized based on the ROI and TCO of existing projects or new investments.

One area where cost optimization is particularly important in the coming year is data transfer egress costs, which can make up a significant portion of an organization's cloud bill. We will see more companies optimize their architecture in order to avoid sticker shock from unanticipated egress costs. For example, using Alluxio caching to reduce the data transferred on the network.

In addition, more enterprises are seeking multi-cloud "freedom," which allows them to use any cloud services without being locked in. Application portability will be the foundation of this "freedom." This will enable them to choose the best option for their specific requirements and budget.

Trend 2: Big models are showing transformative potential, driving innovations in specialized infrastructure

Big models, like OpenAI's ChatGPT for dialogue, DALL-E 2's image generation model, and Google's LaMDA conversation agent, have shown transformative potential in 2022. They are expected to unlock more use cases and applications in 2023. The increased adoption of these models will likely drive the development of specialized infrastructure and solutions for AI.

Training big models using billions of parameters requires specialized infrastructure and solutions to handle the computational demands. As a result, we expect to see the continued development of AI infrastructure that can handle the scale and complexity of these models.

Furthermore, as the capabilities of big models continue to improve, researchers and developers will need to find new ways to apply these models in real-world scenarios. New tools and platforms will emerge to make it easier for developers to work with big models and apply them to a wider range of tasks.

Trend 3: Data sharing, data exchange, and data marketplace will be more prevalent

Data sharing includes both inter-organizational data sharing and cross-company sharing. While adoption remains in the early phases, the ecosystem centered around data sharing, including infrastructure, transactional capabilities and services for both data consumers and data providers will keep evolving in 2023.

Internal data sharing within organizations will be driven by cross-domain data value realization, aiming to share data and remove silos. External data-sharing use cases and success stories are proliferating as more organizations pursue opportunities to monetize their data assets. For example, in academia and research, organizations are exploring ways to share research data via data-sharing platforms to accelerate their studies.

This trend will have a significant impact on data infrastructure, as organizations will need to adapt and evolve their systems to support the sharing of data across regions, organizations, clouds, and platforms. There will also be an increased focus on data governance and security as organizations seek to ensure that their data is managed and accessed in a compliant and secure manner.

Trend 4: The convergence between data warehouses and data lakes and the accelerated adoption of open table formats

The convergence of data warehouses and data lakes is a growing trend in the modern data stack. This trend is being driven by the increasing complexity and diversity of data and by the need for organizations to have flexible and scalable systems that can support a wide range of data science and analytics use cases. As a result, data warehouses and data lakes are becoming more integrated.

The rise of open table formats, such as Apache Iceberg, Hudi, and Delta Lake, has played a role in this trend. These formats act as a layer to efficiently store and manage large amounts of structured and unstructured data in a single system, enabling organizations to derive value from their data more quickly and at a lower cost. In 2023, more enterprise data will be stored in open table formats as these solutions are rapidly adopted.

Trend 5: Data locality will be addressed in Kubernetes

The separation of computation and storage in Kubernetes has long been a challenge when it comes to data locality. While Kubernetes has made it exceptionally easy to deploy and scale data-intensive applications elastically, accessing data from cloud-native data sources (like AWS S3 or sometimes remote data warehouses) becomes more challenging. We expect that the data locality challenges will be addressed in 2023.

The ability to make decisions agnostic to data locality is getting more important for Kubernetes schedulers. This ability will be more crucial for Kubernetes interface to help applications and schedulers to be more efficient. We expect more solutions to emerge to bridge the gap between computation and storage and to make it easier for organizations to manage and optimize their data storage and processing in Kubernetes.

Conclusion

Overall, the next year is shaping up to be an exciting time for the world of big data, AI, and cloud with a wide range of developments and innovations set to shape the future of these fields. Many technological paradigms are merging to form an ecosystem around data as we move into 2023. It will be fascinating to see how these technologies continue to evolve and impact the world around us.