Building a data foundation for AI using Snowflake and AWS

Snowflake

By Daniel Wirjo, Solutions Architect – AWS
By Benny Chun, Solutions Architect – AWS
By Bosco Albuquerque, Sr. Partner Solutions Architect – AWS
By Hans Siebrand, Cloud Data Architect – Snowflake
By Matt Marzillo, Sr. Partner Engineer – Snowflake

With recent advancements, building a data platform to provide a data foundation for generative AI is critical in unlocking efficiencies and innovative use cases. According to McKinsey, leveraging generative AI has the potential to generate between $2.6 trillion to $4.4 trillion in value across various industries annually, delivering up to 40 percent uplift in productivity and economic impact.

In this post, you will learn about key considerations and best-practices for building a data foundation for AI with AWS and Snowflake. To demonstrate, this post delves into an example scenario of building a personalized recommendation engine. It covers key capabilities and seamless integrations for Snowflake and AWS. This includes transforming, governing and collaborating with data on Snowflake. This technology leverages state-of-the-art AWS AI/ML services in recommendations with Amazon Personalize and foundation models on Amazon Bedrock. By combining the best of both worlds, businesses can benefit from the scalability, flexibility and performance of AWS and Snowflake for their AI journey.

Key ingredients for building a data foundation for AI

To build a data foundation for AI, you need a cloud-based data platform that simplifies operations and addresses common challenges across each stage of the data and AI lifecycle.

Data movement: Moving data across platforms is a key challenge, primarily due to challenges across data integrity, security, and transfer efficiency. AWS and Snowflake have deep integration that allows for data to move quickly and securely. In the example solution below, you simply define an external stage to point to your cloud storage on Amazon Simple Storage Service (S3). You then have the choice of loading the data into Snowflake in bulk or continuously using Snowpipe.

Data transformation: Traditionally, data transformation is done with an Extract Transform Load (ETL) pattern, whereby specialized data science and engineering knowledge is required to implement. With Snowflake, you can also perform transformations using SQL (known as ELT, or Extract Load Transform), a popular and widely known skill set that can be conducted without deep data expertise, including by analysts without in-depth data experience. This enables your data to be engaged and used more widely by your organization for their use cases. This is especially critical with AI and machine learning models requiring data to be prepared specifically for the use case. Your ability to transform data into its appropriate form is a critical component of your AI foundation.

Data governance: With data being produced and used across the organization, classifying, governing and securing access to data can be challenging. With Snowflake, achieving your data security and compliance goals is simple. With capabilities such as column-level security, dynamic data masking and tag-based masking policies, you can centrally govern your data at scale with granular control on who can access what information. This means that organizations with sensitive data or in regulated industries (such as financial services or healthcare) can adhere to their stringent operational and compliance policies in an efficient and cost-effective manner.

Extensibility and access to AI services: More organizations are realizing that the power of AI comes from leveraging multiple foundation models and using best-in-class AI services for specific use cases. For example, obtaining recommendations based on your users’ history can leverage recommender systems and AI services such as Amazon Personalize. In addition, generating tailored marketing copy requires a foundation model with language capabilities on Amazon Bedrock. Along the way, you can also augment your data with 3rd party APIs. Snowflake External Access enables you to centralize, secure and govern access to AWS AI services, allowing you to benefit from the breadth and depth of AI services that AWS offers.

Example solution for a foundational data platform for AI

Let’s take a look at an example solution on how to build a recommendation engine using generative AI. At a high-level, the example solution demonstrates how AWS and Snowflake excels at the key capabilities needed for a successful data foundation for AI.

Solution overview:

• Prepare data for AI by transforming raw data from Amazon S3 using SQL-based transformations on Snowflake
• Govern and secure your data by dynamically masking sensitive data (such as PII) using Snowflake’s data governance capabilities
• Leverage a state-of-the-art recommendation engine using Amazon Personalize and Snowflake External Functions
• Leverage a generative AI foundation model to create personalised copy from generated recommendations using Amazon Bedrock and Snowflake External Access without the complexities traditionally required with building proxies
• Securely share the results of your AI recommendations using Snowflake’s data sharing capabilities

The example solution provides guidance on how to build a secure and scalable data foundation for AI on AWS and Snowflake without the operational complexities associated with managing infrastructure.

To learn more, contact Snowflake.

Conclusion

In this post, you learned about how you can leverage the latest AI innovation with Snowflake as the foundational data platform. By using Snowflake and AWS, organizations can accelerate their AI transformation initiatives with best-in-class capabilities across the data and AI lifecycle.

You can also learn more about Snowflake in AWS Marketplace.

Snowflake – AWS Partner Spotlight

Snowflake is an AWS Competency Partner that is bringing generative AI into data, empowering teams to maximize the value of the data by identifying the right data points, assets, and insights.

Contact Snowflake | Partner Overview | AWS Marketplace | Case Studies