Ingestion of NetSuite and Shopify with Airflow and Snowflake

Automated Data Ingestion from NetSuite and Shopify into Snowflake Using Airflow

PROBLEM

STATEMENT

The client needed some help with ingesting the data from NetSuite and Shopify into Snowflake using Airflow. They were trying to handle it internally but finding it hard to manage given they were not from the technical domain. They wanted me to create an automated system for this transition of data and teach some basics to their internal resources.

SOLUTION

I started by creating connections to Shopify, NetSuite, and Snowflake from Airflow. Once the connections were established, I implemented several Python functions to extract the data from NetSuite and Shopify (through their APIs) and created an Airflow DAG utilizing those functions. After retrieving the data, the DAG copied it into Snowflake. The complete workflow was scheduled at a frequency set by the client.

Input

Access to NetSuite, Shopify, and Snowflake accounts

Output

The desired data was transferred to Snowflake (from NetSuite and Shopify) and a scheduled run was set up