XCom Sending Data Between Airflow Tasks

Introduction

In this article we will walk through the most commonly used method to sharing data between airflow tasks with example. In airflow XCom use to communicate small message between tasks and a good way to passing large data between tasks is to use a remote storage such as S3 or HDFS. In this article we will see how to use XCom to pass data between tasks.

What is XCom?

Xcoms (cross-communications) are a mechanism that let tasks talk to each other. But they only appropriate for small amounts of data. Do not use them to pass around large values like dataframes and others. Xcom identified by a key as well as the task_id and dag_id it came from. XCom storage using xcom_push and xcom_pull methods. xcom_pull can be used in task to receive data. Many operators will auto-push their results into XCom key called return_value if do_xcom_push is set to True.

Example of XCom

Let’s write a code. This is a simple example. First we import all necessary libraries.

from datetime import datetime
from airflow.models import DAG
from airflow.operators.python import PythonOperator

Let’s create a DAG to execute multiple tasks. Here we are doing xcom_pull using task_id.

def data_collect1(num1):
    print("...........Task1.........")
    num2 = num1 + 10000
    return num2


def data_collect2(ti):
    print("...........Task2...............")
    num = ti.xcom_pull(task_ids='Data_Source1')
    num = num-40000
    print(num)
	
with DAG(dag_id='xcom_example', schedule_interval='@daily', start_date=datetime(year=2022, month=7, day=21), catchup=False) as dag:

    data_collect1 = PythonOperator(
        task_id='Data_Source1',
        python_callable=data_collect1,
        op_kwargs={"num1": 59},
        do_xcom_push=True
    )

    data_collect2 = PythonOperator(
        task_id='Data_Source2',
        python_callable=data_collect2
    )

    data_collect1 >> data_collect2 

Apart from this there are many ways by which we can pass the data in between tasks.

More Info

XComs — Airflow Documentation (apache.org)

(588) Airflow XCom for Beginners – All you have to know in 10 mins – YouTube

Airflow – pytechie.com

If you face any problem in this article then definitely let us know.

keep reading 📖📖📖📖📖📖📖😁