Apache Airflow DAG

An Apache Airflow DAG is a automated workflow DAG used by Apache Airflow.

• Context:
  • It can (typically) be represented by an Airflow DAG (Python) File.
  • …
• See: Apache Airflow DAG File, Automated Workflow, Airflow DAG Validation Test, Airflow DAG Unit Test.

---

---

References

2021

• [Do not perform data processing in DAG files]."
  • QUOTE: ... Since DAGs are python-based, we will definitely be tempted to use pandas or similar stuff in DAG, but we should not. Airflow is an orchestrator, not an execution framework. All computation should be delegated to a specific target system. Follow the fire and track approach. Use the operator to start the task and the sensor to track the completion. Airflow is not designed for long-running tasks.

2018

• https://medium.com/@chandukavar/testing-in-airflow-part-1-dag-validation-tests-dag-definition-tests-and-unit-tests-2aa94970570c
  • QUOTE: ... In Airflow, a DAG– or a Directed Acyclic Graph – is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies. ...
    • 
  • … DAG validation tests are common for all the DAGs in Airflow, hence we don’t need to write a separate test for each DAG. This test will check the correctness of each DAG. It will also check whether a graph contains cycle or not. Tests will fail even if we have a typo in any of the DAG. Moreover, if we want to enforce developers to add certain default arguments to each DAG, we can write a test around that as well. Here are few validation tests ...

---