![]() ![]() For more information, refer to Airflow documentation. In recent releases, the community has added more time zone-aware features to the Airflow UI. If you’re based in US Pacific Time, a DAG run of 19:00 will correspond to 12:00 local time. You should not expect your DAG executions to correspond to your local timezone. This behavior is shared by many databases and APIs, but it’s worth clarifying. Airflow Time ZonesĪirflow stores datetime information in UTC internally and in the database. You might be creating timetables without even knowing it: if you define a schedule-interval, Airflow 2.2+ will convert it to a timetable behind the scenes. We recommend using timetables as your de facto scheduling mechanism in Airflow 2.2+. And because timetables are imported as Airflow plugins, you can use community-developed timetables to quickly - and literally - get your DAG up to speed. You can program varying schedules, conditional logic, and more, directly within your DAG schedule. Essentially, timetable is a DAG-level parameter that you can set to a Python function that contains your execution schedule.Ī timetable is significantly more customizable than a cron job or timedelta. This is why Airflow 2.2 introduced timetables as the new default scheduling method. Scheduling DAGs to skip holidays, run only at certain times, or otherwise run on varying intervals can cause major headaches if you’re relying solely on cron jobs or timedeltas. There are some data engineering use cases that are difficult or even impossible to address with Airflow’s original scheduling method. Thankfully, Airflow 2.2+ simplifies DAG scheduling with the introduction of the timetables! Use Timetables for Simpler Scheduling ![]() This quirk is specific to Apache Airflow, and it’s important to remember - especially if you’re using default variables and macros. This happens because Airflow can’t ensure that all of the data from 2:00 PM - 3:00 PM is present until the end of that hourly interval. An hourly DAG, for example, will execute its 2:00 PM run when the clock strikes 3:00 PM.
0 Comments
Leave a Reply. |