Gtjkyu

How to dynamically update parameters of an existing Airflow (1.9 version)Connection within code? I have defined a SSH connection via Airflow Admin UI. However I am only defining a service account , host and port in the UI. I am retrieving the password in the first task instance and I need to update the SSH connection with the password in the second task instance and use it in the third task instance. Currently t1 and t2 work as expected ,however t3 fails since the password is not getting updated and it is looking for .ssh key file based authentication. Can someone please suggest how this can be implemented ? .ssh Here is my code snippet : from airflow import models from airflow.contrib.operators.ssh_operator import SSHOperator from airflow.operators.python_operator import PythonOperator from airflow.operators.bash_operator import BashOperator from datetime import datetime, timedelta from airflow.contrib.hooks.ssh_hook import SSHHook from airflow.models impo...

Airflow Why the scheduler doesn't start my DAG? I have the following Dag: The first Dag with 0 1 * * * ran without any problem. The end DAG 0 10 1 * * Did not run. When I do: 0 1 * * * 0 10 1 * * import datetime print datetime.datetime.now() I get: 2018-07-01 12:14:15.632812 So I don't understand why this DAG hasn't been scheduled. I understand that it's not mandatory to run exactly at 10:00 but the stat should be Running . Running According to the "Latest Run" of the first task with is 2018-06-30 01:00 I suspect that I don't actually understand Airflow clock. From my point of view the last run was on 2018-07-01 01:00 Because it ran today morning not yesterday. 2018-06-30 01:00 2018-07-01 01:00 Edit: I saw this paragraph at the documntation: "Note that if you run a DAG on a schedule_interval of one day, the run stamped 2016-01-01 will be trigger soon after 2016-01-01T23:59. In other words, the job instance is started once the period it covers has ...