# Initialize Domino API object with the api_key and hostĪpi_key =Variable. dummy_operator import DummyOperatorįrom airflow. I have to add that I didn't encounter this scheduler problem with a self-made Airflow Kubernetes setup, but then I was using a bleeding-edge Airflow version with PostgreSQL.From datetime import datetime, timedeltaįrom airflow. The Airflow Scheduler image is gcr.io/cloud-airflow-releaser/airflow-worker-scheduler-1.9.0:cloud_composer_service_-RC3. However, this is an out-of-the-box Google Composer with the default environment, so I wonder if anyone else has had a similar problem or has some idea what's going on? I've understood that Composer uses Google Cloud SQL for the DB, and apparently(?) MySQL backend. I'm out of my depth with technical RDBMS errors. KubernetesExecutor requires a non-sqlite database in the backend. The scheduler itself does not necessarily need to be running on Kubernetes, but does need access to a Kubernetes cluster. KubernetesExecutor runs as a process in the Airflow Scheduler. When I commented out the app.run, it worked. The Kubernetes executor runs each task instance in its own pod on a Kubernetes cluster. In my case, I was running gunicorn run:app where run.py had: from app import app app.run (debugTrue) in it. : (_mysql_exceptions.OperationalError) (1205, 'Lock wait timeout exceeded try restarting transaction') (Background on this error at: ) For anyone who has nothing running on the port to kill, if app.run () is in the file gunicorn is running, then it will run twice on the port and keep attempting to retry the connection. Docker configuration for Airflow airflow-scheduler - The scheduler monitors. Reraise(type(exception), exception, tb=exc_tb, cause=cause)įile "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_contextįile "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 508, in do_executeįile "/usr/local/lib/python2.7/site-packages/MySQLdb/cursors.py", line 250, in executeįile "/usr/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 50, in defaulterrorhandler We use airflow helm charts to deploy airflow. Return connection._execute_clauseelement(self, multiparams, params)įile "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelementįile "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_contextįile "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exceptionįile "/usr/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause Result = conn.execute(querycontext.statement, self._params)įile "/usr/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in executeįile "/usr/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection Airflow executes tasks of a DAG on different servers in case you are using Kubernetes executor or Celery executor.Therefore, you should not store any file or config in the local filesystem as the next task is likely to run on a different server without access to it for example, a task that downloads the data file that the next task processes. Return self._execute_and_instances(context)įile "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2958, in _execute_and_instances filter(or_(*filter_for_tis), TI.state.in_(resettable_states))įile "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2783, in allįile "/usr/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2935, in _iter_ Self.reset_state_for_orphaned_tasks(session=session)įile "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapperįile "/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 266, in reset_state_for_orphaned_tasks Here's the traceback from Stackdriver: Traceback (most recent call last):įile "/usr/local/bin/airflow", line 27, in įile "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 826, in schedulerįile "/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 198, in runįile "/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 1549, in _executeįile "/usr/local/lib/python2.7/site-packages/airflow/jobs.py", line 1594, in _execute_helper The error is very similar to what I've also had before. Communication between workers and the Scheduler is performed through a message broker interface (can be implemented using Redis or RabbitMQ). This time, there is a CrashLoopBackOff and the scheduler pod cannot restart anymore. The scheduler container crashes sometimes, and it can get into a locked situation in which it cannot start any new tasks (an error with the database connection) so I have to re-create the whole Composer environment. I've been using Google Composer for a while ( composer-0.5.2-airflow-1.9.0), and had some problems with the Airflow scheduler.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |