PythonVirtualenvOperator using airflow module fails to execute with AttributeError: module 'airflow' has no attribute 'utils'
Question:
I have Airflow deployed in virtual env and in case I try to execute PythonVirtualenvOperator with import of the Airflow module (to get Variables for example) it gives me the AttributeError. Guess I do not fully understand how Airflow executes VirtualenvOperator, and therefore what to do to overcome it, so any suggestions and insights will be highly appreciated
My test DAG code
from airflow import DAG
from airflow.operators.python import PythonVirtualenvOperator
from airflow.utils.dates import days_ago
args = {
'owner': 'airflow',
}
dag = DAG(
dag_id='example_python_operator_shieet',
default_args=args,
schedule_interval=None,
start_date=days_ago(2),
tags=['example'],
)
def callable_virtualenv():
from time import sleep
from colorama import Back, Fore, Style
from airflow.models import Variable
print(Fore.RED + 'some red text')
print(Back.GREEN + 'and with a green background')
print(Style.DIM + 'and in dim text')
print(Style.RESET_ALL)
for _ in range(10):
print(Style.DIM + 'Please wait...', flush=True)
sleep(10)
print('Finished')
virtualenv_task = PythonVirtualenvOperator(
task_id="virtualenv_python",
python_callable=callable_virtualenv,
requirements=["colorama==0.4.0",'apache-airflow==2.0.1'],
system_site_packages=False,
dag=dag,
)
running which gives me such log same for all cases when PythonVirtualenvOperator uses airflow module
*** Reading local file: /opt1/app_data/airflow/logs/weather_update/updating_forecasts/2021-04-19T12:27:09.086267+00:00/1.log
[2021-04-19 15:27:10,221] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [queued]>
[2021-04-19 15:27:10,234] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [queued]>
[2021-04-19 15:27:10,234] {taskinstance.py:1042} INFO -
--------------------------------------------------------------------------------
[2021-04-19 15:27:10,234] {taskinstance.py:1043} INFO - Starting attempt 1 of 1
[2021-04-19 15:27:10,234] {taskinstance.py:1044} INFO -
--------------------------------------------------------------------------------
[2021-04-19 15:27:10,241] {taskinstance.py:1063} INFO - Executing <Task(PythonVirtualenvOperator): updating_forecasts> on 2021-04-19T12:27:09.086267+00:00
[2021-04-19 15:27:10,244] {standard_task_runner.py:52} INFO - Started process 24052 to run task
[2021-04-19 15:27:10,248] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'weather_update', 'updating_forecasts', '2021-04-19T12:27:09.086267+00:00', '--job-id', '73', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/weather_dag.py', '--cfg-path', '/tmp/tmp2j8ueojr', '--error-file', '/tmp/tmpvh9s78fv']
[2021-04-19 15:27:10,248] {standard_task_runner.py:77} INFO - Job 73: Subtask updating_forecasts
[2021-04-19 15:27:10,289] {logging_mixin.py:104} INFO - Running <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [running]> on host sk-bi-08.erkapharm.ru
[2021-04-19 15:27:10,330] {taskinstance.py:1257} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=#####################
AIRFLOW_CTX_DAG_OWNER=###########
AIRFLOW_CTX_DAG_ID=weather_update
AIRFLOW_CTX_TASK_ID=updating_forecasts
AIRFLOW_CTX_EXECUTION_DATE=2021-04-19T12:27:09.086267+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-19T12:27:09.086267+00:00
[2021-04-19 15:27:10,331] {process_utils.py:135} INFO - Executing cmd: virtualenv /tmp/venvxuo1gayl
[2021-04-19 15:27:10,336] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:10,556] {process_utils.py:141} INFO - created virtual environment CPython3.7.3.final.0-64 in 134ms
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - creator CPython3Posix(dest=/tmp/venvxuo1gayl, clear=False, no_vcs_ignore=False, global=False)
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/RKF/###########/.local/share/virtualenv)
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - added seed packages: pip==21.0.1, setuptools==54.2.0, wheel==0.36.2
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
[2021-04-19 15:27:10,574] {process_utils.py:135} INFO - Executing cmd: /tmp/venvxuo1gayl/bin/pip install pandas numpy pyodbc sqlalchemy apache-airflow==2.0.1
[2021-04-19 15:27:10,579] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:11,603] {process_utils.py:141} INFO - Collecting pandas
[2021-04-19 15:27:11,644] {process_utils.py:141} INFO - Using cached pandas-1.2.4-cp37-cp37m-manylinux1_x86_64.whl (9.9 MB)
[2021-04-19 15:27:12,048] {process_utils.py:141} INFO - Collecting numpy
[2021-04-19 15:27:12,112] {process_utils.py:141} INFO - Using cached numpy-1.20.2-cp37-cp37m-manylinux2010_x86_64.whl (15.3 MB)
[2021-04-19 15:27:12,287] {process_utils.py:141} INFO - Collecting pyodbc
[2021-04-19 15:27:12,287] {process_utils.py:141} INFO - Using cached pyodbc-4.0.30-cp37-cp37m-linux_x86_64.whl
[2021-04-19 15:27:12,519] {process_utils.py:141} INFO - Collecting sqlalchemy
[2021-04-19 15:27:12,530] {process_utils.py:141} INFO - Using cached SQLAlchemy-1.4.9-cp37-cp37m-manylinux2014_x86_64.whl (1.5 MB)
[2021-04-19 15:27:12,634] {process_utils.py:141} INFO - Collecting apache-airflow==2.0.1
[2021-04-19 15:27:12,699] {process_utils.py:141} INFO - Using cached apache_airflow-2.0.1-py3-none-any.whl (4.5 MB)
[2021-04-19 15:27:14,536] {process_utils.py:141} INFO - Collecting sqlalchemy-jsonfield~=1.0
[2021-04-19 15:27:14,538] {process_utils.py:141} INFO - Using cached SQLAlchemy_JSONField-1.0.0-py3-none-any.whl (10 kB)
[2021-04-19 15:27:14,603] {process_utils.py:141} INFO - Collecting attrs<21.0,>=20.0
[2021-04-19 15:27:14,606] {process_utils.py:141} INFO - Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
[2021-04-19 15:27:14,662] {process_utils.py:141} INFO - Collecting python-slugify<5.0,>=3.0.0
[2021-04-19 15:27:14,662] {process_utils.py:141} INFO - Using cached python_slugify-4.0.1-py2.py3-none-any.whl
[2021-04-19 15:27:14,716] {process_utils.py:141} INFO - Collecting itsdangerous>=1.1.0
[2021-04-19 15:27:14,718] {process_utils.py:141} INFO - Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
[2021-04-19 15:27:14,799] {process_utils.py:141} INFO - Collecting graphviz>=0.12
[2021-04-19 15:27:14,801] {process_utils.py:141} INFO - Using cached graphviz-0.16-py2.py3-none-any.whl (19 kB)
[2021-04-19 15:27:14,895] {process_utils.py:141} INFO - Collecting markdown<4.0,>=2.5.2
[2021-04-19 15:27:14,897] {process_utils.py:141} INFO - Using cached Markdown-3.3.4-py3-none-any.whl (97 kB)
[2021-04-19 15:27:14,970] {process_utils.py:141} INFO - Collecting flask<2.0,>=1.1.0
[2021-04-19 15:27:14,973] {process_utils.py:141} INFO - Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
[2021-04-19 15:27:15,104] {process_utils.py:141} INFO - Collecting connexion[flask,swagger-ui]<3,>=2.6.0
[2021-04-19 15:27:15,107] {process_utils.py:141} INFO - Using cached connexion-2.7.0-py2.py3-none-any.whl (77 kB)
[2021-04-19 15:27:15,199] {process_utils.py:141} INFO - Collecting requests>=2.20.0
[2021-04-19 15:27:15,202] {process_utils.py:141} INFO - Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
[2021-04-19 15:27:15,286] {process_utils.py:141} INFO - Collecting markupsafe<2.0,>=1.1.1
[2021-04-19 15:27:15,288] {process_utils.py:141} INFO - Using cached MarkupSafe-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl (33 kB)
[2021-04-19 15:27:15,366] {process_utils.py:141} INFO - Collecting python-dateutil<3,>=2.3
[2021-04-19 15:27:15,370] {process_utils.py:141} INFO - Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
[2021-04-19 15:27:15,494] {process_utils.py:141} INFO - Collecting rich==9.2.0
[2021-04-19 15:27:15,498] {process_utils.py:141} INFO - Using cached rich-9.2.0-py3-none-any.whl (164 kB)
[2021-04-19 15:27:15,555] {process_utils.py:141} INFO - Collecting marshmallow-oneofschema>=2.0.1
[2021-04-19 15:27:15,556] {process_utils.py:141} INFO - Using cached marshmallow_oneofschema-2.1.0-py2.py3-none-any.whl (5.7 kB)
[2021-04-19 15:27:15,614] {process_utils.py:141} INFO - Collecting typing-extensions>=3.7.4
[2021-04-19 15:27:15,616] {process_utils.py:141} INFO - Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
[2021-04-19 15:27:15,682] {process_utils.py:141} INFO - Collecting flask-caching<2.0.0,>=1.5.0
[2021-04-19 15:27:15,684] {process_utils.py:141} INFO - Using cached Flask_Caching-1.10.1-py3-none-any.whl (34 kB)
[2021-04-19 15:27:15,744] {process_utils.py:141} INFO - Collecting tabulate<0.9,>=0.7.5
[2021-04-19 15:27:15,745] {process_utils.py:141} INFO - Using cached tabulate-0.8.9-py3-none-any.whl (25 kB)
[2021-04-19 15:27:15,813] {process_utils.py:141} INFO - Collecting jinja2<2.12.0,>=2.10.1
[2021-04-19 15:27:15,817] {process_utils.py:141} INFO - Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
[2021-04-19 15:27:15,872] {process_utils.py:141} INFO - Collecting apache-airflow-providers-http
[2021-04-19 15:27:15,874] {process_utils.py:141} INFO - Using cached apache_airflow_providers_http-1.1.1-py3-none-any.whl (20 kB)
[2021-04-19 15:27:16,201] {process_utils.py:141} INFO - Collecting cryptography>=0.9.3
[2021-04-19 15:27:16,208] {process_utils.py:141} INFO - Using cached cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
[2021-04-19 15:27:16,314] {process_utils.py:141} INFO - Collecting croniter<0.4,>=0.3.17
[2021-04-19 15:27:16,316] {process_utils.py:141} INFO - Using cached croniter-0.3.37-py2.py3-none-any.whl (13 kB)
[2021-04-19 15:27:16,398] {process_utils.py:141} INFO - Collecting alembic<2.0,>=1.2
[2021-04-19 15:27:16,400] {process_utils.py:141} INFO - Using cached alembic-1.5.8-py2.py3-none-any.whl (159 kB)
[2021-04-19 15:27:16,456] {process_utils.py:141} INFO - Collecting python-nvd3~=0.15.0
[2021-04-19 15:27:16,456] {process_utils.py:141} INFO - Using cached python_nvd3-0.15.0-py3-none-any.whl
[2021-04-19 15:27:16,507] {process_utils.py:141} INFO - Collecting unicodecsv>=0.14.1
[2021-04-19 15:27:16,507] {process_utils.py:141} INFO - Using cached unicodecsv-0.14.1-py3-none-any.whl
[2021-04-19 15:27:16,568] {process_utils.py:141} INFO - Collecting setproctitle<2,>=1.1.8
[2021-04-19 15:27:16,570] {process_utils.py:141} INFO - Using cached setproctitle-1.2.2-cp37-cp37m-manylinux1_x86_64.whl (36 kB)
[2021-04-19 15:27:16,628] {process_utils.py:141} INFO - Collecting flask-login<0.5,>=0.3
[2021-04-19 15:27:16,628] {process_utils.py:141} INFO - Using cached Flask_Login-0.4.1-py2.py3-none-any.whl
[2021-04-19 15:27:16,682] {process_utils.py:141} INFO - Collecting apache-airflow-providers-imap
[2021-04-19 15:27:16,684] {process_utils.py:141} INFO - Using cached apache_airflow_providers_imap-1.0.1-py3-none-any.whl (15 kB)
[2021-04-19 15:27:16,768] {process_utils.py:141} INFO - Collecting colorlog>=4.0.2
[2021-04-19 15:27:16,770] {process_utils.py:141} INFO - Using cached colorlog-5.0.1-py2.py3-none-any.whl (10 kB)
[2021-04-19 15:27:16,874] {process_utils.py:141} INFO - Collecting pygments<3.0,>=2.0.1
[2021-04-19 15:27:16,878] {process_utils.py:141} INFO - Using cached Pygments-2.8.1-py3-none-any.whl (983 kB)
[2021-04-19 15:27:16,976] {process_utils.py:141} INFO - Collecting argcomplete~=1.10
[2021-04-19 15:27:16,978] {process_utils.py:141} INFO - Using cached argcomplete-1.12.2-py2.py3-none-any.whl (38 kB)
[2021-04-19 15:27:17,055] {process_utils.py:141} INFO - Collecting importlib-resources~=1.4
[2021-04-19 15:27:17,057] {process_utils.py:141} INFO - Using cached importlib_resources-1.5.0-py2.py3-none-any.whl (21 kB)
[2021-04-19 15:27:17,113] {process_utils.py:141} INFO - Collecting apache-airflow-providers-sqlite
[2021-04-19 15:27:17,114] {process_utils.py:141} INFO - Using cached apache_airflow_providers_sqlite-1.0.2-py3-none-any.whl (14 kB)
[2021-04-19 15:27:17,185] {process_utils.py:141} INFO - Collecting cattrs~=1.1
[2021-04-19 15:27:17,187] {process_utils.py:141} INFO - Using cached cattrs-1.5.0-py3-none-any.whl (19 kB)
[2021-04-19 15:27:17,274] {process_utils.py:141} INFO - Collecting pyjwt<2
[2021-04-19 15:27:17,275] {process_utils.py:141} INFO - Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
[2021-04-19 15:27:17,397] {process_utils.py:141} INFO - Collecting lazy-object-proxy
[2021-04-19 15:27:17,398] {process_utils.py:141} INFO - Using cached lazy_object_proxy-1.6.0-cp37-cp37m-manylinux1_x86_64.whl (55 kB)
[2021-04-19 15:27:17,457] {process_utils.py:141} INFO - Collecting flask-wtf<0.15,>=0.14.3
[2021-04-19 15:27:17,459] {process_utils.py:141} INFO - Using cached Flask_WTF-0.14.3-py2.py3-none-any.whl (13 kB)
[2021-04-19 15:27:17,603] {process_utils.py:141} INFO - Collecting flask-appbuilder~=3.1.1
[2021-04-19 15:27:17,632] {process_utils.py:141} INFO - Using cached Flask_AppBuilder-3.1.1-py3-none-any.whl (1.7 MB)
[2021-04-19 15:27:17,696] {process_utils.py:141} INFO - Collecting apache-airflow-providers-ftp
[2021-04-19 15:27:17,698] {process_utils.py:141} INFO - Using cached apache_airflow_providers_ftp-1.0.1-py3-none-any.whl (14 kB)
*some other apache-airflow dependencies installation logs*
[2021-04-19 15:27:23,603] {process_utils.py:141} INFO - Collecting certifi>=2017.4.17
[2021-04-19 15:27:23,607] {process_utils.py:141} INFO - Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
[2021-04-19 15:27:24,201] {process_utils.py:141} INFO - Collecting isodate
[2021-04-19 15:27:24,203] {process_utils.py:141} INFO - Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
[2021-04-19 15:27:26,537] {process_utils.py:141} INFO - Installing collected packages: zipp, six, pyrsistent, importlib-metadata, attrs, markupsafe, jsonschema, isodate, werkzeug, urllib3, PyYAML, pytz, openapi-schema-validator, jinja2, itsdangerous, idna, greenlet, defusedxml, click, chardet, certifi, WTForms, text-unidecode, sqlalchemy, requests, python3-openid, pyjwt, pycparser, openapi-spec-validator, marshmallow, inflection, flask, dnspython, clickclick, Babel, apispec, typing-extensions, swagger-ui-bundle, sqlalchemy-utils, pytzdata, python-slugify, python-editor, python-dateutil, pygments, prison, numpy, natsort, marshmallow-sqlalchemy, marshmallow-enum, Mako, lockfile, flask-wtf, Flask-SQLAlchemy, Flask-OpenID, flask-login, Flask-JWT-Extended, Flask-Babel, email-validator, docutils, connexion, commonmark, colorama, cffi, unicodecsv, termcolor, tenacity, tabulate, sqlalchemy-jsonfield, setproctitle, rich, python-nvd3, python-daemon, psutil, pendulum, pandas, marshmallow-oneofschema, markdown, lazy-object-proxy, iso8601, importlib-resources, gunicorn, graphviz, flask-caching, flask-appbuilder, dill, cryptography, croniter, colorlog, cattrs, cached-property, argcomplete, apache-airflow-providers-sqlite, apache-airflow-providers-imap, apache-airflow-providers-http, apache-airflow-providers-ftp, alembic, pyodbc, apache-airflow
[2021-04-19 15:27:35,766] {process_utils.py:141} INFO - Successfully installed Babel-2.9.0 Flask-Babel-1.0.0 Flask-JWT-Extended-3.25.1 Flask-OpenID-1.2.5 Flask-SQLAlchemy-2.5.1 Mako-1.1.4 PyYAML-5.4.1 WTForms-2.3.3 alembic-1.5.8 apache-airflow-2.0.1 apache-airflow-providers-ftp-1.0.1 apache-airflow-providers-http-1.1.1 apache-airflow-providers-imap-1.0.1 apache-airflow-providers-sqlite-1.0.2 apispec-3.3.2 argcomplete-1.12.2 attrs-20.3.0 cached-property-1.5.2 cattrs-1.5.0 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 click-7.1.2 clickclick-20.10.2 colorama-0.4.4 colorlog-5.0.1 commonmark-0.9.1 connexion-2.7.0 croniter-0.3.37 cryptography-3.4.7 defusedxml-0.7.1 dill-0.3.3 dnspython-2.1.0 docutils-0.17.1 email-validator-1.1.2 flask-1.1.2 flask-appbuilder-3.1.1 flask-caching-1.10.1 flask-login-0.4.1 flask-wtf-0.14.3 graphviz-0.16 greenlet-1.0.0 gunicorn-19.10.0 idna-2.10 importlib-metadata-1.7.0 importlib-resources-1.5.0 inflection-0.5.1 iso8601-0.1.14 isodate-0.6.0 itsdangerous-1.1.0 jinja2-2.11.3 jsonschema-3.2.0 lazy-object-proxy-1.6.0 lockfile-0.12.2 markdown-3.3.4 markupsafe-1.1.1 marshmallow-3.11.1 marshmallow-enum-1.5.1 marshmallow-oneofschema-2.1.0 marshmallow-sqlalchemy-0.23.1 natsort-7.1.1 numpy-1.20.2 openapi-schema-validator-0.1.5 openapi-spec-validator-0.3.0 pandas-1.2.4 pendulum-2.1.2 prison-0.1.3 psutil-5.8.0 pycparser-2.20 pygments-2.8.1 pyjwt-1.7.1 pyodbc-4.0.30 pyrsistent-0.17.3 python-daemon-2.3.0 python-dateutil-2.8.1 python-editor-1.0.4 python-nvd3-0.15.0 python-slugify-4.0.1 python3-openid-3.2.0 pytz-2021.1 pytzdata-2020.1 requests-2.25.1 rich-9.2.0 setproctitle-1.2.2 six-1.15.0 sqlalchemy-1.4.9 sqlalchemy-jsonfield-1.0.0 sqlalchemy-utils-0.37.0 swagger-ui-bundle-0.0.8 tabulate-0.8.9 tenacity-6.2.0 termcolor-1.1.0 text-unidecode-1.3 typing-extensions-3.7.4.3 unicodecsv-0.14.1 urllib3-1.26.4 werkzeug-1.0.1 zipp-3.4.1
[2021-04-19 15:27:36,306] {process_utils.py:135} INFO - Executing cmd: /tmp/venvxuo1gayl/bin/python /tmp/venvxuo1gayl/script.py /tmp/venvxuo1gayl/script.in /tmp/venvxuo1gayl/script.out /tmp/venvxuo1gayl/string_args.txt
[2021-04-19 15:27:36,312] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:36,875] {process_utils.py:141} INFO - [[34m2021-04-19 15:27:36,875[0m] {[34mlogging_config.py:[0m59} ERROR[0m - Unable to load the config, contains a configuration error.[0m
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 93, in _resolve
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - found = getattr(found, n)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - AttributeError: module 'airflow' has no attribute 'utils'
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - During handling of the above exception, another exception occurred:
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 542, in configure
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - formatters[name])
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 672, in configure_formatter
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - c = _resolve(cname)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 96, in _resolve
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - found = getattr(found, n)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - AttributeError: module 'airflow' has no attribute 'utils'
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - The above exception was the direct cause of the following exception:
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/script.py", line 132, in <module>
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - res = weather_update(*arg_dict["args"], **arg_dict["kwargs"])
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/script.py", line 40, in weather_update
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - from airflow.models import Variable
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/__init__.py", line 46, in <module>
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - settings.initialize()
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/settings.py", line 434, in initialize
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - LOGGING_CLASS_PATH = configure_logging()
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/logging_config.py", line 62, in configure_logging
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - raise e
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/logging_config.py", line 57, in configure_logging
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - dictConfig(logging_config)
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 799, in dictConfig
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - dictConfigClass(config).configure()
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 545, in configure
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - 'formatter %r' % name) from e
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - ValueError: Unable to configure formatter 'airflow_coloured'
[2021-04-19 15:27:37,210] {taskinstance.py:1455} ERROR - Command '['/tmp/venvxuo1gayl/bin/python', '/tmp/venvxuo1gayl/script.py', '/tmp/venvxuo1gayl/script.in', '/tmp/venvxuo1gayl/script.out', '/tmp/venvxuo1gayl/string_args.txt']' returned non-zero exit status 1.
Traceback (most recent call last):
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
result = task_copy.execute(context=context)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py", line 493, in execute
super().execute(context=serializable_context)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py", line 117, in execute
return_value = self.execute_callable()
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py", line 531, in execute_callable
string_args_filename,
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/utils/process_utils.py", line 145, in execute_in_subprocess
raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/tmp/venvxuo1gayl/bin/python', '/tmp/venvxuo1gayl/script.py', '/tmp/venvxuo1gayl/script.in', '/tmp/venvxuo1gayl/script.out', '/tmp/venvxuo1gayl/string_args.txt']' returned non-zero exit status 1.
[2021-04-19 15:27:37,212] {taskinstance.py:1503} INFO - Marking task as FAILED. dag_id=weather_update, task_id=updating_forecasts, execution_date=20210419T122709, start_date=20210419T122710, end_date=20210419T122737
[2021-04-19 15:27:37,227] {configuration.py:356} WARNING - section/key [smtp/smtp_user] not found in config
Answers:
It seems that you are confusing the use-cases for PythonVirtualenvOperator and PythonOperator.
If you simply want to run a Python callable in a task (callable_virtualenv() in your case) you can use PythonOperator. In this case, it does not matter if you installed Airflow in a virtual environment, system wide, or using Docker.
What happens in your code is the following: PythonVirtualenvOperator creates another virtual environment (which is completely unrelated to the one in which you run Airflow), installs Airflow into it, and tries to import Variable. But this another Airflow installation is not configured and that is why you get those exceptions. You could set the AIRFLOW_HOME environment variable for this second Airflow installation to the same directory as used by the first Airflow installation, and this should actually work, but it looks like an overkill to me.
So, what you can do is install colorama into the same environment in which you installed Airflow and replace PythonVirtualenvOperator by PythonOperator.
BTW, those print() inside the callable would be redirected into a log file and not printed to terminal, so it probably does not make much sense to use colorama with them.
I have Airflow deployed in virtual env and in case I try to execute PythonVirtualenvOperator with import of the Airflow module (to get Variables for example) it gives me the AttributeError. Guess I do not fully understand how Airflow executes VirtualenvOperator, and therefore what to do to overcome it, so any suggestions and insights will be highly appreciated
My test DAG code
from airflow import DAG
from airflow.operators.python import PythonVirtualenvOperator
from airflow.utils.dates import days_ago
args = {
'owner': 'airflow',
}
dag = DAG(
dag_id='example_python_operator_shieet',
default_args=args,
schedule_interval=None,
start_date=days_ago(2),
tags=['example'],
)
def callable_virtualenv():
from time import sleep
from colorama import Back, Fore, Style
from airflow.models import Variable
print(Fore.RED + 'some red text')
print(Back.GREEN + 'and with a green background')
print(Style.DIM + 'and in dim text')
print(Style.RESET_ALL)
for _ in range(10):
print(Style.DIM + 'Please wait...', flush=True)
sleep(10)
print('Finished')
virtualenv_task = PythonVirtualenvOperator(
task_id="virtualenv_python",
python_callable=callable_virtualenv,
requirements=["colorama==0.4.0",'apache-airflow==2.0.1'],
system_site_packages=False,
dag=dag,
)
running which gives me such log same for all cases when PythonVirtualenvOperator uses airflow module
*** Reading local file: /opt1/app_data/airflow/logs/weather_update/updating_forecasts/2021-04-19T12:27:09.086267+00:00/1.log
[2021-04-19 15:27:10,221] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [queued]>
[2021-04-19 15:27:10,234] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [queued]>
[2021-04-19 15:27:10,234] {taskinstance.py:1042} INFO -
--------------------------------------------------------------------------------
[2021-04-19 15:27:10,234] {taskinstance.py:1043} INFO - Starting attempt 1 of 1
[2021-04-19 15:27:10,234] {taskinstance.py:1044} INFO -
--------------------------------------------------------------------------------
[2021-04-19 15:27:10,241] {taskinstance.py:1063} INFO - Executing <Task(PythonVirtualenvOperator): updating_forecasts> on 2021-04-19T12:27:09.086267+00:00
[2021-04-19 15:27:10,244] {standard_task_runner.py:52} INFO - Started process 24052 to run task
[2021-04-19 15:27:10,248] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'weather_update', 'updating_forecasts', '2021-04-19T12:27:09.086267+00:00', '--job-id', '73', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/weather_dag.py', '--cfg-path', '/tmp/tmp2j8ueojr', '--error-file', '/tmp/tmpvh9s78fv']
[2021-04-19 15:27:10,248] {standard_task_runner.py:77} INFO - Job 73: Subtask updating_forecasts
[2021-04-19 15:27:10,289] {logging_mixin.py:104} INFO - Running <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [running]> on host sk-bi-08.erkapharm.ru
[2021-04-19 15:27:10,330] {taskinstance.py:1257} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=#####################
AIRFLOW_CTX_DAG_OWNER=###########
AIRFLOW_CTX_DAG_ID=weather_update
AIRFLOW_CTX_TASK_ID=updating_forecasts
AIRFLOW_CTX_EXECUTION_DATE=2021-04-19T12:27:09.086267+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-19T12:27:09.086267+00:00
[2021-04-19 15:27:10,331] {process_utils.py:135} INFO - Executing cmd: virtualenv /tmp/venvxuo1gayl
[2021-04-19 15:27:10,336] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:10,556] {process_utils.py:141} INFO - created virtual environment CPython3.7.3.final.0-64 in 134ms
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - creator CPython3Posix(dest=/tmp/venvxuo1gayl, clear=False, no_vcs_ignore=False, global=False)
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/RKF/###########/.local/share/virtualenv)
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - added seed packages: pip==21.0.1, setuptools==54.2.0, wheel==0.36.2
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
[2021-04-19 15:27:10,574] {process_utils.py:135} INFO - Executing cmd: /tmp/venvxuo1gayl/bin/pip install pandas numpy pyodbc sqlalchemy apache-airflow==2.0.1
[2021-04-19 15:27:10,579] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:11,603] {process_utils.py:141} INFO - Collecting pandas
[2021-04-19 15:27:11,644] {process_utils.py:141} INFO - Using cached pandas-1.2.4-cp37-cp37m-manylinux1_x86_64.whl (9.9 MB)
[2021-04-19 15:27:12,048] {process_utils.py:141} INFO - Collecting numpy
[2021-04-19 15:27:12,112] {process_utils.py:141} INFO - Using cached numpy-1.20.2-cp37-cp37m-manylinux2010_x86_64.whl (15.3 MB)
[2021-04-19 15:27:12,287] {process_utils.py:141} INFO - Collecting pyodbc
[2021-04-19 15:27:12,287] {process_utils.py:141} INFO - Using cached pyodbc-4.0.30-cp37-cp37m-linux_x86_64.whl
[2021-04-19 15:27:12,519] {process_utils.py:141} INFO - Collecting sqlalchemy
[2021-04-19 15:27:12,530] {process_utils.py:141} INFO - Using cached SQLAlchemy-1.4.9-cp37-cp37m-manylinux2014_x86_64.whl (1.5 MB)
[2021-04-19 15:27:12,634] {process_utils.py:141} INFO - Collecting apache-airflow==2.0.1
[2021-04-19 15:27:12,699] {process_utils.py:141} INFO - Using cached apache_airflow-2.0.1-py3-none-any.whl (4.5 MB)
[2021-04-19 15:27:14,536] {process_utils.py:141} INFO - Collecting sqlalchemy-jsonfield~=1.0
[2021-04-19 15:27:14,538] {process_utils.py:141} INFO - Using cached SQLAlchemy_JSONField-1.0.0-py3-none-any.whl (10 kB)
[2021-04-19 15:27:14,603] {process_utils.py:141} INFO - Collecting attrs<21.0,>=20.0
[2021-04-19 15:27:14,606] {process_utils.py:141} INFO - Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
[2021-04-19 15:27:14,662] {process_utils.py:141} INFO - Collecting python-slugify<5.0,>=3.0.0
[2021-04-19 15:27:14,662] {process_utils.py:141} INFO - Using cached python_slugify-4.0.1-py2.py3-none-any.whl
[2021-04-19 15:27:14,716] {process_utils.py:141} INFO - Collecting itsdangerous>=1.1.0
[2021-04-19 15:27:14,718] {process_utils.py:141} INFO - Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
[2021-04-19 15:27:14,799] {process_utils.py:141} INFO - Collecting graphviz>=0.12
[2021-04-19 15:27:14,801] {process_utils.py:141} INFO - Using cached graphviz-0.16-py2.py3-none-any.whl (19 kB)
[2021-04-19 15:27:14,895] {process_utils.py:141} INFO - Collecting markdown<4.0,>=2.5.2
[2021-04-19 15:27:14,897] {process_utils.py:141} INFO - Using cached Markdown-3.3.4-py3-none-any.whl (97 kB)
[2021-04-19 15:27:14,970] {process_utils.py:141} INFO - Collecting flask<2.0,>=1.1.0
[2021-04-19 15:27:14,973] {process_utils.py:141} INFO - Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
[2021-04-19 15:27:15,104] {process_utils.py:141} INFO - Collecting connexion[flask,swagger-ui]<3,>=2.6.0
[2021-04-19 15:27:15,107] {process_utils.py:141} INFO - Using cached connexion-2.7.0-py2.py3-none-any.whl (77 kB)
[2021-04-19 15:27:15,199] {process_utils.py:141} INFO - Collecting requests>=2.20.0
[2021-04-19 15:27:15,202] {process_utils.py:141} INFO - Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
[2021-04-19 15:27:15,286] {process_utils.py:141} INFO - Collecting markupsafe<2.0,>=1.1.1
[2021-04-19 15:27:15,288] {process_utils.py:141} INFO - Using cached MarkupSafe-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl (33 kB)
[2021-04-19 15:27:15,366] {process_utils.py:141} INFO - Collecting python-dateutil<3,>=2.3
[2021-04-19 15:27:15,370] {process_utils.py:141} INFO - Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
[2021-04-19 15:27:15,494] {process_utils.py:141} INFO - Collecting rich==9.2.0
[2021-04-19 15:27:15,498] {process_utils.py:141} INFO - Using cached rich-9.2.0-py3-none-any.whl (164 kB)
[2021-04-19 15:27:15,555] {process_utils.py:141} INFO - Collecting marshmallow-oneofschema>=2.0.1
[2021-04-19 15:27:15,556] {process_utils.py:141} INFO - Using cached marshmallow_oneofschema-2.1.0-py2.py3-none-any.whl (5.7 kB)
[2021-04-19 15:27:15,614] {process_utils.py:141} INFO - Collecting typing-extensions>=3.7.4
[2021-04-19 15:27:15,616] {process_utils.py:141} INFO - Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
[2021-04-19 15:27:15,682] {process_utils.py:141} INFO - Collecting flask-caching<2.0.0,>=1.5.0
[2021-04-19 15:27:15,684] {process_utils.py:141} INFO - Using cached Flask_Caching-1.10.1-py3-none-any.whl (34 kB)
[2021-04-19 15:27:15,744] {process_utils.py:141} INFO - Collecting tabulate<0.9,>=0.7.5
[2021-04-19 15:27:15,745] {process_utils.py:141} INFO - Using cached tabulate-0.8.9-py3-none-any.whl (25 kB)
[2021-04-19 15:27:15,813] {process_utils.py:141} INFO - Collecting jinja2<2.12.0,>=2.10.1
[2021-04-19 15:27:15,817] {process_utils.py:141} INFO - Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
[2021-04-19 15:27:15,872] {process_utils.py:141} INFO - Collecting apache-airflow-providers-http
[2021-04-19 15:27:15,874] {process_utils.py:141} INFO - Using cached apache_airflow_providers_http-1.1.1-py3-none-any.whl (20 kB)
[2021-04-19 15:27:16,201] {process_utils.py:141} INFO - Collecting cryptography>=0.9.3
[2021-04-19 15:27:16,208] {process_utils.py:141} INFO - Using cached cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
[2021-04-19 15:27:16,314] {process_utils.py:141} INFO - Collecting croniter<0.4,>=0.3.17
[2021-04-19 15:27:16,316] {process_utils.py:141} INFO - Using cached croniter-0.3.37-py2.py3-none-any.whl (13 kB)
[2021-04-19 15:27:16,398] {process_utils.py:141} INFO - Collecting alembic<2.0,>=1.2
[2021-04-19 15:27:16,400] {process_utils.py:141} INFO - Using cached alembic-1.5.8-py2.py3-none-any.whl (159 kB)
[2021-04-19 15:27:16,456] {process_utils.py:141} INFO - Collecting python-nvd3~=0.15.0
[2021-04-19 15:27:16,456] {process_utils.py:141} INFO - Using cached python_nvd3-0.15.0-py3-none-any.whl
[2021-04-19 15:27:16,507] {process_utils.py:141} INFO - Collecting unicodecsv>=0.14.1
[2021-04-19 15:27:16,507] {process_utils.py:141} INFO - Using cached unicodecsv-0.14.1-py3-none-any.whl
[2021-04-19 15:27:16,568] {process_utils.py:141} INFO - Collecting setproctitle<2,>=1.1.8
[2021-04-19 15:27:16,570] {process_utils.py:141} INFO - Using cached setproctitle-1.2.2-cp37-cp37m-manylinux1_x86_64.whl (36 kB)
[2021-04-19 15:27:16,628] {process_utils.py:141} INFO - Collecting flask-login<0.5,>=0.3
[2021-04-19 15:27:16,628] {process_utils.py:141} INFO - Using cached Flask_Login-0.4.1-py2.py3-none-any.whl
[2021-04-19 15:27:16,682] {process_utils.py:141} INFO - Collecting apache-airflow-providers-imap
[2021-04-19 15:27:16,684] {process_utils.py:141} INFO - Using cached apache_airflow_providers_imap-1.0.1-py3-none-any.whl (15 kB)
[2021-04-19 15:27:16,768] {process_utils.py:141} INFO - Collecting colorlog>=4.0.2
[2021-04-19 15:27:16,770] {process_utils.py:141} INFO - Using cached colorlog-5.0.1-py2.py3-none-any.whl (10 kB)
[2021-04-19 15:27:16,874] {process_utils.py:141} INFO - Collecting pygments<3.0,>=2.0.1
[2021-04-19 15:27:16,878] {process_utils.py:141} INFO - Using cached Pygments-2.8.1-py3-none-any.whl (983 kB)
[2021-04-19 15:27:16,976] {process_utils.py:141} INFO - Collecting argcomplete~=1.10
[2021-04-19 15:27:16,978] {process_utils.py:141} INFO - Using cached argcomplete-1.12.2-py2.py3-none-any.whl (38 kB)
[2021-04-19 15:27:17,055] {process_utils.py:141} INFO - Collecting importlib-resources~=1.4
[2021-04-19 15:27:17,057] {process_utils.py:141} INFO - Using cached importlib_resources-1.5.0-py2.py3-none-any.whl (21 kB)
[2021-04-19 15:27:17,113] {process_utils.py:141} INFO - Collecting apache-airflow-providers-sqlite
[2021-04-19 15:27:17,114] {process_utils.py:141} INFO - Using cached apache_airflow_providers_sqlite-1.0.2-py3-none-any.whl (14 kB)
[2021-04-19 15:27:17,185] {process_utils.py:141} INFO - Collecting cattrs~=1.1
[2021-04-19 15:27:17,187] {process_utils.py:141} INFO - Using cached cattrs-1.5.0-py3-none-any.whl (19 kB)
[2021-04-19 15:27:17,274] {process_utils.py:141} INFO - Collecting pyjwt<2
[2021-04-19 15:27:17,275] {process_utils.py:141} INFO - Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
[2021-04-19 15:27:17,397] {process_utils.py:141} INFO - Collecting lazy-object-proxy
[2021-04-19 15:27:17,398] {process_utils.py:141} INFO - Using cached lazy_object_proxy-1.6.0-cp37-cp37m-manylinux1_x86_64.whl (55 kB)
[2021-04-19 15:27:17,457] {process_utils.py:141} INFO - Collecting flask-wtf<0.15,>=0.14.3
[2021-04-19 15:27:17,459] {process_utils.py:141} INFO - Using cached Flask_WTF-0.14.3-py2.py3-none-any.whl (13 kB)
[2021-04-19 15:27:17,603] {process_utils.py:141} INFO - Collecting flask-appbuilder~=3.1.1
[2021-04-19 15:27:17,632] {process_utils.py:141} INFO - Using cached Flask_AppBuilder-3.1.1-py3-none-any.whl (1.7 MB)
[2021-04-19 15:27:17,696] {process_utils.py:141} INFO - Collecting apache-airflow-providers-ftp
[2021-04-19 15:27:17,698] {process_utils.py:141} INFO - Using cached apache_airflow_providers_ftp-1.0.1-py3-none-any.whl (14 kB)
*some other apache-airflow dependencies installation logs*
[2021-04-19 15:27:23,603] {process_utils.py:141} INFO - Collecting certifi>=2017.4.17
[2021-04-19 15:27:23,607] {process_utils.py:141} INFO - Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
[2021-04-19 15:27:24,201] {process_utils.py:141} INFO - Collecting isodate
[2021-04-19 15:27:24,203] {process_utils.py:141} INFO - Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
[2021-04-19 15:27:26,537] {process_utils.py:141} INFO - Installing collected packages: zipp, six, pyrsistent, importlib-metadata, attrs, markupsafe, jsonschema, isodate, werkzeug, urllib3, PyYAML, pytz, openapi-schema-validator, jinja2, itsdangerous, idna, greenlet, defusedxml, click, chardet, certifi, WTForms, text-unidecode, sqlalchemy, requests, python3-openid, pyjwt, pycparser, openapi-spec-validator, marshmallow, inflection, flask, dnspython, clickclick, Babel, apispec, typing-extensions, swagger-ui-bundle, sqlalchemy-utils, pytzdata, python-slugify, python-editor, python-dateutil, pygments, prison, numpy, natsort, marshmallow-sqlalchemy, marshmallow-enum, Mako, lockfile, flask-wtf, Flask-SQLAlchemy, Flask-OpenID, flask-login, Flask-JWT-Extended, Flask-Babel, email-validator, docutils, connexion, commonmark, colorama, cffi, unicodecsv, termcolor, tenacity, tabulate, sqlalchemy-jsonfield, setproctitle, rich, python-nvd3, python-daemon, psutil, pendulum, pandas, marshmallow-oneofschema, markdown, lazy-object-proxy, iso8601, importlib-resources, gunicorn, graphviz, flask-caching, flask-appbuilder, dill, cryptography, croniter, colorlog, cattrs, cached-property, argcomplete, apache-airflow-providers-sqlite, apache-airflow-providers-imap, apache-airflow-providers-http, apache-airflow-providers-ftp, alembic, pyodbc, apache-airflow
[2021-04-19 15:27:35,766] {process_utils.py:141} INFO - Successfully installed Babel-2.9.0 Flask-Babel-1.0.0 Flask-JWT-Extended-3.25.1 Flask-OpenID-1.2.5 Flask-SQLAlchemy-2.5.1 Mako-1.1.4 PyYAML-5.4.1 WTForms-2.3.3 alembic-1.5.8 apache-airflow-2.0.1 apache-airflow-providers-ftp-1.0.1 apache-airflow-providers-http-1.1.1 apache-airflow-providers-imap-1.0.1 apache-airflow-providers-sqlite-1.0.2 apispec-3.3.2 argcomplete-1.12.2 attrs-20.3.0 cached-property-1.5.2 cattrs-1.5.0 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 click-7.1.2 clickclick-20.10.2 colorama-0.4.4 colorlog-5.0.1 commonmark-0.9.1 connexion-2.7.0 croniter-0.3.37 cryptography-3.4.7 defusedxml-0.7.1 dill-0.3.3 dnspython-2.1.0 docutils-0.17.1 email-validator-1.1.2 flask-1.1.2 flask-appbuilder-3.1.1 flask-caching-1.10.1 flask-login-0.4.1 flask-wtf-0.14.3 graphviz-0.16 greenlet-1.0.0 gunicorn-19.10.0 idna-2.10 importlib-metadata-1.7.0 importlib-resources-1.5.0 inflection-0.5.1 iso8601-0.1.14 isodate-0.6.0 itsdangerous-1.1.0 jinja2-2.11.3 jsonschema-3.2.0 lazy-object-proxy-1.6.0 lockfile-0.12.2 markdown-3.3.4 markupsafe-1.1.1 marshmallow-3.11.1 marshmallow-enum-1.5.1 marshmallow-oneofschema-2.1.0 marshmallow-sqlalchemy-0.23.1 natsort-7.1.1 numpy-1.20.2 openapi-schema-validator-0.1.5 openapi-spec-validator-0.3.0 pandas-1.2.4 pendulum-2.1.2 prison-0.1.3 psutil-5.8.0 pycparser-2.20 pygments-2.8.1 pyjwt-1.7.1 pyodbc-4.0.30 pyrsistent-0.17.3 python-daemon-2.3.0 python-dateutil-2.8.1 python-editor-1.0.4 python-nvd3-0.15.0 python-slugify-4.0.1 python3-openid-3.2.0 pytz-2021.1 pytzdata-2020.1 requests-2.25.1 rich-9.2.0 setproctitle-1.2.2 six-1.15.0 sqlalchemy-1.4.9 sqlalchemy-jsonfield-1.0.0 sqlalchemy-utils-0.37.0 swagger-ui-bundle-0.0.8 tabulate-0.8.9 tenacity-6.2.0 termcolor-1.1.0 text-unidecode-1.3 typing-extensions-3.7.4.3 unicodecsv-0.14.1 urllib3-1.26.4 werkzeug-1.0.1 zipp-3.4.1
[2021-04-19 15:27:36,306] {process_utils.py:135} INFO - Executing cmd: /tmp/venvxuo1gayl/bin/python /tmp/venvxuo1gayl/script.py /tmp/venvxuo1gayl/script.in /tmp/venvxuo1gayl/script.out /tmp/venvxuo1gayl/string_args.txt
[2021-04-19 15:27:36,312] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:36,875] {process_utils.py:141} INFO - [[34m2021-04-19 15:27:36,875[0m] {[34mlogging_config.py:[0m59} ERROR[0m - Unable to load the config, contains a configuration error.[0m
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 93, in _resolve
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - found = getattr(found, n)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - AttributeError: module 'airflow' has no attribute 'utils'
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - During handling of the above exception, another exception occurred:
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 542, in configure
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - formatters[name])
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 672, in configure_formatter
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - c = _resolve(cname)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 96, in _resolve
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - found = getattr(found, n)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - AttributeError: module 'airflow' has no attribute 'utils'
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - The above exception was the direct cause of the following exception:
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/script.py", line 132, in <module>
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - res = weather_update(*arg_dict["args"], **arg_dict["kwargs"])
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/script.py", line 40, in weather_update
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - from airflow.models import Variable
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/__init__.py", line 46, in <module>
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - settings.initialize()
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/settings.py", line 434, in initialize
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - LOGGING_CLASS_PATH = configure_logging()
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/logging_config.py", line 62, in configure_logging
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - raise e
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/logging_config.py", line 57, in configure_logging
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - dictConfig(logging_config)
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 799, in dictConfig
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - dictConfigClass(config).configure()
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 545, in configure
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - 'formatter %r' % name) from e
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - ValueError: Unable to configure formatter 'airflow_coloured'
[2021-04-19 15:27:37,210] {taskinstance.py:1455} ERROR - Command '['/tmp/venvxuo1gayl/bin/python', '/tmp/venvxuo1gayl/script.py', '/tmp/venvxuo1gayl/script.in', '/tmp/venvxuo1gayl/script.out', '/tmp/venvxuo1gayl/string_args.txt']' returned non-zero exit status 1.
Traceback (most recent call last):
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
result = task_copy.execute(context=context)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py", line 493, in execute
super().execute(context=serializable_context)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py", line 117, in execute
return_value = self.execute_callable()
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py", line 531, in execute_callable
string_args_filename,
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/utils/process_utils.py", line 145, in execute_in_subprocess
raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/tmp/venvxuo1gayl/bin/python', '/tmp/venvxuo1gayl/script.py', '/tmp/venvxuo1gayl/script.in', '/tmp/venvxuo1gayl/script.out', '/tmp/venvxuo1gayl/string_args.txt']' returned non-zero exit status 1.
[2021-04-19 15:27:37,212] {taskinstance.py:1503} INFO - Marking task as FAILED. dag_id=weather_update, task_id=updating_forecasts, execution_date=20210419T122709, start_date=20210419T122710, end_date=20210419T122737
[2021-04-19 15:27:37,227] {configuration.py:356} WARNING - section/key [smtp/smtp_user] not found in config
It seems that you are confusing the use-cases for PythonVirtualenvOperator and PythonOperator.
If you simply want to run a Python callable in a task (callable_virtualenv() in your case) you can use PythonOperator. In this case, it does not matter if you installed Airflow in a virtual environment, system wide, or using Docker.
What happens in your code is the following: PythonVirtualenvOperator creates another virtual environment (which is completely unrelated to the one in which you run Airflow), installs Airflow into it, and tries to import Variable. But this another Airflow installation is not configured and that is why you get those exceptions. You could set the AIRFLOW_HOME environment variable for this second Airflow installation to the same directory as used by the first Airflow installation, and this should actually work, but it looks like an overkill to me.
So, what you can do is install colorama into the same environment in which you installed Airflow and replace PythonVirtualenvOperator by PythonOperator.
BTW, those print() inside the callable would be redirected into a log file and not printed to terminal, so it probably does not make much sense to use colorama with them.