great_expectations create datasource of csv files on ADLS Gen2

Question

I want to run great_expectation test suites against csv files in my ADLS Gen2. On my ADLS, I have a container called "data" in which I have a file at mypath/test/mydata.csv. I use a InferredAssetAzureDataConnector. I was able to create and test/validate the data source configuration but I believe there is a "silent" issue which was not caught.

The problem is that I cannot create a test suite based on this data source. When I run great_expectations suite new,

I select (3) to create the suite with the profiler, and then
select my newly created datasource, and then
instead of showing me the available files at the data source, it crashes with the following error (see below for full stacktrace):

TypeError: __init__() missing 1 required positional argument: 'data_asset_name'

When I execute this with a local data source (InferredAssetFilesystemDataConnector), it would show me the available files at the data source for selection in the CLI.

Does the error mean it cannot find the csv file on the ADLS and thus has no names to show? How do I fix this?

My Code to create the data source:

import great_expectations as ge
from great_expectations.cli.datasource import sanitize_yaml_and_save_datasource, check_if_datasource_name_exists
context = ge.get_context()
datasource_name = "my_datasource_name"


example_yaml = f"""
name: {datasource_name}
class_name: Datasource
execution_engine:
  class_name: SparkDFExecutionEngine
  azure_options:
      account_url: https://<ACCOUNT-NAME>.blob.core.windows.net
      credential: <ACCOUNT-KEY>
data_connectors:
  default_inferred_data_connector_name:
    class_name: InferredAssetAzureDataConnector
    azure_options:
        account_url: https://<ACCOUNT-NAME>.blob.core.windows.net
        credential: <ACCOUNT-KEY>
    container: data
    name_starts_with: mypath/test
    default_regex:
      group_names:
        - data_asset_name
      pattern: (.csv)
  default_runtime_data_connector_name:
    class_name: RuntimeDataConnector
    assets:
      my_runtime_asset_name:
        batch_identifiers:
          - runtime_batch_identifier_name
"""
print(example_yaml)
# Test the yml:
context.test_yaml_config(yaml_config=example_yaml)

The output after creating the data source via the Jupyter Notebook:

Attempting to instantiate class from config...
    Instantiating as a Datasource, since class_name is Datasource
    Successfully instantiated Datasource


ExecutionEngine class name: SparkDFExecutionEngine
Data Connectors:
    default_inferred_data_connector_name : InferredAssetAzureDataConnector
    Available data_asset_names (0 of 0):
    Unmatched data_references (0 of 0):[]
    default_runtime_data_connector_name:RuntimeDataConnector
    default_runtime_data_connector_name : RuntimeDataConnector
    Available data_asset_names (1 of 1):
        my_runtime_asset_name (0 of 0): []
    Unmatched data_references (0 of 0):[]
<great_expectations.datasource.new_datasource.Datasource at 0x1cdc9e01f70>

Full error stack:

Traceback (most recent call last):
  File "c:codingpython38librunpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:codingpython38librunpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:codingmyrepovenvScriptsgreat_expectations.exe__main__.py", line 7, in <module>
  File "C:codingmyrepovenvlibsite-packagesgreat_expectationsclicli.py", line 190, in main
    cli()
  File "C:codingmyrepovenvlibsite-packagesclickcore.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "C:codingmyrepovenvlibsite-packagesclickcore.py", line 1053, in main
    rv = self.invoke(ctx)
  File "C:codingmyrepovenvlibsite-packagesclickcore.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:codingmyrepovenvlibsite-packagesclickcore.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:codingmyrepovenvlibsite-packagesclickcore.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:codingmyrepovenvlibsite-packagesclickcore.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:codingmyrepovenvlibsite-packagesclickdecorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:codingmyrepovenvlibsite-packagesgreat_expectationsclisuite.py", line 151, in suite_new
    _suite_new_workflow(
  File "C:codingmyrepovenvlibsite-packagesgreat_expectationsclisuite.py", line 335, in _suite_new_workflow
    raise e
  File "C:codingmyrepovenvlibsite-packagesgreat_expectationsclisuite.py", line 279, in _suite_new_workflow
    toolkit.add_citation_with_batch_request(
  File "C:codingmyrepovenvlibsite-packagesgreat_expectationsclitoolkit.py", line 1020, in add_citation_with_batch_request
    and BatchRequest(**batch_request)
TypeError: __init__() missing 1 required positional argument: 'data_asset_name'

Asked By: Cribber

||

Source

Answer 1

I had a mistake in my regex … with the following pattern it works flawlessly:

    default_regex:
      group_names:
        - data_asset_name
      pattern: (.*.csv)

Answered By: Cribber

great_expectations create datasource of csv files on ADLS Gen2

Question:

Answers: