great_expectations.datasource.data_connector.configured_asset_sql_data_connector

Module Contents

Classes

ConfiguredAssetSqlDataConnector(name: str, datasource_name: str, execution_engine: Optional[ExecutionEngine] = None, assets: Optional[Dict[str, dict]] = None, batch_spec_passthrough: Optional[dict] = None)

A DataConnector that requires explicit listing of SQL tables you want to connect to.
great_expectations.datasource.data_connector.configured_asset_sql_data_connector.sa
class great_expectations.datasource.data_connector.configured_asset_sql_data_connector.ConfiguredAssetSqlDataConnector(name: str, datasource_name: str, execution_engine: Optional[ExecutionEngine] = None, assets: Optional[Dict[str, dict]] = None, batch_spec_passthrough: Optional[dict] = None)

Bases: great_expectations.datasource.data_connector.data_connector.DataConnector

A DataConnector that requires explicit listing of SQL tables you want to connect to.

Parameters

  • name (str) – The name of this DataConnector

  • datasource_name (str) – The name of the Datasource that contains it

  • execution_engine (ExecutionEngine) – An ExecutionEngine

  • assets (str) – assets

  • batch_spec_passthrough (dict) – dictionary with keys that will be added directly to batch_spec

property assets(self)
add_data_asset(self, name: str, config: dict)

Add data_asset to DataConnector using data_asset name as key, and data_asset config as value.

_update_data_asset_name_from_config(self, data_asset_name: str, data_asset_config: dict)
_get_batch_identifiers_list_from_data_asset_config(self, data_asset_name, data_asset_config)
_refresh_data_references_cache(self)
_get_column_names_from_splitter_kwargs(self, splitter_kwargs)
get_available_data_asset_names(self)

Return the list of asset names known by this DataConnector.

Returns

A list of available names

get_unmatched_data_references(self)

Returns the list of data_references unmatched by configuration by looping through items in _data_references_cache and returning data_reference that do not have an associated data_asset.

Returns

list of data_references that are not matched by configuration.

get_batch_definition_list_from_batch_request(self, batch_request: BatchRequest)
_get_data_reference_list_from_cache_by_data_asset_name(self, data_asset_name: str)

Fetch data_references corresponding to data_asset_name from the cache.

_map_data_reference_to_batch_definition_list(self, data_reference, data_asset_name: Optional[str] = None)
build_batch_spec(self, batch_definition: BatchDefinition)

Build BatchSpec from batch_definition by calling DataConnector’s build_batch_spec function.

Parameters

batch_definition (BatchDefinition) – to be used to build batch_spec

Returns

BatchSpec built from batch_definition

_generate_batch_spec_parameters_from_batch_definition(self, batch_definition: BatchDefinition)
Build BatchSpec parameters from batch_definition with the following components:

  1. data_asset_name from batch_definition

  2. batch_identifiers from batch_definition

  3. data_asset from data_connector

Parameters

batch_definition (BatchDefinition) – to be used to build batch_spec

Returns

dict built from batch_definition

_get_table_name_from_batch_definition(self, batch_definition: BatchDefinition)
Helper method called by _get_batch_identifiers_list_from_data_asset_config() to parse table_name from data_asset_name in cases

where schema is included.

data_asset_name in those cases are [schema].[table_name].

function will split data_asset_name on [schema]. and return the resulting table_name.

_split_on_whole_table(self, table_name: str)

‘Split’ by returning the whole table

Note: the table_name parameter is a required to keep the signature of this method consistent with other methods.

_split_on_column_value(self, table_name: str, column_name: str)

Split using the values in the named column

_split_on_converted_datetime(self, table_name: str, column_name: str, date_format_string: str = '%Y-%m-%d')

Convert the values in the named column to the given date_format, and split on that

_split_on_divided_integer(self, table_name: str, column_name: str, divisor: int)

Divide the values in the named column by divisor, and split on that

_split_on_mod_integer(self, table_name: str, column_name: str, mod: int)

Divide the values in the named column by divisor, and split on that

_split_on_multi_column_values(self, table_name: str, column_names: List[str])

Split on the joint values in the named columns

_split_on_hashed_column(self, table_name: str, column_name: str, hash_digits: int)

Note: this method is experimental. It does not work with all SQL dialects.