great_expectations.datasource.data_connector.configured_asset_sql_data_connector
¶
Module Contents¶
Classes¶
|
A DataConnector that requires explicit listing of SQL tables you want to connect to. |
-
great_expectations.datasource.data_connector.configured_asset_sql_data_connector.
sa
¶
-
class
great_expectations.datasource.data_connector.configured_asset_sql_data_connector.
ConfiguredAssetSqlDataConnector
(name: str, datasource_name: str, execution_engine: Optional[ExecutionEngine] = None, assets: Optional[Dict[str, dict]] = None, batch_spec_passthrough: Optional[dict] = None)¶ Bases:
great_expectations.datasource.data_connector.data_connector.DataConnector
A DataConnector that requires explicit listing of SQL tables you want to connect to.
- Parameters
name (str) – The name of this DataConnector
datasource_name (str) – The name of the Datasource that contains it
execution_engine (ExecutionEngine) – An ExecutionEngine
assets (str) – assets
batch_spec_passthrough (dict) – dictionary with keys that will be added directly to batch_spec
-
property
assets
(self)¶
-
add_data_asset
(self, name: str, config: dict)¶ Add data_asset to DataConnector using data_asset name as key, and data_asset config as value.
-
_update_data_asset_name_from_config
(self, data_asset_name: str, data_asset_config: dict)¶
-
_get_batch_identifiers_list_from_data_asset_config
(self, data_asset_name, data_asset_config)¶
-
_refresh_data_references_cache
(self)¶
-
_get_column_names_from_splitter_kwargs
(self, splitter_kwargs)¶
-
get_available_data_asset_names
(self)¶ Return the list of asset names known by this DataConnector.
- Returns
A list of available names
-
get_unmatched_data_references
(self)¶ Returns the list of data_references unmatched by configuration by looping through items in _data_references_cache and returning data_reference that do not have an associated data_asset.
- Returns
list of data_references that are not matched by configuration.
-
get_batch_definition_list_from_batch_request
(self, batch_request: BatchRequest)¶
-
_get_data_reference_list_from_cache_by_data_asset_name
(self, data_asset_name: str)¶ Fetch data_references corresponding to data_asset_name from the cache.
-
_map_data_reference_to_batch_definition_list
(self, data_reference, data_asset_name: Optional[str] = None)¶
-
build_batch_spec
(self, batch_definition: BatchDefinition)¶ Build BatchSpec from batch_definition by calling DataConnector’s build_batch_spec function.
- Parameters
batch_definition (BatchDefinition) – to be used to build batch_spec
- Returns
BatchSpec built from batch_definition
-
_generate_batch_spec_parameters_from_batch_definition
(self, batch_definition: BatchDefinition)¶ - Build BatchSpec parameters from batch_definition with the following components:
data_asset_name from batch_definition
batch_identifiers from batch_definition
data_asset from data_connector
- Parameters
batch_definition (BatchDefinition) – to be used to build batch_spec
- Returns
dict built from batch_definition
-
_get_table_name_from_batch_definition
(self, batch_definition: BatchDefinition)¶ - Helper method called by _get_batch_identifiers_list_from_data_asset_config() to parse table_name from data_asset_name in cases
where schema is included.
data_asset_name in those cases are [schema].[table_name].
function will split data_asset_name on [schema]. and return the resulting table_name.
-
_split_on_whole_table
(self, table_name: str)¶ ‘Split’ by returning the whole table
Note: the table_name parameter is a required to keep the signature of this method consistent with other methods.
-
_split_on_column_value
(self, table_name: str, column_name: str)¶ Split using the values in the named column
-
_split_on_converted_datetime
(self, table_name: str, column_name: str, date_format_string: str = '%Y-%m-%d')¶ Convert the values in the named column to the given date_format, and split on that
-
_split_on_divided_integer
(self, table_name: str, column_name: str, divisor: int)¶ Divide the values in the named column by divisor, and split on that
-
_split_on_mod_integer
(self, table_name: str, column_name: str, mod: int)¶ Divide the values in the named column by divisor, and split on that
-
_split_on_multi_column_values
(self, table_name: str, column_names: List[str])¶ Split on the joint values in the named columns
-
_split_on_hashed_column
(self, table_name: str, column_name: str, hash_digits: int)¶ Note: this method is experimental. It does not work with all SQL dialects.