great_expectations.datasource.data_connector.configured_asset_s3_data_connector

Module Contents

Classes

ConfiguredAssetS3DataConnector(name: str, datasource_name: str, bucket: str, assets: dict, execution_engine: Optional[ExecutionEngine] = None, default_regex: Optional[dict] = None, sorters: Optional[list] = None, prefix: str = ‘’, delimiter: str = ‘/’, max_keys: int = 1000, boto3_options: Optional[dict] = None, batch_spec_passthrough: Optional[dict] = None)

Extension of ConfiguredAssetFilePathDataConnector used to connect to S3
great_expectations.datasource.data_connector.configured_asset_s3_data_connector.boto3
great_expectations.datasource.data_connector.configured_asset_s3_data_connector.logger
class great_expectations.datasource.data_connector.configured_asset_s3_data_connector.ConfiguredAssetS3DataConnector(name: str, datasource_name: str, bucket: str, assets: dict, execution_engine: Optional[ExecutionEngine] = None, default_regex: Optional[dict] = None, sorters: Optional[list] = None, prefix: str = '', delimiter: str = '/', max_keys: int = 1000, boto3_options: Optional[dict] = None, batch_spec_passthrough: Optional[dict] = None)

Bases: great_expectations.datasource.data_connector.configured_asset_file_path_data_connector.ConfiguredAssetFilePathDataConnector

Extension of ConfiguredAssetFilePathDataConnector used to connect to S3

DataConnectors produce identifying information, called “batch_spec” that ExecutionEngines can use to get individual batches of data. They add flexibility in how to obtain data such as with time-based partitioning, downsampling, or other techniques appropriate for the Datasource.

The ConfiguredAssetS3DataConnector is one of two classes (InferredAssetS3DataConnector being the other one) designed for connecting to data on S3.

A ConfiguredAssetS3DataConnector requires an explicit listing of each DataAsset you want to connect to. This allows more fine-tuning, but also requires more setup.

static sanitize_prefix_for_s3(text: str)

Takes in a given user-prefix and cleans it to work with file-system traversal methods (i.e. add ‘/’ to the end of a string meant to represent a directory)

Customized for S3 paths, ignoring the path separator used by the host OS

build_batch_spec(self, batch_definition: BatchDefinition)

Build BatchSpec from batch_definition by calling DataConnector’s build_batch_spec function.

Parameters

batch_definition (BatchDefinition) – to be used to build batch_spec

Returns

BatchSpec built from batch_definition

_get_data_reference_list_for_asset(self, asset: Optional[Asset])
_get_full_file_path_for_asset(self, path: str, asset: Optional[Asset] = None)