great_expectations.datasource.data_connector.inferred_asset_s3_data_connector

Module Contents

Classes

InferredAssetS3DataConnector(name: str, datasource_name: str, bucket: str, execution_engine: Optional[ExecutionEngine] = None, default_regex: Optional[dict] = None, sorters: Optional[list] = None, prefix: str = ‘’, delimiter: str = ‘/’, max_keys: int = 1000, boto3_options: Optional[dict] = None, batch_spec_passthrough: Optional[dict] = None)

Extension of InferredAssetFilePathDataConnector used to connect to S3

Functions

_check_valid_s3_path(path: str)

Performs a basic check for validity of the S3 path

great_expectations.datasource.data_connector.inferred_asset_s3_data_connector.boto3
great_expectations.datasource.data_connector.inferred_asset_s3_data_connector.logger
great_expectations.datasource.data_connector.inferred_asset_s3_data_connector.INVALID_S3_CHARS = ['*']
class great_expectations.datasource.data_connector.inferred_asset_s3_data_connector.InferredAssetS3DataConnector(name: str, datasource_name: str, bucket: str, execution_engine: Optional[ExecutionEngine] = None, default_regex: Optional[dict] = None, sorters: Optional[list] = None, prefix: str = '', delimiter: str = '/', max_keys: int = 1000, boto3_options: Optional[dict] = None, batch_spec_passthrough: Optional[dict] = None)

Bases: great_expectations.datasource.data_connector.inferred_asset_file_path_data_connector.InferredAssetFilePathDataConnector

Extension of InferredAssetFilePathDataConnector used to connect to S3

The InferredAssetS3DataConnector is one of two classes (ConfiguredAssetS3DataConnector being the other one) designed for connecting to filesystem-like data, more specifically files on S3. It connects to assets inferred from bucket, prefix, and file name by default_regex.

InferredAssetS3DataConnector that operates on S3 buckets and determines the data_asset_name implicitly (e.g., through the combination of the regular expressions pattern and group names)

build_batch_spec(self, batch_definition: BatchDefinition)

Build BatchSpec from batch_definition by calling DataConnector’s build_batch_spec function.

Parameters

batch_definition (BatchDefinition) – to be used to build batch_spec

Returns

BatchSpec built from batch_definition

_get_data_reference_list(self, data_asset_name: Optional[str] = None)

List objects in the underlying data store to create a list of data_references.

This method is used to refresh the cache.

_get_full_file_path(self, path: str, data_asset_name: Optional[str] = None)
great_expectations.datasource.data_connector.inferred_asset_s3_data_connector._check_valid_s3_path(path: str) → None

Performs a basic check for validity of the S3 path