Skip to main content
Version: 0.18.21

DatabricksSQLDatasource

class great_expectations.datasource.fluent.DatabricksSQLDatasource(*, type: Literal['databricks_sql'] = 'databricks_sql', name: str, id: Optional[uuid.UUID] = None, assets: List[Union[great_expectations.datasource.fluent.sql_datasource.TableAsset, great_expectations.datasource.fluent.sql_datasource.QueryAsset]] = [], connection_string: Union[great_expectations.datasource.fluent.config_str.ConfigStr, great_expectations.datasource.fluent.databricks_sql_datasource.DatabricksDsn], create_temp_table: bool = False, kwargs: Dict[str, Union[great_expectations.datasource.fluent.config_str.ConfigStr, Any]] = )#

Adds a DatabricksSQLDatasource to the data context.

Parameters

  • name – The name of this DatabricksSQL datasource.

  • connection_string – The SQLAlchemy connection string used to connect to the postgres database. For example: “databricks://token:<token>@<host>:<port>/<database>?http_path=<http_path>&catalog=<catalog>&schema=<schema>””

  • assets – An optional dictionary whose keys are TableAsset or QueryAsset names and whose values are TableAsset or QueryAsset objects.

add_query_asset(name: str, query: str, order_by: Optional[SortersDefinition] = None, batch_metadata: Optional[BatchMetadata] = None) → QueryAsset#

Adds a query asset to this datasource.

Parameters

  • name – The name of this table asset.

  • query – The SELECT query to selects the data to validate. It must begin with the “SELECT”.

  • order_by – A list of Sorters or Sorter strings.

  • batch_metadata – BatchMetadata we want to associate with this DataAsset and all batches derived from it.

Returns

The query asset that is added to the datasource. The type of this object will match the necessary type for this datasource. eg, it could be a QueryAsset or a SqliteQueryAsset.

add_table_asset(name: str, table_name: str = '', schema_name: Optional[str] = None, order_by: Optional[SortersDefinition] = None, batch_metadata: Optional[BatchMetadata] = None) → TableAsset#

Adds a table asset to this datasource.

Parameters

  • name – The name of this table asset.

  • table_name – The table where the data resides.

  • schema_name – The schema that holds the table.

  • order_by – A list of Sorters or Sorter strings.

  • batch_metadata – BatchMetadata we want to associate with this DataAsset and all batches derived from it.

Returns

The table asset that is added to the datasource. The type of this object will match the necessary type for this datasource. eg, it could be a TableAsset or a SqliteTableAsset.