Deploy the GX Agent
For access to the GX Agent, reach out to Support at support@greatexpectations.io.
GX Cloud is a fully hosted data quality solution and you do not need to deploy the GX Agent to make use of its full suite of features. However, you can optionally deploy the GX Agent to exert additional control over how users access GX Cloud and how data is accessed by GX Cloud processes.
The GX Agent is an intermediary between GX Cloud and your organization's data stores. GX Cloud does not connect directly to your data with this deployment option, and all data access occurs within the GX Agent. GX Cloud sends jobs to the GX Agent, the GX Agent executes these jobs against your data, and then sends the job results to GX Cloud.
A local deployment of the GX Agent will allow you to test GX Cloud setup or processes from a single machine before moving to a shared production deployment. Alternatively, you can deploy the GX Agent to your development environment and leverage GX Cloud while connecting to Data Sources using your organization's cloud infrastructure, for enhanced control and security.
GX Agent versioning
GX uses a date-based versioning format for its weekly GX Agent Docker image releases: YYYYMMMDD.#
for stable releases and YYYYMMDD.#.dev#
for pre-releases. GX uses the stable
and dev
Docker image tags to identify the release type. The stable
tag indicates the image is fully tested and ready for use. The dev
tag indicates a pre-release image. This documentation assumes you're using the latest stable
GX Agent Docker image.
Prerequisites
- You have a GX Cloud account.
- You have reached out to Support at support@greatexpectations.io to request a GX Agent deployment.
Get your access token and organization ID
You need your access token and organization ID to deploy the GX Agent. Access tokens shouldn't be committed to version control software.
If you've used GX Cloud previously, you have your access token and organization ID, and you need to restart the GX Agent, see Deploy the GX Agent.
- New GX Cloud account
- Existing GX Cloud account
-
Sign in to GX Cloud.
-
Complete the survey and then click Continue to GX Cloud.
-
Copy and then paste the Access token and Organization ID values into a temporary file. You'll need them to deploy the GX Agent.
-
Click Deploy the GX Agent and deploy the GX Agent.
Use the information provided here to view your organization ID or create a new access token. This can be helpful if you've forgotten your organization ID or access token, and you need to restart the GX Agent.
-
In GX Cloud, click Settings > Tokens.
-
In the User access tokens pane, click Create user access token.
-
In the Token name field, enter a name for the token that will help you quickly identify it.
-
Click Create.
-
Copy and then paste the user access token into a temporary file. The token can't be retrieved after you close the dialog.
-
Click Close.
-
Copy the value in the Organization ID field into the temporary file with your user access token and then save the file.
GX recommends deleting the temporary file after you set the environment variables.
Deploy the GX Agent
The GX Agent allows you to securely access your data without connecting to it or interacting with it directly. To learn more about the GX Agent and deployment patterns, see About GX Cloud.
Prerequisites
-
You have a GX Cloud access token and organization ID. See Get your access token and organization ID.
-
You have a Docker instance or kubectl.
- Org-hosted
- Self-hosted
You can deploy the GX Agent container in any deployment environment where you can run Docker container images.
To learn how to deploy a Docker container image in a specific environment, see the following documentation:
You can deploy the GX Agent in any environment in which you create Kubernetes clusters. For example:
-
Any Kubernetes cluster version 1.21 or greater which uses standard Kubernetes
- Docker
- Kubernetes
-
Download the GX Agent Docker container image from Docker Hub.
-
After configuring your cloud service to run Docker containers, run the following Docker command to start the GX Agent:
Terminal inputdocker run -it \
-e GX_CLOUD_ACCESS_TOKEN= YOUR_ACCESS_TOKEN \
-e GX_CLOUD_ORGANIZATION_ID= YOUR_ORGANIZATION_ID \
greatexpectations/agent:stableReplace
YOUR_ACCESS_TOKEN
andYOUR_ORGANIZATION_ID
with the values you copied previously. -
Optional. If you created a temporary file to record your user access token and Organization ID, delete it.
-
Optional. Run the following command to use the GX Agent image as the base image and optionally add custom commands:
Terminal inputFROM greatexpectations/agent
RUN echo "custom_commands" -
Optional. Run the following command to rebuild the Docker image:
Terminal inputdocker build -t myorg/agent
-
Optional. Run
docker ps
or open Docker Desktop to confirm the agent is running.
-
Install kubectl. See Install Tools.
-
Run the following command to provide the access credentials to the Kubernetes container:
kubectl create secret generic gx-agent-secret \
--from-literal=GX_CLOUD_ORGANIZATION_ID=YOUR_ORGANIZATION_ID \
--from-literal=GX_CLOUD_ACCESS_TOKEN=YOUR_ACCESS_TOKEN \Replace
YOUR_ORGANIZATION_ID
andYOUR_ACCESS_TOKEN
with the values you copied previously. -
Optional. If you created a temporary file to record your user access token and Organization ID, delete it.
-
Create and save a file named
deployment.yaml
, with the following configuration:apiVersion: apps/v1
kind: Deployment
metadata:
name: gx-agent
labels:
app: gx-agent
spec:
replicas: 1
selector:
matchLabels:
app: gx-agent
template:
metadata:
labels:
app: gx-agent
spec:
containers:
name: gx-agent
image: greatexpectations/agent:stable
imagePullPolicy: Always
envFrom:
secretRef:
name: gx-agent-secret -
Run the following command to use the
deployment.yaml
configuration file to deploy the GX Agent:kubectl apply -f deployment.yaml
-
Optional. Run the following command to confirm the agent is running:
kubectl logs -l app=gx-agent
-
Optional. Run the following command to terminate running resources gracefully:
kubectl delete -f deployment.yaml
kubectl delete secret gx-agent-secret
-
Start the Docker Engine.
-
Run the following code to set the
GX_CLOUD_ACCESS_TOKEN
andGX_CLOUD_ORGANIZATION_ID
environment variables, install GX Cloud and its dependencies, and start the GX Agent:Terminal inputdocker run --rm --pull=always -e GX_CLOUD_ACCESS_TOKEN="<user_access_token>" -e GX_CLOUD_ORGANIZATION_ID="<organization_id>" greatexpectations/agent
Replace
user_access_token
andorganization_id
with the values you copied previously. -
In GX Cloud, confirm the GX Agent status is Active Agent and the icon is green. This indicates the GX Agent is active. If it isn't, repeat step 2 and confirm the
user_access_token
andorganization_id
values are correct. -
Optional. If you created a temporary file to record your user access token and Organization ID, delete it.
-
Optional. Run
docker ps
or open Docker Desktop to confirm the agent is running.If you stop the GX Agent, close the terminal, and open a new terminal you'll need to set the environment variables again.
To edit an environment variable, stop the GX Agent, edit the environment variable, save the change, and then restart the GX Agent.
View GX Cloud logs
If you encounter an issue deploying the GX Agent or performing a GX Cloud task, review log information to troubleshoot the cause and determine a fix.
-
In GX Cloud, click Logs.
-
Click Show log next to a log entry to display additional log details.
-
Optional. Click Hide log to close the log details view.
Next steps
Want to continue testing GX Cloud features and functionality?
Ready to connect GX Cloud to your deployment environment?