You can run Python scripts from the TD Workflow using the Python operator (py>:). Create the workflow definition using TD Console or using TD Workflow from the command line.
In the workflow definition, specify a Docker image to use for running the Python script. When the workflow task starts, a new Docker container is created based on the specified docker image. Then, the Python script is executed in the container in an isolated environment.
Make sure this feature is enabled for your TD account.
- Basic Knowledge of Treasure Workflow's syntax.
If you intend to use the CLI, you need to do the following:
You might want to view examples, for basics such as:
How to call functions
How to pass parameters to functions
How to use environment variables
How to import functions
Add your Python Script to Treasure Workflow
If you have more than a few scripts to add, using the command line method is recommended.
Using TD Console
Navigate to Data Workbench > Workflows.
Select the workflow to which you would like to add the Python scripts.
Select Launch Project Editor.
Select Edit Files.
Select Add New File.
- Type in your dig filename.
py>operator and specify a Docker image that you want to use. Your script might look like this sample:
You can add each script or copy-paste the text of each script into the new script editor window.
Select Save & Commit.
Using td CLI
You can add a python script to your existing workflow using the command line. New users may need to first create a workflow using the command line.
Add a workflow definition .dig file and Python script to the workflow directory.
Specify a Docker image that you want to use for the py>: operator in the .dig file.
Add syntax similar to the following to your workflow dig file to add the py> operator and specify the Docker image. Your script might look like the following sample:
- Push the workflow to Treasure Data using td CLI command `td wf push <project_name>`
The Python scripts in TD Workflows are managed and run by Treasure Data in isolated Docker containers. Treasure Data provides a number of base Docker images to run in the container. You can pick the appropriate Docker image to run your Python script in, based on the Python version and libraries supported by the image.
Learn more about Docker image versioning for Custom Scripts.
View the below sample using the Python 3.9 Docker image.
Install your own Python Libraries
In addition to the libraries provided by the Docker image, you can install additional 3rd-party libraries using the pip install command within the Python script.
Using Docker Images on Your Local Laptop
Docker images are also published in Dockerhub and publicly available for use on your own laptop for evaluation or testing purposes.
Prerequisite: Docker runtime installed.
You can confirm the python version as follows on your laptop:
To run an interactive session, you can run as follows:
Python interactive shell is launched when running digdag/digdag-python:3.9 without arguments:
You can get a complete list of library versions using pip freeze: