This document describes the new rootless docker images and provides instructions for migrating Docker images used in existing TD Workflows.

We strongly recommend migrating to the new secure, rootless Docker image because these images reduce potential security risks, provide a better user experience through new semantic versioning scheme, and provide the latest stable versions of packages.

Learn more about:


Rootless Docker Images


The following rootless docker images are based on the latest stable Python’s official docker image, “3.9-slim-buster”, which is composed of the latest stable Debian 10 buster (LTS) and the latest stable Python version 3.9.

  • digdag/digdag-python:3.9 (major version)
  • digdag/digdag-python:3.9.1 (minor version)

Learn more about versioning in Understanding Versioning. 

We generally recommend using the major version and only use the minor version for a temporary workaround when you find API compatibility issues. (then, please start migration). Contact Support if you need assistance with the migration process.

Migration to the Rootless Docker Image 


Minimum changes are required. Changes the image version to use the new rootless docker image in the treasure workflow as follows:

+task:
  +rootless3_9:
    py>: tasks.scipy_numpy.main
    docker:
      image: "digdag/digdag-python:3.9"

  +rootless3_9_1:
    py>: tasks.scipy_numpy.main
    docker:
      image: "digdag/digdag-python:3.9.1"


Existing pip install code does not require changes because pip install falls back to user space install (--user option). 

import os
import sys

# falls back to --user space install
os.system(f"{sys.executable} -m pip install asn1")

# same as the above
# os.system(f"{sys.executable} -m pip install --user asn1")


  • “pip install” automatically falls back to “pip install --user” installing libraries to user space. Unless your pip installation requires root user privileges, no changes are required. Please confirm python library APIs etc. when using new rootless Docker image.
  • “apt-get install” is prohibited in the rootless Docker image. So, you cannot run os.system("apt-get install xxx") in python.
  • Some libraries may require root privileges for pip install. It’s a known limitation in the rootless Docker image considering security risks. Contact Support with questions about this issue.
  • If you are using pre-installed python libraries (such as scikit, numpy, or lightgbm), please check that it still works. APIs may be changed among versions.

Changes With the New Docker Images


Here is the list of pre-installed libraries (with version information) in the new rootless Docker image. “+” indicates libraries introduced from digdag-python:3.9.1.

+hyperopt==0.2.5
+optuna==2.7.0
+memory_profiler==0.58.0
+pytest==6.2.3
+simplejson==3.17.2
+futures==3.1.1
+nltk==3.5
pytd[spark]==1.4.0
td-client==1.2.1
td-pyspark==21.3.0
pyspark==3.1.1
numpy==1.20.2
scipy==1.6.2
pandas==1.2.3
matplotlib==3.4.1
boto3==1.15.18
plotly==4.14.3
tqdm==4.60.0
seaborn==0.11.1
requests==2.25.1
scikit-learn==0.24.1
lightgbm==3.2.0
pystan==2.19.1.1
fbprophet==0.7.1



All existing libraries in digdag/digdag-python:3.7 are supported in digdag/digdag-python:3.9 except TensorFlow because they do not support Python 3.9.
Contact Support to request the complete list of libraries and their dependencies. 


  • No labels