Skip to content
Products
/
/
/
Annotated Schema APIs for...
Last updated

Annotated Schema APIs for Data Classification

You can use schema Annotated Schema API to annotate your columns with tags for data classification. You can create custom tags according to your business needs, for example, tags for sensitive data, personal identifiable data (PII), financial data, and so on. With Annotated Schema API, you can list existing custom tags, tag a column, search for columns that contain specific tags, and so on.

The API endpoint is https://api-data-def-repo.treasuredata.com/.

In the following request and response examples, please note that the field sensitive is no longer supported in the UI for tagging columns. The sensitive field remains in the API only to support back compatibility.

APIDescription
GET /v1/column-annotationRequests a list of column annotations with the option to restrict to a table.
POST /v1/column-annotation/bulkCreate bulk annotations.
GET /v1/column-annotation/lookupRequest a list of columns that contain a specific column annotation.
GET /v1/annotation-type/defaultsRequest a list of default annotation types.
GET /v1/annotation-type/customRequest a list of custom annotation types.
POST /v1/annotation-type/customCreate a customized annotation type. Only administrators can call this API.
DELETE /v1/annotation-type/custom/{UUID}This operation deletes an annotation type using the UUID. Only administrators can call this API.

Column annotations have upper limits. See List of Upper Limits (MAX) on Treasure Data for details.

GET /v1/column-annotation

Requests a list of column annotations with the option to restrict to a table.

MethodResourceDescription
GET/v1/column-annotationGet a list of column annotations with the option to restrict them to a table

URI Parameters

Parameter Name

Parameter NameRequiredTypeDescription
tableIDNoStringDatabase Name and Table Name
Format: DatabaseId.TableId
Example: 1234563311.123455121
keysetNoStringUnique ID for the next page
Example: `2020-07-13T16:49:08.171013Z

Sample Request

Make sure that you use your API Key and the database id and table id you want to use.

Filter TableID

curl -s -H "Authorization: TD1 $TD_AWS_API" 'https://api-data-def-repo.treasuredata.com/v1/column-annotation?filter%5BtableId%5D=56789589.567890314'

Filter Keyset

curl -s -H "Authorization: TD1 $TD_AWS_API" 'https://api-data-def-repo.treasuredata.com/v1/column-annotation?filter%5Bkeyset%5D=2020-07-13T16:49:08.171013Z|ffbc617e-1235-4567-8901-033c75dc6601'

Sample Response

Status: 200 Success

{
  "data": [
    {
      "type": "column-annotation",
      "id": "00000000-0000-0000-0000-000000000000",
      "attributes": {
        "comment": "string",
        "createdAt": "2020-07-09T19:08:01Z",
        "sensitive": true
      },
      "relationships": {
        "annotation-type": {
          "data": {
            "id": "00000000-0000-0000-0000-000000000000",
            "type": "annotation-type"
          }
        },
        "table": {
          "data": {
            "id": "1234.5678",
            "type": "treasure-data-table"
          }
        },
        "column": {
          "data": {
            "id": "column",
            "type": "treasure-data-column"
          }
        },
        "created-by": {
          "data": {
            "id": "user",
            "type": "treasure-data-user"
          }
        }
      }
    }
  ],
  "links": {
    "first": "/v1/foo",
    "self": "/v1/foo",
    "next": "/v1/foo?page%5Bkeyset%5D=abc123"
  },
  "meta": {
    "perPage": 250
  }
}

POST /v1/column-annotation/bulk

Create bulk annotations.

MethodResourceDescription
POST/v1/column-annotation/bulkCreate bulk annotations.

URI Parameters

Parameter Name

RequiredTypeDescription
tableIDYesString

Request Body

Property NameValueDescription
typestringtreasure-data-column
attributesobjectParameters :attributes.comment: A description of the relationship between the annotation and the tableattributes.sensitive: If true, indicates that the column contains sensitive data

relationship| object| Parameters: Name of relationshiprelationships.annotation-type: The annotation to attach to the columnrelationships.column: The column to be annotated

Sample Request

Annotate two columns (“c1” and “c2") in database.table “496329.2811294” with tag “PII” (tag_id = “62290bc0-9027-461f-81d6-75c146311299”)

{
  "data": [
    {
      "type": "column-annotation",
      "attributes": {
        "sensitive": false
      },
      "relationships": {
        "annotation-type": {
          "data": {
            "id": "62290bc0-9027-461f-81d6-75c146311234",
            "type": "annotation-type"
          }
        },
        "column": {
          "data": {
            "id": "c1",
            "type": "treasure-data-column"
          }
        }
      }
    },
    {
      "type": "column-annotation",
      "attributes": {
        "sensitive": false
      },
      "relationships": {
        "annotation-type": {
          "data": {
            "id": "62290bc0-9027-461f-81d6-75c146311234",
            "type": "annotation-type"
          }
        },
        "column": {
          "data": {
            "id": "c2",
            "type": "treasure-data-column"
          }
        }
      }
    }
  ]
}

Sample Response

Status: 201 Success

{
  "data": [
    {
      "id": "361b73d4-f82c-4ea3-9230-34966436faf4",
      "attributes": {
        "comment": null,
        "createdAt": "2022-02-01T23:00:36.379169Z",
        "sensitive": false
      },
      "relationships": {
        "annotation-type": {
          "data": {
            "type": "annotation-type",
            "id": "62290bc0-9027-461f-81d6-75c146311234",
            "name": "PII"
          }
        },
        "table": {
          "data": {
            "type": "treasure-data-table",
            "id": "496789.2811294"
          }
        },
        "column": {
          "data": {
            "type": "treasure-data-column",
            "id": "c1"
          }
        },
        "created-by": {
          "data": {
            "type": "treasure-data-user",
            "id": "27770"
          }
        }
      },
      "type": "column-annotation"
    },
    {
      "id": "abd0cbf4-d66d-4fc9-a8e8-8fe45897f741",
      "attributes": {
        "comment": null,
        "createdAt": "2022-02-01T23:00:36.379169Z",
        "sensitive": false
      },
      "relationships": {
        "annotation-type": {
          "data": {
            "type": "annotation-type",
            "id": "62290bc0-9027-461f-81d6-75c146311234",
            "name": "PII"
          }
        },
        "table": {
          "data": {
            "type": "treasure-data-table",
            "id": "496789.2811294"
          }
        },
        "column": {
          "data": {
            "type": "treasure-data-column",
            "id": "c2"
          }
        },
        "created-by": {
          "data": {
            "type": "treasure-data-user",
            "id": "27770"
          }
        }
      },
      "type": "column-annotation"
    }
  ],
  "links": {
    "first": "/v1/column-annotation?filter%5BtableId%5D=496789.2811294",
    "self": "/v1/column-annotation?filter%5BtableId%5D=496789.2811294"
  },
  "meta": {
    "perPage": 2
  }
}

GET /v1/column-annotation/lookup

Request a list of columns that contain a specific column annotation.

MethodResourceDescription
GET/v1/column-annotation/lookupGet a list of the specific annotation type.

URI Parameters

Parameter Name

RequiredTypeDescription
annotationTypeNameYesString

Sample Response

Status: 200 Success

{
  "data": [
    {
      "type": "column-annotation",
      "id": "c822f1ee-95b9-440e-ac2c-3efd4c6211b7",
      "attributes": {
        "comment": "",
        "createdAt": "2021-04-19T22:06:06.585109Z",
        "sensitive": false
      },
      "relationships": {
        "annotation-type": {
          "data": {
            "type": "annotation-type",
            "id": "ecd4fa1e-a11f-11ea-b759-acde48001122",
            "name": "home-address"
          }
        },
        "table": {
          "data": {
            "type": "treasure-data-table",
            "id": "484834.2781752"
          }
        },
        "column": {
          "data": {
            "type": "treasure-data-column",
            "id": "host"
          }
        },
        "created-by": {
          "data": {
            "type": "treasure-data-user",
            "id": "18174"
          }
        }
      }
    }
  ],
  "links": {
    "first": "/v1/column-annotation/lookup?annotationTypeName=home-address",
    "self": "/v1/column-annotation/lookup?annotationTypeName=home-address"
  },
  "meta": {
    "perPage": 250
  }
}

GET /v1/annotation-type/defaults

Request a list of default annotation types.

MethodResourceDescription
GETv1/annotation-type/defaultsGet a list of annotation types provided by default to all accounts.

Sample Response

Status: 200 Success

{
  "data": [
    {
      "attributes": {
        "comment": "Identifies an email box to which messages can be delivered",
        "createdAt": "2019-08-24T14:15:22Z",
        "humanReadableName": "Email Address",
        "name": "email",
        "namespace": "Identity",
        "updatedAt": "2019-08-24T14:15:22Z",
        "version": 1
      },
      "id": "00000000-0000-0000-0000-000000000000",
      "type": "annotation-type"
    }
  ],
  "links": {
    "first": "/v1/foo",
    "next": "/v1/foo?page[keyset]=abc123",
    "self": "/v1/foo"
  },
  "meta": {
    "perPage": 0
  }
}

GET /v1/annotation-type/custom

Request a list of custom annotation types.

MethodResourceDescription
GET/v1/annotation-type/customGet a list of custom annotation types.

Sample Response

Status: 200 Success

POST /v1/annotation-type/custom

Create a customized annotation type. Only administrators can call this API.

MethodResourceDescription
POST/v1/annotation-type/customCreate a custom annotation type.

Request Body

Property NameValueDescription
attributesobjectParameters:
• attributes.comment: Description of the annotation type.
• attributes.humanReadableName: Display name shown in the UI.
• attributes.name: Annotation name used in the Permission Policy API.
• attributes.namespace: Namespace that specifies the tag type (for example, POLICY or RESOURCE).
• attributes.color: Tag color used in the UI. Accepted values:
  • POLICY_DEFAULT
  • RESOURCE_DEFAULT
  • NON_PII
  • PII
  • SECONDARY
  • SECONDARY_2

Check Create Tags documentation for color mapping in the UI.

Sample Request

Sample Response

DELETE /v1/annotation-type/custom/{UUID}

This operation deletes an annotation type using the UUID. Only administrators can call this API.

MethodResourceDescription
DELETE/v1/annotation-type/custom/UUIDDeletes a custom annotation type by using the UUID.
Previous page
Next page