Data Product Specification

The Data Product Specification is an open initiative to define a common data product format. It can be used on its own, or in combination with the Data Contract Specification.

A data product is a logical unit that contains all components to process domain data and provide data sets via output ports for analytical use.

Version

0.0.1

Example

dataProductSpecification: 0.0.1
id: shelf_warmers
info:
  title: Shelf Warmers
  description: Calculated shelfwarmers. Read about calculation in docs.
  status: active
  archetype: consumer-aligned
  maturity: managed
  owner: fulfillment
inputPorts: []
outputPorts:
  - id: glue_catalog_database_shelf_warmers_v1
    name: 'Glue Catalog Database: Shelf Warmers (v1)'
    description: All Shelf Warmers represented as a Glue Catalog table
    dataContractId: shelf_warmers_v1
    type: Glue
    status: active
    location: arn:aws:glue:eu-central-1:528115139298:table/fulfillment-shelf-warmers/shelf_warmers
    containsPii: false
    links:
      Athena Query Editor: https://eu-central-1.console.aws.amazon.com/athena/home?region=eu-central-1#/query-editor
      Glue Table: https://eu-central-1.console.aws.amazon.com/glue/home?region=eu-central-1#/v2/data-catalog/tables/view/fulfillment_shelf_warmers?database=fulfillment-shelf-warmers&catalogId=528115139298
    custom:
      platform: aws
    tags:
      - glue
      - athena
  - id: s3_bucket_shelf_warmers_v1
    name: 'S3 Bucket: Shelf Warmers (v1)'
    description: All Shelf Warmers represented as a Glue Catalog table
    type: S3
    status: active
    location: arn:aws:s3:::fulfillment-shelf-warmers
    links:
      s3: https://console.aws.amazon.com/s3/buckets/fulfillment-shelf-warmers?region=eu-central-1&prefix=output/data/
    custom:
      platform: aws
    containsPii: false
    tags:
      - s3
links:
  Discovery: https://qbffyn0bze.execute-api.eu-central-1.amazonaws.com/prod
  Documentation: http://example.com/datamesh/fulfillment/shelf_warmers
custom:
  platform: aws
tags:
  - athena
  - glue
  - s3

Specification

We currently offer the Specification in form of a JSON Schema dataproduct.schema.json.

Roadmap

• Support for specifing internals such as the pipelines

Template

We’re currently working on providing a template to help you fill out the data product specification. Stay tuned!

Tooling

• VS Code: name your file dataproduct.yaml and you get code completion within your editor.
• Data Mesh Manager supports the data product specification and allows the user to import or export data products using this specification.

License

MIT License

This site is open source. Improve this page.