Designing data products

What are data products?

Before we begin designing data products, let’s first establish a shared
understanding of what they are and what they aren’t.

Data products are the building blocks
of a data mesh, they serve analytical data, and must exhibit the
eight characteristics outlined by Zhamak in her book
Data Mesh: Delivering Data-Driven Value
at Scale.

Simply put, it’s a
self-contained, deployable, and valuable way to work with data. The
concept applies the proven mindset and methodologies of software product
development to the data space.

In modern software development, we decompose software systems into
easily composable units, ensuring they are discoverable, maintainable, and
have committed service level objectives (SLOs).
Similarly, a data product
is the smallest valuable unit of analytical data, sourced from data
streams, operational systems, or other external sources and also other
data products, packaged specifically in a way to deliver meaningful
business value. It includes all the necessary machinery to efficiently
achieve its stated goal using automation.

Data products package structured, semi-structured or unstructured
analytical data for effective consumption and data driven decision making,
keeping in mind specific user groups and their consumption pattern for
these analytical data.

What they are not

I believe a good definition not only specifies what something is, but
also clarifies what it isn’t.

Since data products are the foundational building blocks of your
data mesh, a narrower and more specific definition makes them more
valuable to your organization. A well-defined scope simplifies the
creation of reusable blueprints and facilitates the development of
“paved paths” for building and managing data products efficiently.

Conflating data product with too many different concepts not only creates
confusion among teams but also makes it significantly harder to develop
reusable blueprints.

With data products, we apply many
effective software engineering practices to analytical data to address
common ownership and quality issues. These issues, however, aren’t limited
to analytical data—they exist across software engineering. There’s often a
tendency to tackle all ownership and quality problems in the enterprise by
riding on the coattails of data mesh and data products. While the
intentions are good, we’ve found that this approach can undermine broader
data mesh transformation efforts by diluting the language and focus.

One of the most prevalent misunderstandings is conflating data
products with data-driven applications. Data products are natively
designed for programmatic access and composability, whereas
data-driven applications are primarily intended for human interaction
and are not inherently composable.

Here are some common misrepresentations that I’ve observed and the
reasoning behind it :

Working backwards from a use case

Working backwards from the end goal is a core principle of software
development,
and we’ve found it to be highly effective
in modelling data products as well. This approach forces us to focus on
end users and systems, considering how they prefer to consume data
products (through natively accessible output ports). It provides the data
product team with a clear objective to work towards, while also
introducing constraints that prevent over-design and minimise wasted time
and effort.

It may seem like a minor detail, but we can’t stress this enough:
there’s a common tendency to start with the data sources and define data
products. Without the constraints of a tangible use case, you won’t know
when your design is good enough to move forward with implementation, which
often leads to analysis paralysis and lots of wasted effort.

How to do it?

The setup

This process is typically conducted through a series of short workshops. Participants
should include potential users of the data
product, domain experts, and the team responsible for building and
maintaining it. A white-boarding tool and a dedicated facilitator
are essential to ensure a smooth workflow.

The process

Let’s take a common use case we find in fashion retail.

Use case:

As a customer relationship manager, I need timely reports that
provide insights into our most valuable and least valuable customers.
This will help me take action to retain high-value customers and
improve the experience of low-value customers.

To address this use case, let’s define a data product called
“Customer Lifetime Value” (CLV). This product will assign each
registered customer a score that represents their value to the
business, along with recommendations for the next best action that a
customer relationship manager can take based on the predicted
score.

Figure 1: The Customer Relations team
uses the Customer Lifetime Value data product through a weekly
report to guide their engagement strategies with high-value customers.

Working backwards from CLV, we should consider what additional
data products are needed to calculate it. These would include a basic
customer profile (name, age, email, etc.) and their purchase
history.

Figure 2: Additional source data
products are required to calculate Customer Lifetime Values

If you find it difficult to describe a data product in one
or two simple sentences, it’s likely not well-defined

The key question we need to ask, where domain expertise is
crucial, is whether each proposed data product represents a cohesive
information concept. Are they valuable on their own? A useful test is
to define a job description for each data product. If you find it
difficult to do so concisely in one or two simple sentences, or if
the description becomes too long, it’s likely not a well-defined data
product.

Let’s apply this test to above data products

Customer Lifetime Value (CLV) :

Delivers a predicted customer lifetime value as a score along
with a suggested next best action for customer representatives.

Customer-marketing 360 :

Offers a comprehensive view of the
customer from a marketing perspective.

Historical Purchases:

Provides a list of historical purchases
(SKUs) for each customer.

Returns :

List of customer-initiated returns.

By working backwards from the “Customer – Marketing 360”,
“Historical Purchases”, and “Returns” data
products, we should identify the system
of records for this data. This will lead us to the relevant
transactional systems that we need to integrate with in order to
ingest the necessary data.

Figure 3: System of records
or transactional systems that expose source data products

Overlay additional use cases and generalise

Now, let’s explore another use case that can be addressed using the
same data products. We’ll apply the same method of working backwards, but
this time we’ll first attempt to generalise the existing data products
to fit the new use case. If that approach isn’t sufficient, we’ll then
consider developing new data products. This way we’ll ensure that we are
not overfitting our data products just one specific use case and they are
mostly reusable.

To address this use case, let’s create a data product called
“Product Recommendations” which will generate a list of suggested
products for each customer based on their purchase history.

While we can reuse most of the existing data products, we’ll need to
introduce a new data product called “Products” containing details about
all the items we sell. Additionally, we need to expand the
“Customer-Marketing 360” data product to include gender
information.

Figure 4: Overlaying Product
Recommendations use case while generalizing existing
data products

So far, we’ve been incrementally building a portfolio (interaction map) of
data products to address two use cases. We recommend continuing this exercise up
to five use cases; beyond that, the marginal value decreases, as most of the
essential data products within a given domain should be mapped out by then.

Assigning domain ownership

After identifying the data products, the next step is to determine the
Bounded Context or
domains they logically belong to.

This is done by consulting domain experts and discussing each data
product in detail. Key factors include who owns the source systems that
contribute to the data product, which domain has the greatest need for it,
and who is best positioned to build and manage it. In most cases, if the
data product is well defined and cohesive, i.e. “valuable on its own”, the
ownership will be clear. When there are multiple contenders, it’s more
important to assign a single owner and move forward—usually, this should
be the domain with the most pressing need. A key principle is that no
single data product should be owned by multiple domains, as this can
lead to confusion and finger-pointing over quality issues.

Figure 5: Mapping data products to their
respective domains.

The process of identifying the set of domains in
your organization is beyond the scope of this article. For that, I
recommend referring to Eric Evans’ canonical book on Domain-Driven Design and the Event Storming technique.

While it’s important to consider domain ownership early, it’s
often more efficient to have a single team develop all the necessary data
products to realise the use case at the start of your data mesh journey.
Splitting the work among multiple teams too early can increase
coordination overhead, which is best delayed. Our recommendation is to
begin with a small, cohesive team that handles all data products for the
use case. As you progress, use “team cognitive
load” as a guide for when to split into specific domain teams.

Having a consistent blueprints for all data products will make this
transition of ownership easier when the time comes. The new team can
focus solely on the business logic encapsulated within the data
products, while the organization-wide knowledge of how data products are
built and operated is carried forward.

Defining service level objectives (SLOs)

The next step is to define service level objectives (SLOs) for the
identified data products. This process involves asking several key
questions, outlined below. It is crucial to perform this exercise,
particularly for consumer-oriented data products, as the desired SLOs for
source-oriented products can often be inferred from these. The defined
SLOs will guide the architecture, solution design and implementation of
the data product, such as whether to implement a batch or real-time
processing pipeline, and will also shape the initial platform capabilities
needed to support it

Figure 6: Guiding questions to help define
Service level objectives for data products

During implementation, measurable Service Level Indicators (SLIs) are
derived from the defined SLOs, and platform capabilities are utilized to
automatically measure and publish the results to a central dashboard or a
catalog. This approach enhances transparency for data product consumers
and helps build trust. Here are some excellent resources on how to
achieve this:
A step-by-step guide and
Building An “Amazon.com” For Your Data Products.