Data Mesh is primarily a proposal to change the way data projects are managed within organizations. Above all, it is a cultural change.
We’ve seen this before with DataOps. Therefore, in this article we will not be discussing any technical issues, although obviously in order to implement this paradigm, we need the appropriate technological support.
We’re doing something wrong
It’s no coincidence that, more or less, most data projects fail 😣. Predictions for 2022 indicated that only 20% of projects would generate any kind of actionable results for the business, and that 80% would be isolated projects that could only be understood and managed by the engineering or data science teams that carried them out.
While we could list a multitude of technical causes — for example, poor data quality or non-robust data pipelines — and non-technical ones — such as not solving the specified problem — reality shows us that the root of failures is the disconnection 🔌 between data and business areas. The problem is so deep that we can find machine learning contests in which high school students achieve better results than data professionals 😲.
For more than 30 years, when the Data Warehouse emerged, organizations have run their data projects with a centralizing area acting as a hub that provides services to the rest of the organization.
While there’s nothing wrong with this, nowadays when we say that data is the new oil, given its value and usefulness, we cannot expect this way of working to be able to meet the new challenges we face.
Data permeates every aspect of any company’s operations and is essential for making better decisions and obtaining competitive advantages. There are countless examples of companies that are reinventing themselves to organize around a data-driven culture 📈.
Data Mesh as a Solution Proposal
In this dynamic and complex context, Data Mesh appeared in 2018 from a consulting firm called ThoughtWorks. Its author, Zhamak Dehghani, proposes it as a response to the recurring problems in data projects associated with the disconnection we mentioned. Among the most notable we can list:
The centralization of developments in a data team, independent of the rest of the organization, ultimately leads to them becoming a bottleneck and a problem in the value delivery chain.
Knowledge about domains (more on this below) is in the hands of data engineers who will then work on another domain.
Related to the above, we are asking a data engineer, who moves from project to project, to learn the details of each area of the organization, becoming a specialist in everything and nothing at the same time 😕.
Mega analytical platforms in which the knowledge of the entire company is poured, with multiple teams and areas using the same resources at the same time. This leads to inefficiencies in which specific problems can become general problems.
Lack of agility associated with monolithic solutions, not only in terms of architecture but mostly associated with the rigidity of the processes that need to be carried out so that data consumers have information available at the exact time and place.
A very large gap between the operational and analytical planes of data. While the data generated from operating a business is under the control and knowledge of the area that operates it, the analytical plane depends on external actors leading to situations such as poor interpretation of the business or incorrect assumptions, to name a few.
Multiple copies of the same data, in the best scenario with coherence between them, in the worst, with conceptual and quantitative differences of things that should mean the same thing.
Unusable data due to quality problems or loss of opportunity, that is, the data is available much later than when it had the most value. We all know that the value of data decreases as its age increases.
Dark data, we see this when there is an impediment to navigating or discovering what is available due to lack of metadata.
Inability to reach the ideal where the end users who need the data to make better actions and decisions can do so autonomously (the utopia of self-service).
About Domains and Data Products
To understand Data Mesh, we need to delve into these two concepts.
Domains represent a unit in a company associated with a particular theme. For example, in her book, the author presents the case of a digital streaming company. In which we could find areas such as product and finance that, in turn, could be composed of domains such as Podcasts and Artists the first, or Payments and Subscriptions the second.
The proposal is that these domains work on their data autonomously, independently of the others, with their own data team that will be part of the area, responding to the responsible parties.
The justification behind this is that the people who are closest to where the data is generated and worked with are the ones who will best be able to understand, interpret and decide what kind of analytics to run on them.
Then, to form the mesh, the domains will communicate with each other through data products. They are atomic and functionally cohesive units, the result of the work of a team within a particular domain to solve a problem. Following the example, the Podcasts domain could use the demographic data of the people who listen to them to provide a table with information indicating the most popular themes by age group. It is important to note that, while what the domain will share for consumption by other domains is this table, all the components necessary to reach the final result are also part of the product: code (ETL and orchestration), data used for development along with their quality tests, infrastructure and access policies.
Other examples of products could be a machine learning model or an API. Beyond the defined asset, the fundamental thing is to think of them as products, so that they will have an associated life cycle and that their construction is always focused on providing a better experience for the end user, as happens in any industry with the products it produces.
To achieve this goal, we can consider that products have certain characteristics:
Self-describable: all metadata associated with the product must clearly define it, without ambiguities so that the consumer understands exactly what it consists of.
Addressable: identification that indicates from where and in what way the product is consumed.
Discoverable: it must be part of a catalog in which users can find it easily 🔎.
Safe: access must be governed and audited. If it contains personal or sensitive information, it must be clearly identified.
Reliable: there must be quantifiable metrics to determine reliability, and the values must be published. For example, we can establish that the data is updated daily and, in case of any delay, the number of days since the last update must be included in the catalog.
Interoperable: in order to share information between domains, there must be a standardized way to do so, therefore, the products must comply with this.
One last fundamental issue associated with domains and products is ownership. Traditionally, the ultimate responsible party and source of consultation on a data asset has been the data area, or ultimately, the engineer who worked on its construction. By working oriented to domains and with well-defined products, that must change to a focus in which each product has an associated owner, with knowledge of the domain and the decisions that were made to reach the final product.
Some additional points to consider with Data Mesh are:
Data infrastructure as a platform: Data Mesh aims to allow domains to consume standardized, self-service infrastructure to facilitate the deployment of tools that make up an analytical platform. This not only speeds up implementations, but also avoids differences in the technological stack of each area, which often leads to an inability to share information because the tools cannot “speak to each other”.
Federated computational governance, to understand this we need to focus on each of the terms that make up the principle:
Governance: it refers to having control of the data from different angles. On the one hand, security to ensure that only authorized people have access according to what has been defined. Then, it is necessary to ensure compliance with legal and organizational regulations, this is especially relevant with regulations such as GDPR. It is also part of the government to verify that the policies that apply to each of the characteristics of the data products mentioned previously are met.
Federated: While decentralizing work by domains sounds very attractive, it is easy to imagine how everything could turn into chaos in 3️⃣2️⃣1️⃣. To make governance possible without falling into the extreme of total centralization, some issues are proposed to be federated and others centralized. For example, policies such as access management or the definition of protocols that will allow interoperability of data products can be centralized, while domains can be delegated to determine how to comply with all central policies. To make this process as smooth as possible, it will be essential to automate as many tasks as possible associated with everything mentioned above. Then, the domains can delegate the way to comply with all central policies, where roles such as the Data Steward can emerge.
Computational: in order for the above to function as smoothly as possible, it will be fundamental to automate as many tasks as possible associated with everything we have detailed. For example, if it is defined as a rule that products with sensitive data cannot be published, there should be a machine learning process that can identify products that do not comply with this without the need for human review.
In conclusion
Data Mesh is not something that we can simply take and implement like a data architecture. It is an invitation to deeply reflect on how we work and what our north is as data professionals.
If we become aware that we are not currently taking advantage of the full potential of data, then the discussion will not be “Data Mesh yes or no,” but rather identifying what is preventing us from delivering value to the business in a timely and effective manner and then taking action accordingly.
It may be that a change in culture, processes, or governance is necessary, but the important thing is to start a conversation and take the first steps towards a more value-driven approach to data.
Since we are mainly talking about a cultural change, it is probably best to take small steps 👣, involving some areas and actors of the organization initially and scaling incrementally and iteratively, adjusting and adapting what we have thought and planned to what we observe in real life.
By Minimalistech´s editorial team.
Minimalistech has more than 10 years of experience in providing a wide range of technology solutions based on the latest standards. We have built successful partnerships with several SF Bay Area, top performing companies, enhancing their potential and growth by providing the highest skilled IT engineers to work on their developments and projects. I
Comments