Future-Proof The Value Of Your Data Science Capability
This article will cover a topic that is a commonly overlooked requirement for building & future-proofing highly valuable data science capabilities.
Covered Here
- Why integrating data capabilities matters
- Accomplishing integration within an organization
- Overlapping priorities & skillsets for success
Why Does This Matter?
For every data scientist, it is vital to stay on top of technology trends and tools as the industry evolves. With a recent boom in Artificial Intelligence, there is a great deal of focus on emerging technology like chatGPT as a large-scale LLM-powered data product, Github Copilot to assist programmers in writing code with on-the-fly-suggestions, and of course many more.
However, a data scientists' ability to employ these new technology & skills are heavily impacted by a phrase we all know and love: "Garbage in, garbage out". This concept revolves around the idea that solid data pipelines are a crux for good data science. While many understand this to be true, the reality is that data-centric organizations don't often supplement their data science teams with dedicated, or sometimes any, data engineering support.
An unfortunately frequent set of consequences stemming from the siloing of data science teams from their engineering counterparts cause headaches all around, such as:
- Data Scientists must wade through an ocean of data spread across the organization's infrastructure and are often unequipped to properly engineer access to resources needed, resulting in a lot of time spent hacking together "quick"-fixes.
- Data Engineers encounter "hand-off"s of models or code being provided with very few requirements and the context that is vital for efficiently deploying to production environments & maintaining with quality support.
- Impressive (and sometimes expensive to build) data products never make it out the door to the hands of customers!
So as a crucial component in the success-map for enabling a highly-valuable Data Science capability, the integration of data engineering roles or support with data-product-driven teams is a guaranteed way to ensure high value delivery across the board. Data can be messy, expectations can change over time and so follows some natural chaos. But with the right capabilities in place enabling & supporting one another, the messiness will not prevail!
Data Engineering Integration Approaches
So what does it mean, or look like for a data science team to be truly supported with integrated engineering capabilities? Successful integration of a data engineering capability can take multiple forms, but the goal is always the same: empower data scientists to conduct the data "science", and spend less time doing the work of a mad-scientist working without proper foundations.
Role Additions:
A new head is added directly to data science team, or hired to directly support it. The primary benefit of this category is the swift change in data science production readiness that will come as a result, supplemented with strong skillset that will always be available to the team. Overall this is a more expensive option simply through adding a new head.
- Dedicated Data Engineer: A data engineer is hired to become dedicated to either support, or actually sit on the data science team.
Pros: This person can be a liaison to the rest of engineering & could even support other teams' data engineering teams if there is capacity.
Cons: There is a higher likelihood this individual may lack knowledge of machine-learning principles & find more trouble than success in efficiently building infrastructure to productionize data science products.
- Machine Learning Engineer: A machine-learning trained data engineer is hired to become dedicated to either support, or actually sit on the data science team.
Pros: This individual brings the pros of a dedicated data engineer with additional value stemming from comprehension of both data platform architecture and machine learning/analytics.
Cons: Due to the depth of prior experience required, this is most likely to be the most expensive option – however, the experience will be well-worth the delivery.
Existing Role Skill Enhancement
An existing team member is provided with tools, resources, reimbursement or whatever else is needed to develop new skills that fill the gaps. This category is the overall least expensive and the primary benefits include organic enhancement of cross-team collaboration and cost-effectiveness. The primary challenge with these approaches is simply time-to-launch.
- Data Scientist Learns Engineering: A data scientist develops & learns to apply the skills of a data engineer.
Pros: Enabling a data scientist to engineer will continuously reduce the dependence of the data science team on the engineering team and allow them to eventually address their own engineering challenges with ease.
Cons: Skill development takes time! There is a vast amount of information, tools, standard practices & processes, technology providers and so the overall learning curve can be steep.
- "Floating" Data Engineer: A data engineer develops & learns to apply the foundational skills of a data scientist and provides split support to engineering & data science teams.
Pros: For enabling a data science team, the foundations of machine learning & advanced analytics are likely sufficient to support productionization & deployment of models and other products. Less time to develop sufficient skills.
Cons: Capacity is never what we set it out to be. It becomes very easy for this data engineer to be swept away into other priorities as data-science is not their primary skill set. Complex mathematical principles may be difficult to learn informally if deeper knowledge of machine learning principals become required for support.
Correctly Identifying the Overlap-Gap
Now that the "why" & the "how" are addressed, it's time to discuss the "what". What technical overlap is there that when present across a team or possessed by a data science-engineer, data products will continuously fly out the door?
Below is a list of technical efforts that can be tackled & optimized best with an integrated effort. Doing so will ensure a scalable foundation for maximizing value and future-proofing the ability of a data science capability to innovate continuously.
Data Storage & Access
Database & Data Warehouse Design: A comprehensive understanding of database design concepts & how to optimize performance for as many use cases as possible is they key to being prepared for future growth, especially when it happens rapidly.
- Next to applications, data scientists are often the heaviest consumers of internal data, and often access it with more complex needs. This results in knowledge that can be used to fortify database design early on with additional requirements for advanced use cases that applications or customers may eventually embrace as well.
- Storage with some big-data tools like NoSQL databases become the most powerful when they are designed query-first.
Unified Data Platform: Data storage resources spread variably across the organization's different infrastructure services cause headaches when there is an interest in combining sources to drive new ideas. Considering data science use cases & access needs can help drive better decisions.
- Gaining & monitoring access to multiple platforms & resources becomes not just a nightmare, but sometimes a complete blocker altogether for the development of new, innovative ideas.
- It becomes easy to lose a sense of "ground truth" when there are multiple sources sharing the same data, with varying degrees of consistency.
Outside-Data Ingestion Pipelines: Data is gold! Surprisingly, a great deal of that gold is also in the public domain (considering appropriate license agreements). Making use of this data for larger-scale validation & enabling new analytical/modeling focus areas is a severaly under-utilized value-add. Consequently there are often significant roadblocks for data scientists to utilize this "gold" in development.
Machine Learning Pipelines
Experimentation Infrastructure: There are so many tools & environments for data scientists to work on models & AI these days. Designing a strategy from the start for how experimental services or outputs of said work can be integrated on a production scale is the only way to ensure the time is worth being spent.
- Without data, there is no modeling! Ensuring that sandbox and experimental environments have or can connect to valid data will ensure that models are being trained against bias & their value will scale when deployed. Simply licensing new Technology without an integration plan is a sure-fire way to burn through cash.
- Developing reliable, large-scale models (like LLMs) can be both time-consuming & costly. Comprehensive understanding of both modeling needs & how to optimize the implementation & use of infrastructure to conduct training can keep costs down long-term.
Integrated Deployment Strategy: For modern modeling & AI efforts, the typical goal is to deploy & scale in order maximize value. However, it is not often easy to plug newly developed modeling capabilities (especially with new technology) into existing (and in some cases fragile) data pipelines. As much as "hand-offs" sound like a dream, the reality is that deployment requires finesse, ingenuity & dedication from many sides.
- Embracing new concepts with like Model Registries & Model Feature Stores paired with ETL technology developed to handle & efficiently process complex transformations on big-data like PySpark will ensure that the use of models can scale quickly & efficiently.
- Emerging cloud services like AWS SageMaker & IBM Cognos/Watson can be mighty useful, but provide the most long-term value if their seamless integration functionality can be embraced, where they are utilized adjacent to data warehousing efforts & data platform architecture with the same provider.
- Once data science deployment services & pipelines have been developed, they will likely provide an avenue for data scientists to productionize similar work that may be developed down the line, with ease
Conclusion
Setting ambitious goals of delivering highly impactful and innovational data-driven products is a clear trait of many successful companies, especially those that become house-hold names. But what really drives their success is the efficiency of their engine, relying heavily on the right components that make it up, working together as whole. The organizations that fail to grow are not necessarily those that avoid AI & focus on basic data science and development, but those who do not spend the time & resources to set the proper foundations to scalable enable innovation.
A crucial step in building the proper foundations for a future-proofed data science & AI capability is the integration of data engineering & data science capabilities internally. There are several ways to do this with flexibility in money spent, who & how many people are provided with career-advancing opportunities and so on. What matters most is that data science teams are considered stakeholders of data engineering products, if not integrated in the effort from the start. Likewise, data engineers or someone with the skillset on the data science team should be thinking about plans early on for productionizing & scaling.
Thank you for reading!
