CIO OPINION

Daniel Zagales , VP of Data Engineering at 66degrees . com talks through his top five key data engineering and management challenges that tools can ’ t solve on their own .

However , those platforms come with a cost , and they may not support every type of integration you need to build .

Yet tools alone don ’ t solve the complexities of modern data management . To maximize the value of data engineering solutions , you need to pair your tools with the right processes and people .

To illustrate that point , I ’ d like to talk through five key data engineering and management challenges that tools can ’ t solve on their own .

1 . Bringing data under one roof

To ensure that you can integrate data in a costeffective way you must leverage integration platforms in a strategic way .

Determine which types of integrations will deliver the most value , and which ones are most costly to build yourself , then choose a data integration platform accordingly , while also ensuring that you have the engineering talent in place to build any critical integrations that your platform can ’ t handle .

Siloed data is not valuable data – and unfortunately , many businesses find that their data is highly siloed between different platforms and applications .

Finding a way to integrate data from multiple sources so that you can process and analyze it from a central location is paramount for transforming data into value .

Integration can be difficult , however , because every two data sources that you want to integrate often require a bespoke solution .

That means your engineers get stuck having to build and maintain myriad custom integrations – an approach that distracts them from other tasks and that is difficult to scale .

Managed data integration and replication platforms , which automate the process of moving data between disparate sources , can help solve this challenge .

The bottom line here is that while data integration tools can solve the data silo challenge to a significant extent , many organizations will find that data integration platforms can ’ t bring all of their data seamlessly under one roof – or , if they can , the cost of the tools may outweigh their benefits . Rather than blindly tossing tools at the challenge , businesses must be strategic about how they balance tools with costs , while also reserving the personnel required to handle integrations that tools can ’ t manage .

2 . DevOpsifying data management

By now , the typical organization has embraced agile DevOps processes as the basis for software delivery strategies . But in many cases , data management strategies haven ’ t kept pace . Organizations struggle to update data schema and pipelines as quickly as they can update application code , which means that slow data management can

44 INTELLIGENTCIO NORTH AMERICA www . intelligentcio . com