Jan 18th, 2021
Lessons From the Field
Part 5: 'You need a data lake.'
Written by: Craig Vosper, Chief Delivery Officer
A few years ago, we met with one of our longest standing customers and they asked us if we could assist them in implementing a data lake. A vendor of theirs (that sold data lake technology) had convinced them that they needed to do this in order to “unlock” their data. Of course, every business wants to get the most out of their data, so with the expectation of being able to make better choices, they moved forward on it.
As always with our engagements, we began by meeting with the project stakeholders and formulating an understanding their objectives. As it turns out, our customer had quite a few great ideas on how they’d like to use their data. They clearly provided us with the types of business problems and opportunities they felt it could be applied to. This just further solidified the fact that they have a comprehensive understanding of their business, regardless of their data being “locked” or “unlocked”!
With their objectives defined, we began to solve for them. We evaluated the option to use a data lake as well as a few other alternatives as part of our analysis. Ultimately the approach that was selected was an incremental development of solutions intended to provide business value fast and allowing for later expansion.
This meant that our designs needed to accommodate a quick delivery of value and consider how to limit refactoring as we moved to a more strategic platform. Throughout the project we continuously worked alongside the stakeholders to ensure they understood the tradeoffs of each decision and what it meant for their release timeframe.
After three releases of the solution, our customer decided that their objectives had been met and that they wanted to direct future investment elsewhere. We never did get around to implementing the data lake as envisioned at the start of the project. In the end, our solution still provided them with the necessary data to improve sales, reduce process costs and errors, and support them in making decisions on future projects.
It’s not wrong to start with a solution, start with whatever helps you brainstorm and clarify your ideas. In this case, it highlighted an opportunity for our customer that they may have not considered otherwise. Start wherever you are, but always ensure that your objectives get met!