Back in 2011, I walked into a tiny little office on the second floor of a building that used to be a fish processing facility in the port of Tel-Aviv.
I was there for an interview with a couple of founders of a 1-year-old startup that had ten employees at the time (it would grow to over 1,000 with sales of over $150M USD)
One of the founders was in shorts and flip-flops, and the other was barefooted. It was the last time I came to that office in a buttoned shirt and jeans. I had found a new home for my career and my own bare feet!
Over the next eight years, this company (and me by association) would be at the front and center of multiple evolutions in the data and analytics space.
The first was the idea of data being visualized. No more did decision-makers, executives, and board members want to see long, dull reports. They wanted pictures and graphs!
Next, very naturally, came the idea of democratizing Business Intelligence – now that data visualization was cool, everyone wanted to do it but doing it with old-school solutions meant too much investment and long turnaround times.
Enter the self-serve BI solutions – That was some evolution! – Products like Tableau, Domo, Sisense, Looker, and Power BI flooded the market like a tsunami, and over time they washed away many of the legacy enterprise solutions.
In parallel to these solutions gaining incredible popularity were two other trends:
- Business users started to demand better scale and performance of analytics, so IT teams needed to buy data storage solutions that could stand up to these new demands.
- The ownership of analytics started to shift from IT to business functions. This was profound because IT teams found it challenging to decouple what was, up until then, a single workflow owned by them. Additionally, business teams didn’t have the resources or experience to manage software solutions like this.
- As a result of point 1 above, IT emerged from the world of on-prem heavy databases to a new era of cloud-based super data stores like SnowFlake, Amazon Redshift, Azure SQL, Google Cloud, and more recently, Firebolt.
- The result of point 2 was far more profound – companies formed new departments that bridged the gap between the business and IT and armed them with hefty budgets. This is where the BI&A teams and the Chief Data Officer were born.
Now, back to my story – I was deep into enterprise software sales by then – my team was working on selling a BI solution to Fortune-500 companies. We all saw the impact of this evolution and how well it enabled businesses to become more agile while starting to build a truly data-driven culture.
Not all evolutions succeeded, and some were massive failures. The biggest one I saw was companies attempting to consolidate all their data ecosystem into one pipeline with a single tool for each section (storage, ETL, governance, analytical store, BI & visualization).
Companies spent millions of dollars and years of manpower researching and attempting to execute these plans. Most failed, and huge projects were shelved while teams were dismantled.
I learned from this that trying to force users into a certain behavior when it is massively counterintuitive is extremely risky. Following this came a period of stabilization rather than aggressive changes and innovations. The cloud data platforms became more standardized and saw the phenomenal expansion of Snowflake and their subsequent IPO, multiple acquisitions in the BI space like Tableau and Looker and a new set of modern ETL and governance tools emerged to support the move to the cloud.
Over the last few years, a new challenge has emerged – as data became more widely available to everyone in the business, it rose the expectation of people to be more data-driven.
Across the board, people are asked to bring data to the table and have data evidence for their decisions. But how to truly scale data access so that anyone can use it? What will be of those who lack the understanding of data structures or the experience of how to analyze data?
How can a new analyst in a company get familiar with a deep and ever-growing data environment? How can an operations manager know and prove the data used by their team is accurate and complete? Having a reliable way to ensure accuracy is becoming very critical.
Executives are losing confidence in their data-driven decisions because they have no way of being reassured the data they were given was accurate. Furthermore, the people who put the analysis together in the first place have no formal way of proving this.
Data cataloging solutions started to emerge as a middle layer that bridges this gap. Solutions like Collibra and Alation built enterprise-grade applications to record and manage a company’s metadata as a means to document and formalize analytics.
Following them came Data.World, Atlan, and Acryl, all offering a slightly more agile solution for cataloging data.
Six months ago I joined illumex to revolutionize the space of metadata management. We built a fully automated solution for cataloging data and custom ontologies.
Not only are we building these ontologies for each customer, but we are also feeding them into a global graph that will serve as the standard in data modeling and analytics metrics, to then impart its knowledge to all our clients.
Our cause – To enable any business questions and make every answer reliable! I feel inspired by this cause and hope to see the entire industry join hands to create an ecosystem that supports this.