Is the Hype Hurricane Setting the Right Expectations for Employers and Data Scientists?

Recently, I was asked to speak about thoughts on the future of data science and data scientists to a government audience. Not being a data scientist myself, rather carving my niche in the world as a solution architect turned evangelist, I pondered on a number of alternative messages I could share with the audience. Quite obviously, I am in no position to postulate on changes to analytic theories or the future of models, algorithms and such. I chose to focus on an area where I have a bit of experience, the impact of marketing and media on the perceptions of the public.

My starting point was based on what I call the Hype Hurricane. A Hype Hurricane (HH) is what happens when a topic is spun so far out of control that once done, all that’s left is a swath of damage and destruction. The HH sets unrealistic expectations and creates unsustainable demand, thus wreaking havoc on companies and careers. History is littered with Hype Hurricanes and to illustrate my point, I opted to compare the rise and fall of the “Webmaster” as the ultimate World Wide Web Technologist to the current rise of the Data Scientist as the “Sexist Job of the 21st Century”.

My story goes like this…

Rolling back to the mid-90s, the World Wide Web was the newest, shiniest thing on the planet. Very quickly a HH was started and a new Hero was born, the Webmaster. Maybe you were there for this, maybe not (showing my age, I was right in the thick of it.), but, according to the “conventional wisdom” of the time (per the HH), there was nothing a Webmaster couldn’t do. This guy or girl knew everything from networking, to servers, to operating systems, to software platforms, to databases, to programming, to site design, to graphics, to content development, to security, to…

I assume you get my point. The role was hyped to the point where Webmasters were expected to be able to do anything and everything equally well. Think of it this way, that’s akin to suggesting every musician in an orchestra be equally proficient on all instruments. Yes, there are exceptions in the world, prodigies and virtuosos who can rise to such a challenge, but, they are VERY few.

One more thing, the HH also had a hand in driving one more expectation; that companies should expect to pay $200k a year for a Webmaster, an exorbitant amount of money in the 90’s. Arguably, anyone capable of living up to the inflated expectations deserved such a payday, but, as time would show, so few could deliver.

OK, what happened? In a nutshell, there was a clamor for schools to develop programs and publishers to deliver books as employers went crazy trying to find budget to hire these expensive but, absolutely required people. Within a few months, anyone could stroll down to the local Borders, spend $150.00 on books, to include “HTML for Dummies”, or sign up to take an online class and in less than a few weeks, dub themselves a Webmaster! And employers were snatching these people up like proverbial hotcakes. But, it didn’t take long for many to realize, the people truly capable of living up to the expectations were really hard to come by. If you think back to this time, do you remember how many bad websites there were? Yes, of course, there were a few truly great sites, but, these were usually delivered by companies that recognized very early on that the best sites required a team and were not the result of a Hero.

And, thus, with disillusioned employers now paying too much money for unqualified people began the demise of the Webmaster. No longer the Hero, they faded into the background.

However, what REALLY happened? The skills once touted to be the domain of the Webmaster were still needed (in fact, still are), however, a combination of specialization and tools set in. As the best of the early Web pioneers learned, the role of the Webmaster needed to be broken into smaller chunks. People specialized so they could, in fact, get truly proficient at their jobs. At the same time, entrepreneurial companies recognized a skill gap that could be addressed with technology. Tools were developed to broaden the range of people who could successfully contribute. In addition, the tools brought the ability automate and govern development and operational processes, so when combined with best practices, an environment of rigor was brought to the realm of the WWW.

In the end, it mostly worked out; however, there were casualties along the way in the form of shattered dreams and failed companies/projects. Of course, some were doomed to failure because the ideas were just plain bad, however, no doubt, enough failed due to poor execution and/or financial problems created by hiring the wrong people and paying too much for them. Did the Hype Hurricane play a part? Of course, I don’t believe it can be proven one way or the other, however, as a guy who lived through those years and experienced them first hand, I believe it did.

Now, fast forward to today.

I believe we have a new HH with data scientists in the eye of the storm. Just Google “Data Scientist” and read the hype for yourself. Depending on the articles you read, the expectations have been set that this $300k a year person has super skills, that they know networking, servers, operating systems, software platforms (i.e., Hadoop), databases, data warehousing, programming, data storage/management, meta-data strategies, analytics, statistics, deep domain expertise, creativity, curiosity, and on and on. Likewise, we have schools quickly ramping up programs and there are books all over Amazon. And yep, rounding out the similarities, there are employers out there hiring as quickly as possible, paying top dollar for skills many didn’t even know they needed a year ago.

Deja Vu all over again?

Perhaps. Already, there are signs of specialization in the role within the companies that are pioneers. And, the availability of more easily usable tools for data scientists is on the rise, tools that offer the ability for data scientists, and, in some cases, even business analysts and senior management, the ability to dig deep into data, seeking answers, without the need, for example, to be a highly skilled Java programmer. And yes, based on the experiences from the Webmaster era, I interpret these developments as good things. Very good things!

However, I would like to note one possible exception, or perhaps caution. Unlike the unqualified Webmasters of the past whose damage was generally limited to creating unsatisfying experiences or in extreme cases, the collapse of a startup, an unqualified data scientist could possibly do far more harm. The ripple effect of a mistaken conclusion can go a long way, and stating the obvious, could have a truly detrimental effect, even to the extent of putting people in danger.

In the end, will the HH have a similar effect on data scientists and the companies hiring them as it did on Webmasters and their companies? Only time will tell. However, in the meantime, for employers and data scientists, be careful with your expectations.

For employers, before shelling out $300k a piece for shiny new data scientists, first evaluate your needs and current resources. Maybe the gap can be filled by applying technology. And, assuming you determine you need a data scientist, watch out for the posers, for obvious reasons.

For data scientists, two things. First, stay true to the skills that make you unique, the ability to use data and tools to answer questions no one ever thought of before. Don’t get too caught up in the other skills that ultimately will be divvied off to others anyway. And second, you may want to be careful buying that $800,000 home. Maybe the market will sustain your $300k a year salary, maybe it won’t. One thing experience has taught me, if I want an indicator of my future, look to the past. Technology has a way of quickly filling in gaps and changing market dynamics, thus re-valuing certain skills.

I sincerely do believe there is a great career path for data scientists, just don’t let the Hype Hurricane pull you into the vortex and leave you amidst the debris. Exercising a modicum of patience may reveal a more prudent path for employers and data scientists alike.