More on data mining... show me how to find the gold!

Dean Abbott posted this great comment to my recent post about Data Mining.

"Thanks for your provocative post (at least it is provocative to a data miner!) Data mining, applied well, though, doesn't take the place of the experts.

When you write that those that rely on hard data and analytics say to you "we're afraid of trusting our instincts", I would say that one validates instincts with analytics. My usual experience in building predictive models is that much of what the model finds makes perfect sense, but there are usually additional insights that even the best experts either didn't know, or didn't (or couldn't) express.

As a light-hearted example, which is true? "Birds of a feather flock together" or "opposites attract"? Well, both are, at least to some degree. Experts build intuition, but their intuition isn't always general. So yes, we should be afraid of failing (because few of us have broad enough experience to be definitive in our assessment of customers). Is it not better to augment what we know from experience with what we can learn from the data?"

Thank you Dean, for picking up on this - I was hoping someone would! - and even more, for expanding upon it in your blog.

I want to ask you something. I agree that we can see value in data retrospectively - in other words, like you said in your post, when the data confirms what you think you know. But I have to wonder, is it really naive to think that I can manipulate any data to say what I want it to say? I mean, we see the mainstream media and politicians do it all the time. (Granted this may be a very typically Gen-x way of seeing the world, where we distrust "the Man" and any corporate or mainstream messaging, especially advertising messaging.).

But we (association execs) are often told that we need to know how to ask the right questions in order to not skew the results, and for this we need a "proper" surveying company to conduct membership surveys for us. Do you think that's true?

I know not all data is survey data, but I am operating from the point of view of a very small association where we've only just begun to collect demographic and engagement data as part of our normal operations. So we do need to supplement that with surveys, and we have just completed a full comprehensive membership survey as part of our strategic planning process.

This also brings to mind an old debate on the Acronym blog about whether we are here to lead or to serve our members. Scott ended his post by saying:

"You know one of the seven measures (similar to one of the ways Collins says “great” companies distinguish themselves from “good” ones) is to be data driven. I love the principle but I hate how it’s described in 7 Measures and what it means to most people. Don’t use data to figure out what members want or how to serve them; use it as part of the discussion about what they need and how the organization can lead the profession/industry/interest forward."

Kevin Holland also continued the discussion on his blog:

"Here’s the thing: You should know what your members want and need to know about more than they do themselves, because unlike them, you should have a treasure trove of data about the online behaviors of large numbers of people just like them. You should know what articles get read, what links get clicked, what files get downloaded. And you should also know what articles don’t get read, what links don’t get clicked, what files don’t get downloaded.

Most people think of “data-driven” decisions as those based on surveys or asking people “what do you want?” I’ve always preferred to rely on real data about real behavior. With e-newsletters and dynamic websites, we have more of this data than we ever dreamed possible — and many associations don’t even bother to use it! With a few years worth of this information under your belt, being able to tell what your members want becomes second nature. Until you know and truly understand what it is that your members want, it is simply impossible for you to push through the other side and begin deciding what they need without introducing biases based on your own passions and experiences (or the passions and experiences of a limited number of members).

But, if you understand what your members want, you can give them what they need, in a way that they want to get it."

So - are we gathering data to confirm patterns of behavior (for example) that we already know about? Or are we collecting data to find gaps in information - or experience - or whatever future "need" that we could fill ? And yes, I suppose the answer is "both", but that's not very helpful. Because the real sticky point for me, is that is seems you can make the data say whatever you want. For example - at the meeting I went to, there seemed to be a case for predictive modeling software to be used in order to allocate marketing dollars to certain segments. Like "let's focus on marketing our annual conference to those most likely to attend". Which makes sense, on one level. But what if you used that same data to say, "Let's focus our marketing efforts on those who need more convincing!" and specifically targeted those least likely to attend, then drew them in, and managed to get some newly evangelistic members because they had such an unexpectedly amazing time?




I guess what I am trying to say is that if you think data can be manipulated, and/or if you think data can show patterns of behavior, and/or if you think data can show gaps in behavior, isn't that all just a lot of hard work and money involved, where the end result is that you end up doing whatever you believe is the strategic thing to do regardless?

I know this post in going in circles but that is kinda my point. : ) What do you think?