I recently wrote a blog post discussing the human tendency to have our wants and our fears obscure what is actually the case in the world, and how this causes analytics projects to fail. There are a number of potential responses and objections to my post and I would like to address them here.

Corporate Analysis Paralysis or Misery

The thoughtful reader, arriving at this point, may ask “Doesn’t this lead to analysis paralysis? Isn’t it unrealistic to analyze everything to the nth degree? And even if we could, wouldn’t that be a miserable life?” This is like asking when embarking on a fitness program involving lifting weights — “What happens if I get too muscular? Will I have to buy new clothes? Will people think I look strange because my muscles are too large?” For 99.9% of us, these questions will never even become a problem because the endeavor that we are embarked on is challenging and rigorous enough and requires daily discipline and dedication to an extent that we will never get to this point and the benefits of even trying are so large.

“Well, what if I am part of the .1%?”

It is true that there are a few very large corporations that potentially have taken this process so far that it is “too far.” I have heard that in some parts of Google, attempting to have any sort of brainstorming discussion or idea without already having rigorous data and analysis behind it is essentially automatically rejected. I intuit that this is in many cases actually intentionally done as sort of a power restriction move because those that want to change things may not have the power to gather the data or the time/resources allocated to them to do the analysis in the first place. Most of us are not Google and frankly would be happy to have the problems Google has in terms of being a successful business. This also seems to be mixed up with companies where the engineering technocracy finds project management irritating and wants to find a justification to get rid of it which doesn’t seem to work well for anyone except the engineers who want to be left alone.

In any case, let me know if you are one of these cases and I would love to discuss these problems!

Individual Analysis Paralysis or Misery

“Doesn’t this kind of analysis make life miserable if applied to oneself? I’ve seen those people that measure every single thing and I can’t imagine trying to live that way.”

Overall I think most of us would agree that having a budget and sticking to it is a necessity for financial health. Making personal and professional goals on a regular basis, including writing them down, and periodically returning to review them is important for life achievements. If we are trying to improve our fitness, then measuring things like running speed, body weight, body fat percentage, and lifting power regularly are important.

If you agree with the points above, you have already more-or-less implicitly agreed that you are not good at tracking and improving these important aspects of your life without writing measurements down, periodically returning to review them, questioning the written word versus your intuition, and adjusting accordingly.

Now, some individuals go further then this and pursue a lifestyle called the “Quantified Self” which strives to measure and record, in great detail, every aspect of life. For example, literally carrying a video camera every second of the day and saving the footage is an extreme example of this.

The first article I see on the Quantified Self website as of February 2020 is “Testing Food with Blood Glucose” to help diabetics. What is more miserable if one is a diabetic – rigorously measuring one’s food intake to regulate blood sugar or suffering all of the complications of diabetes, up to and including amputation of limbs and death?

I will admit that I do start to reject measurement and monitoring when it impinges too much on my minute-by-minute experience of “being in life”, being present with other people, or makes me concerned that the data could be used against me by Our Evil Overlords. So there is clearly room for further discussion or thought here. ¹

The Rejection of Intuition?

“I am a creative and intuitive person and this suggests that my human emotions and intuition are useless, and even worse, paints them negatively and as not to be trusted.”

I live in Los Angeles and so I hear a lot of this kind of thing. There is a point in certain kinds of human endeavors where I do think that measurement is not necessarily helpful. In addition to being an analytics professional, I am also a guitarist. My experience of the creative process when writing music is that it is inherently about shutting off the “judgement/censor” part of my brain. That is, the part of my consciousness that makes “good/bad” decisions, which more often than not come out to be “bad”, or at least “not good.” That is exactly the part of my brain that is most active when logically analyzing and measuring things a la Orwell. There was a study that I read a while back, which I can’t find now, which tested to see what the impact of measuring things was on the emotional state of those measuring themselves. The results were that it had a measurable negative impact on individuals involved regarding the enjoyment of the activity that they were involved in. So there is a limit and time for this in certain parts of life.

I do think that there is a place for intuition and creativity in life and that trying to paint the viewpoint described in this article as saying there is not is a false dichotomy. One example would be that we need to use intuition and creativity to determine what parts of our lives to analyze and measure. We practically can’t do this to everything and so have to choose particular times and places to focus our effort. We can also train our intuition over time to at least detect where we are more likely to be making errors in perception or judgement in our lives by measuring and checking ourselves and learning where we more tend towards error.

Being human is ultimately not a nihilistic exercise in self-doubt and criticism. The goal of all of this is to make our lives better and we should use this viewpoint as a tool to do so rather than be dominated by the tool.