The Grey Zone of Small and Big Data

Is it possible to encapsulate the mood of an entire nation, using just one tiny icon? If you’ve ever been to Russia, you’ll know that smiling isn’t an everyday phenomenon. In fact, you’d be rather lucky if you met anyone smiling. In Saudi Arabia, the desert sand dominates the landscape. The humor in the UK is quirky; some may even say sophisticated. But what do all of these observations have in common? They reflect the emotional state of the populace. In fact, the emotional state of people around the world is represented by the volume of emojis they use to communicate. A telling study reveals a dramatic peak of heart emojis used in Russia, plants in Saudi Arabia, and emoji winks for Britain.

Every day, we all leave traces of emotional DNA behind us. It might be how fast we type an inquiry into
a Google search box, the application we've just filled out to secure a loan, or how fast we scroll over a page. Each of these seemingly insignificant observations forms a bridge towards Small data, shaping the next generation of how companies should tap into consumer information.

Every day, we all leave traces of emotional DNA behind us...

In 2012, when Google announced their ability to predict flu outbreaks days before it actually would happen, the entire U.S medical community find finally found a way to prepare themselves well in advance for an expected pharmaceutical demand. However, as good as it sounded, the Big Data had not taken one thing into account, the importance of Small Data. Humans rarely behave in predictable ways. Since their data required people to search Google for flu-like systems, even those flu-like systems, even those who weren't affected - and only curious - were caught in the net of Big Data. this resulted in numbers double what they should have been, according to the Center for Disease Control.

Google’s metrics focused only on the correlation—generated by Big Data—rather than the causation. Causation is the reason why things happen, which, with just a few consumer interactions, could have been detected and calculated by the model.

So, what is the perfect formula needed in the future in order to mine and draw accurate conclusions on consumer behavior? All evidence points towards a combination of both datamining and Small Data. Small Data, or human observations and interactions, create the foundation from which subsequent hypothesis can be created. Then, datamining can commence, identifying correlations and verifying trends.

But in order to fulfill this plan, we must ask this question, “what is Big Data and what is Small Data”? And where do we draw the line? Are the emoji trends from Britain, Saudi Arabia, and Russia Small Data or Big Data? And what about the pauses made in a search box on Google. Are they Small Data too?

The answer is no. Despite the size of these observations, they are merely a bridge towards Small Data. They are not Small Data, as they do not reveal the reason why these actions are occurring. Instead, the frequency of certain emoji usages accumulates, without knowing why the use of those precise symbols seems to be off the chart.

The challenge is that Small Data is rarely electronically generated. Instead, it relies on a combination of human interactions and observations, seeking to understand the reason why drilling down in the layers of human choices.

Some time ago, I went to a medical checkup in Switzerland. The doctor didn’t obtain six gallons of blood from me. Instead, he just took a couple of drops-- enough, based on modern science, to determine my health condition. Counter-intuitive to what one may think, the statistical representative of that Small Data doesn’t need to be as accurate as one may think. That is the job for Big Data. The role of Small Data is to uncritically identify as many hypotheses as possible. From my blood, the doctor can see if I’m diabetic, low on white blood cells, high in cholesterol, or suffering from a host of other anomalies, all from a few drops.

As technology advances, we’ll all be equipped with Fit Bits, Apple Watches, and homes packed with Nests. The Big Data from these devices doesn’t, in any way, explain why a person stops reacting to the never ending buzzing on the wrist, why, after a while, we lose

As technology advances, we'll all be equipped with FitBits, Apple Watches & homes packes with Nests.

interest in the Fit bit and it ends up in the drawer, or why the automatic heating and light control is turned off after a couple of months. Technology only gives us one side of the story and fails to elucidate that people feel stressed with the buzzing, get bored with the Fit Bit, and that grandma complains about all this technology when the Nest tries to turn off her lights.

Here’s the reality. There’s a reason why we’re labeled as human. With all of our flaws, and despite not fitting into a box labeled by Big Data, we should accept that our lives represent two sides of our behavior.

The right and the left brain.
The Creative and rational side.
Small and Big Data.

Once we Realize this and we also allow ourselves to be humans, the data we analyze is more accurate, useful, and relevant to our lives.