What is EDA?

"Get me some insights from this data" is a typical request given to a budding Data Analyst. Businesses often collect vast amounts of data—from customer transactions, manufacturing processes, satisfaction surveys, or marketing campaigns—and want to use it to improve their services, increase profit margins, and gain a competitive advantage. This is where a Data Analyst comes in.

A Data Analyst collects, organizes, and interprets data using statistical methods, translating it into meaningful insights that can drive business decisions. The explanatory artifacts they create from raw data help businesses identify important facts and trends that were not obvious from simply looking at the raw data.

A Data Analyst's typical workflow includes:

  • Performing an initial analysis to assess data quality.
  • Cleaning dirty data as it is discovered.
  • Performing Exploratory Data Analysis (EDA) to understand the meaning of the data.
  • Presenting findings to management with explanatory diagrams, summaries, and narratives.

Typical initial questions to start with include:

  • How is the data distributed?
  • How are the variables correlated?

Key skills required for this task are:

  • Curiosity to dive into an unfamiliar dataset.
  • The ability to identify and remove irrelevant or "dirty" data by asking questions about its source.
  • Knowledge of statistical techniques.
  • A high level of attention to detail and accuracy.
  • Strong interpersonal, teamwork, and communication skills (both written and verbal).

EDA History

Exploratory Data Analysis (EDA) is a technique for informally studying data, first introduced by John Tukey in 1977. The goal of EDA is to discover a hypothesis, which can be presented using two types of artifacts:

  • Pictures: Graphs and plots, such as histograms, bar charts, and scatter plots.

barchart scatter

  • Numerical Inferences: Summary statistics, such as mean, median, minimum, maximum, and standard deviation.
statistics

EDA Goals

The goal of EDA is to "listen" to the data to:

  • Unearth interesting correlations between variables.
  • Identify outliers, patterns, or anomalies.
  • Conclude with a hypothesis, using narratives to form a compelling story.

In its final form, the raw data will have been sifted through multiple times to unearth interesting facts, which are then presented through explanatory diagrams and narratives.

With an EDA approach, a model is not applied to the data upfront. Instead, the data is allowed to reveal its own underlying structure. Once the data is visualized, a hypothesis can be formulated. This is in contrast to standard inferential statistical testing, where you start with a hypothesis and test it against the data. EDA helps you discover a hypothesis from the data itself, based on summaries and visualizations.

An EDA session may, in fact, lead to more questions than answers, even as it clarifies initial ones. It often leads to further data collection and analysis, and can pave the way for more sophisticated hypothesis testing using inferential statistical models.

Results of EDA
  • You become familiar with the data.
  • The exploratory phase concludes with an explanatory phase, consisting of diagrams, numerical inferences, and narratives.
  • You learn which variables to focus on and which to ignore.
  • You lay the groundwork for further hypothesis testing and provide direction for future analysis.
  • You prepare the data for the application of Machine Learning (ML) models to aid in decision-making.

What Are an Analyst's Traits?

Since EDA is applied without any preconceived notions, an analyst must approach the data with an open mind. Often, the analyst will have no initial clue about the variables (also known as features) or their relationships.

A good analyst will:

  • Start with an initial question and seek an answer, then ask more questions until they are reasonably satisfied.
  • Have the perseverance to follow this iterative process.
  • Be curious and skeptical about the data.
  • Be willing to play with the data and numbers to arrive at their own conclusions.
  • Identify oddities in the data through visualizations and summaries.
  • Create narratives and tell an engaging story about the data.

The EDA technique often reveals surprising insights about the data under study and can lead to unexpected conclusions and correlations. The more meaningful diagrams you produce, the better you will understand the data.

Case Study: Hurricane Frances and the Opportunity Seen by the Wal-Mart CIO

This article was originally published in the New York Times in 2004.

Hurricane Frances was on its way, barreling across the Caribbean, threatening a direct hit on Florida’s Atlantic coast. Residents made for higher ground, but far away, in Bentonville, Ark., executives at Walmart Stores decided that the situation offered a great opportunity for one of their newest data-driven weapons … predictive technology.

A week ahead of the storm’s landfall, Linda M. Dillman, Wal-Mart’s chief information officer, pressed her staff to come up with forecasts based on what had happened when Hurricane Charley struck several weeks earlier. Backed by the trillions of bytes’ worth of shopper history that is stored in Wal-Mart’s data warehouse, she felt that the company could ‘start predicting what’s going to happen, instead of waiting for it to happen,’ as she put it. (Hays, 2004)

It would be more valuable to discover patterns due to the hurricane that were not obvious. To do this, analysts might examine the huge volume of Walmart data from prior, similar situations (such as Hurricane Charley) to identify unusual local demand for products. From such patterns, the company might be able to anticipate unusual demand for products and rush stock to the stores ahead of the hurricane’s landfall. Indeed, that is what happened. The New York Times (Hays, 2004) reported that: “… the experts mined the data and found that the stores would indeed need certain products and not just the usual flashlights. ‘We didn’t know in the past that strawberry Pop-Tarts increase in sales, like seven times their normal sales rate, ahead of a hurricane,’ Ms. Dillman said in a recent interview. 'And the pre-hurricane top-selling item was beer.'

Case Study Conclusion
  • Wal-Mart uncovered a very useful insight from its historical data.
  • Wal-Mart did not have a pre-existing hypothesis for the increased sales of Strawberry Pop-Tarts, though they did for bottled water.
  • The business not only arranged for an additional supply of Strawberry Pop-Tarts but also strategically placed them next to bottled water to make it easy for customers to grab both.
  • This case study is a perfect example of the power of EDA.

Real-World Examples of EDA

OSEMN (Pronounced "Awesome") in EDA

A popular data science pipeline is called "OSEMN," which stands for:

  • Obtaining the data.
  • Scrubbing or cleaning the data. Refer to the Data Cleaning chapter for details.
  • Exploring the data using descriptive statistical techniques, looking for outliers or anomalies, and understanding correlations between variables by plotting correlation matrices/diagrams.
  • Modeling the data, which follows the exploratory phase and often involves applying inferential statistical techniques.
  • INterpreting the data, which is the final step and typically ends with actionable decisions.

The ultimate goal of any data science pipeline, including EDA and hypothesis testing, is to make decisions or predictions that can lead to a positive impact.