Source

Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree

The data science pipeline turns raw data into actionable intelligence: a real life problem is translated into data, analysed, and the result is communicated and used for a decision.

Stages and Their Questions

StagePractical sideThinking side
CollectHow to collect relevant data? Does it match the problem and represent reality?What is the real life problem? Can it be related to data?
PrepareIs the data clean enough? Is it structured for analysis?What is the data science problem? How to formulate and solve it with data?
ExploreHow to mine the data and compute vital statistics?How to describe and summarise the data? Which statistics matter?
VisualiseHow to show statistics visually and highlight interesting traits?Can known patterns be seen? Can unknown structure be found?
LearnHow to build learning algorithms, reduce errors and generalise?Can the learning be formulated and automated?
Infer and presentHow to present descriptive and inferential analysis?How to draw conclusions and estimate confidence?
DecideDoes the analysis respect law and ethics?Can the decision be optimised using data?

Machine learning in the "learn" stage relies heavily on algorithmic optimisation.

Three Big Blocks

The lecture groups the stages as:

  • problem formulation: motivation, acquiring and preparing the data
  • descriptive and inferential analytics: exploring, modelling, learning
  • communication and decision: presenting results and acting responsibly

Key Idea

The overall goal is to answer "how to optimally solve a problem using data", which usually falls into one of the Five Common Data Science Problems.