Source
Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree
The data science pipeline turns raw data into actionable intelligence: a real life problem is translated into data, analysed, and the result is communicated and used for a decision.
Stages and Their Questions
| Stage | Practical side | Thinking side |
|---|---|---|
| Collect | How to collect relevant data? Does it match the problem and represent reality? | What is the real life problem? Can it be related to data? |
| Prepare | Is the data clean enough? Is it structured for analysis? | What is the data science problem? How to formulate and solve it with data? |
| Explore | How to mine the data and compute vital statistics? | How to describe and summarise the data? Which statistics matter? |
| Visualise | How to show statistics visually and highlight interesting traits? | Can known patterns be seen? Can unknown structure be found? |
| Learn | How to build learning algorithms, reduce errors and generalise? | Can the learning be formulated and automated? |
| Infer and present | How to present descriptive and inferential analysis? | How to draw conclusions and estimate confidence? |
| Decide | Does the analysis respect law and ethics? | Can the decision be optimised using data? |
Machine learning in the "learn" stage relies heavily on algorithmic optimisation.
Three Big Blocks
The lecture groups the stages as:
- problem formulation: motivation, acquiring and preparing the data
- descriptive and inferential analytics: exploring, modelling, learning
- communication and decision: presenting results and acting responsibly
Key Idea
The overall goal is to answer "how to optimally solve a problem using data", which usually falls into one of the Five Common Data Science Problems.