The data supply chain

Data earns nothing until the last of six steps

Genesisdata is createda customer buysin a shopCollectionit is capturedthe till recordsthe saleProcessingcleaned and joinederrors removed,sources combinedStoragekept safelyheld with yearsof historyAnalysisquestions answeredwhich items selltogether?Activationacted uponshelves and offersare changedData is worth nothing until someone acts on it. Every stage before activation is cost.The guide calls this the data supply chain. A break anywhere stops the value: data that is never collected,sources that are never joined, or an analysis that nobody uses. When a question asks where value is created,the answer is at the end of the chain.

Genesis, collection, processing, storage, analysis and activation. Value appears only when someone acts.

The data supply chain · Going deeper

Four ways data creates value

Four ways data creates valueWhat it meansA retailer's exampleReal-time insightsSeeing what is happening now, in time to responda product selling out this hourTrendsSeeing how things change over months and yearsdemand shifting to online ordersStrategic decisionsChoosing direction on evidence, not instinctwhich towns to open shops inFuelling AIGiving models the examples they learn froma model that forecasts demandAI is only as good as the data behind it. That is why the data section comes before the AI section.An organisation that cannot collect and join its data cannot use AI on it either.

Real-time insights, trends, strategic decisions, and fuel for AI.

Kinds of data

The shape of data decides what you can do with it

STRUCTUREDordercustomertotal1041Ana40.001042Ben12.501043Chi8.75Fits rows and columnsorders, accounts, stock levelsSEMI-STRUCTURED{ "customer": "Ana",    "basket": ["tea", "cup"],    "gift": true,    "note": "leave by door" }Has labels, but no fixed columnsJSON records, logs, emails with headersUNSTRUCTUREDdocuments and messagesphotos and videorecordings of callsNo predefined shapemost of what an organisation holdsThe shape of the data decides where it can be stored and how it can be analysed.Structured data goes straight into tables and SQL. Unstructured data needs object storage, and usually AI,before it yields anything. Semi-structured data sits between: it describes itself, record by record.

Structured fits a table. Semi-structured carries its own labels. Unstructured has no set shape.

Kinds of data · Going deeper

First-, second- and third-party data

Whose data is it?KindWho collected itExampleKeep in mindFirst-partyYou, from your own customers and operationspurchases in your shopsmost trusted, yours to useSecond-partyA partner, from its customers, shared directlyan airline's loyalty listdepends on the agreementThird-partyA company that gathers it from many sourcesbought market statisticsbroad, but less preciseFirst-party data is the most valuable, because no competitor has it.Second-party data is someone else's first-party data. Third-party data is available to anyone who pays,and privacy rules apply to all three.

First-party is yours. Second-party is a partner's. Third-party is bought.

Check yourself

{ "ticket": 8812,   "tags": ["refund", "urgent"],   "device": { "type": "phone" } }

What kind of data is the record shown?

Where data lives

Three stores for three jobs

HoldsUsed byAnswersDatabaseruns the businesscurrent records, one at a timeapplications"What is in basket 4417?"Data warehouseanalyses the businesscleaned history from many systemsanalysts and dashboards"How did sales change by region?"Data lakekeeps everything, rawany data, in its original formdata scientists and AI"What might we learn from this later?"Three stores for three jobs. Most organisations need all of them.A database is built for many small, fast changes. A warehouse is built for large questions over history, sodata is copied into it and cleaned. A lake accepts anything, cheaply, before anyone knows what it is for.

A database runs the business, a data warehouse analyses it, and a data lake keeps everything raw.

Where data lives · Going deeper

The three stores, side by side

Database, warehouse and lake comparedDatabaseData warehouseData lakeShape of datastructured or documentsstructuredany: structured to unstructuredBuilt fortransactionsanalysischeap storage at any scaleTypical workread or change one recordscan millions of rowsstore now, decide laterData arrivesfrom the applicationcopied from many systemsas it is, unprocessedOn Google CloudCloud SQL, Spanner, FirestoreBigQueryCloud StorageTransactions point to a database. Analysis points to a warehouse. "Raw" and "any format" point to a lake.The products are the subject of the next lesson.

Transactions mean a database, analysis a warehouse, raw data a lake.

Data governance

Governance makes data safe to share, and proves it

Knowfind and classify the dataDecideowners, access, retentionProtectenforce the rulesProvemonitor and auditWhat it buysnumbers people can trustprivacy law is metless risk of a breachAI built on sound dataData governance is the process that decides who may use which data, for what, and proves that it happened.It is a loop, because new data arrives every day. Without it the same figure differs between reports, privatedata leaks, and nobody can say where a number came from. Governance is what makes data safe to share.

Know the data, decide the rules, protect it, and prove it. Then repeat.

Data governance · Going deeper

What a governance policy states

What a governance policy statesOwnershipA named person or team answers for each set of dataClassificationHow sensitive it is: public, internal, confidential, personalAccessWho may see or change it, granted by role and needQualityHow complete, accurate and current it must beRetentionHow long it is kept, and when it must be deletedLineageWhere it came from and what was done to it on the wayGovernance is not only security. It is also what lets people find data and trust it.Good governance speeds work up: analysts stop asking whether they are allowed and whether the data is right.

Ownership, classification, access, quality, retention and lineage.

Check yourself

five years of sales,from 12 systems, cleaned?analysts ask questionsin SQL, across regions

Which kind of store belongs at the "?"

Openness

Open formats remove walls inside and outside the organisation

SALESformat AMARKETINGformat BSUPPORTformat CThree silos: nobody sees the whole customer.ONE PLATFORMopen formatssalesmarketingsupportOne question can span all three.elsewhereAnd the data can leave: no lock-in.Openness removes two walls: the ones between your own teams, and the one around your supplier.A silo is data that only one team or one system can reach. Open formats and open standards let any tool readthe data, so it can be joined inside the organisation and moved outside it. The guide calls the two resultseliminating silos and avoiding vendor lock-in.

Open formats eliminate silos between teams and avoid lock-in to one supplier.

Openness · Going deeper

Clue words for the role of data

When the question saysThe answer isBecause"each department has its own copy"Data silosdata only one team can reach"worried about depending on one supplier"Openness, open formatsthe data can move"who may use it, and prove it"Data governanceownership, access, audit"collected from our own customers"First-party datayours, and most trusted"bought from a data provider"Third-party datagathered from many sources"images, video, audio, free text"Unstructured datano predefined shape"JSON", "fields vary by record"Semi-structured datalabels without fixed columns"process orders", "transactions"Databasemany small changes"reports across years and systems"Data warehouseanalysis of cleaned history"keep everything in raw form"Data lakeany format, decide later"turn insight into action"Activationthe last stage of the chainSection 2.1 is vocabulary. Each phrase has one term behind it.

Section 2.1 is vocabulary: each phrase has one term behind it.

Check yourself

First-partybookings made by its own guests?an airline shares its own loyalty membersThird-partytravel statistics bought from a research firm

A hotel chain uses three kinds of data. Which kind is the one marked "?"

Lesson complete

You finished “The role of data in transformation”

Answer the questions along the way to see your score here.

Open in a new tab