Data quality
Quality is measured per dimension, with a rule and a threshold
Measure completeness, validity, uniqueness, consistency, timeliness and accuracy, each as a rule a pipeline can check.
Is this data good enough? The only useful answer is measured. Eight customer rows: one email missing, one malformed, one id twice, one country spelled out instead of coded. Turn each kind of fault into a score. Completeness: 7 of 8 emails present. Validity: 6 of those well formed. Uniqueness: 7 ids in 8 rows. Consistency: 7 of 8 countries coded. Timeliness: loaded 26 hours ago against a promise of 24. Accuracy, whether it is actually true, needs something trusted to compare with.
Data quality · Going deeper
Data contracts and data observability
A data contract states what a producer promises; data observability watches freshness, volume, schema and distribution.
A data contract states what a producer promises; data observability watches freshness, volume, schema and distribution. Agree what “good” means. A data contract is a written, versioned agreement between the team that produces a table and those that use it: the schema, the meaning of each column, quality rules, freshness, and who to call. Breaking changes go through it, like an API. What to monitor automatically.
Agree what “good” means
What to monitor automatically
Quality checks
Each rule needs a decision: record it, drop the row, or stop
A pipeline expectation can warn and keep rows, drop failing rows, or fail the update.
Rules are only useful inside the pipeline. A batch of 1,000 rows arrives; 12 have impossible totals. Warn, the default for Lakeflow expectations: write everything and record the failures. Drop: write the 988 good rows and leave out the rest, ideally into a quarantine table. Fail: stop the update entirely and alert someone. Choose by consequence. A wrong price should stop the load; a missing middle name should only be counted.
Data lineage
Lineage traces each column back to its source and forward to its users
Lineage shows where data came from and what depends on it: upstream for root cause, downstream for impact analysis.
Every number on a dashboard has a history. Revenue in gold is the sum of total in silver, which was cast from a text amount in bronze, which came from the orders app. A check at each hop catches a fault close to where it entered. And downstream, a dashboard and a model read it. So when revenue drops thirty percent, lineage leads upstream to the cause. And before you rename a column, lineage shows everything that would break. Unity Catalog records this automatically, column by column.
Data catalogues
A catalogue says what data exists, what it means and who owns it
A data catalogue holds technical, business and operational metadata, and tags such as PII that drive policy.
A lake with no catalogue is a swamp: the data is there, but nobody can find it or knows which copy is right. A catalogue entry holds technical metadata, collected automatically: format, columns, location, size. Business metadata, added by people: the owner, what one row means, glossary terms. Tags, such as PII, that policies act on. And operational facts: when it was updated, its quality score, how many people use it. Unity Catalog, the Glue Data Catalog, Microsoft Purview, Fabric's OneLake catalog and Snowflake Horizon all play this role.
Access control
Grant the least access to groups, on the object and its containers
Use GRANT and REVOKE on securable objects, to groups, following least privilege; Unity Catalog needs USE CATALOG, USE SCHEMA and SELECT.
Access is granted on objects, and objects sit inside containers. To read the orders table in Unity Catalog, a user needs USE CATALOG on the catalog, USE SCHEMA on the schema, and SELECT on the table. Grant on a schema and everything inside inherits it. The statements are plain SQL: GRANT to give, REVOKE to take away. Three habits: grant to groups, not individuals, so joiners and leavers are one change; give the least access that does the job; and keep the audit log.
Access control · Going deeper
Access models on Snowflake and AWS
Snowflake uses role-based access control; AWS Lake Formation adds table, column and row permissions on top of IAM.
Snowflake uses role-based access control; AWS Lake Formation adds table, column and row permissions on top of IAM. Snowflake. Roles hold privileges; users hold roles GRANT SELECT ON TABLE orders TO ROLE analyst; GRANT ROLE analyst TO USER asha. Roles can be granted to roles, forming a hierarchy. System roles ACCOUNTADMIN (top, use sparingly), SECURITYADMIN (manages grants), USERADMIN (users and roles), SYSADMIN (warehouses and databases), PUBLIC (everyone). Policies masking policies hide column values and row access policies filter rows, both by role, attached to columns or tables. AWS. IAM who can call which service and API, and on which S3 paths. Lake Formation grants on Glue Data Catalog databases, tables and columns, with data filters for rows and cells, enforced for Athena, Redshift Spectrum, EMR and Glue. Tag-based access control (LF-Tags) grants by tag instead of by table.
Snowflake
- Roles hold privileges; users hold roles GRANT SELECT ON TABLE orders TO ROLE analyst; GRANT ROLE analyst TO USER asha. Roles can be granted to roles, forming a hierarchy.
- System roles ACCOUNTADMIN (top, use sparingly), SECURITYADMIN (manages grants), USERADMIN (users and roles), SYSADMIN (warehouses and databases), PUBLIC (everyone).
- Policies masking policies hide column values and row access policies filter rows, both by role, attached to columns or tables.
AWS
- IAM who can call which service and API, and on which S3 paths.
- Lake Formation grants on Glue Data Catalog databases, tables and columns, with data filters for rows and cells, enforced for Athena, Redshift Spectrum, EMR and Glue. Tag-based access control (LF-Tags) grants by tag instead of by table.
Row and column security
One table can return different rows and values to each user
A row filter limits which rows a user sees; a column mask changes what a column shows; both are functions on the table.
Often people may see some of a table, not all of it. An admin runs SELECT star and gets all 5 sales with full emails. An EU analyst runs exactly the same query and gets the 3 EU rows, with emails masked. The row filter is a function that says which rows each user may see; the column mask is a function that says what value they see. Both are attached to the table, so every tool and every query obeys them, with no copies to keep in step.
Masking and anonymising · Going deeper
Masked, hashed and tokenised data is still personal data
Pseudonymisation (hashing, tokenisation) can be linked back and stays personal data; anonymisation cannot, and does not.
Encryption
Encrypt in transit and at rest; whoever holds the key holds the data
Use TLS in transit and encryption at rest, with keys in a key management service; customer-managed keys give you control.
Encryption protects data when everything else fails. Every connection, from the analyst to the engine to storage, uses TLS. On disk, every major cloud now encrypts with AES-256 by default. So the real decision is the keys. A key management service holds them and logs every use; with a customer-managed key you can rotate it, or revoke it and make the data unreadable everywhere at once. And identifiers can be pseudonymised, so analysts can join on them without seeing who they are.
Envelope encryption · Going deeper
A data key encrypts the data; the KMS key encrypts the data key
Envelope encryption stores an encrypted data key beside the data; only the key service can unwrap it.
Privacy and retention
A deleted row survives in history and copies until those are purged
Erasure under GDPR means every copy: after DELETE, time travel still exposes the row until VACUUM, here after 7 days.
A customer exercises their right to be erased. You run DELETE, and they vanish from the current version of the table. But the old data files are still there, and anyone with access can read them through time travel, for the default 7 days. Only VACUUM, after the retention period, deletes those files. And there are bronze copies, exports, backups and caches too. Privacy has to be designed in: collect only what you need, tag personal data, set retention on every layer, and know where each person's data lives.
GDPR in numbers · Going deeper
GDPR in the numbers a data team has to build for
A breach is reported within 72 hours, a request answered within one month, and the controller answers for both.
A breach is reported within 72 hours, a request answered within one month, and the controller answers for both. Who is responsible. Controller decides why and how personal data is processed: your company. It answers to the regulator. Processor processes on the controller’s instructions: the cloud provider, a SaaS tool. Bound by a data processing agreement. What each right asks of the platform. Access and portability (Articles 15, 20) find every row about one person, in every table, and export it. Needs a catalogue that tags personal data and a common person key. Rectification (Article 16) fix it at the source and let the pipeline carry the fix through. Erasure (Article 17) delete from every layer, then purge history: VACUUM, Time Travel, backups. Storage limitation (Article 5) a retention period on every table, enforced by a job or a lifecycle rule, not by memory. Other regimes, and the tools for retention. CCPA, HIPAA, PCI DSS California’s consumer privacy law (know, delete, opt out of sale); US health records; card data, where the security code may never be stored after authorisation. Deleting on time S3 Lifecycle rules move objects to colder classes and expire them; Azure Blob lifecycle management does the same. Keeping on purpose S3 Object Lock and Azure immutable storage make objects write-once: in compliance mode nobody, the root user included, can delete them before the date.
Who is responsible
- Controller decides why and how personal data is processed: your company. It answers to the regulator.
- Processor processes on the controller’s instructions: the cloud provider, a SaaS tool. Bound by a data processing agreement.
What each right asks of the platform
- Access and portability (Articles 15, 20) find every row about one person, in every table, and export it. Needs a catalogue that tags personal data and a common person key.
- Rectification (Article 16) fix it at the source and let the pipeline carry the fix through.
- Erasure (Article 17) delete from every layer, then purge history: VACUUM, Time Travel, backups.
- Storage limitation (Article 5) a retention period on every table, enforced by a job or a lifecycle rule, not by memory.
Other regimes, and the tools for retention
- CCPA, HIPAA, PCI DSS California’s consumer privacy law (know, delete, opt out of sale); US health records; card data, where the security code may never be stored after authorisation.
- Deleting on time S3 Lifecycle rules move objects to colder classes and expire them; Azure Blob lifecycle management does the same.
- Keeping on purpose S3 Object Lock and Azure immutable storage make objects write-once: in compliance mode nobody, the root user included, can delete them before the date.
Lesson complete
You finished “Data quality and governance”
Nicely done.