DATA3404: Scalable Data Management

Learning Outcomes

• Platform tuning: Demonstrate experience with using and tuning data science platforms such as Apache Spark.
• Physical data organisation: Understand different physical data organisations, including data partitioning and data replication.
• Indexing structures: Understand disk-based indexing structures such as B-Trees, extensible hashing, and bitmap indexes.
• Query optimisation: Understand the principles of query processing and query optimisation.
• Distributed platforms: Understand the principles of distributed data science platforms.
• Sharding and replication: Understand data sharding algorithms and data replication protocols.
• Physical design: Make effective physical data design decisions.
• Performance tuning: Identify performance problems and effectively tune the performance of distributed data processing systems.

Takeaways

The most valuable shift this unit gave me was thinking about data systems at the physical layer, not just the logical one. I had always worked with databases through abstraction (ORMs, high-level query APIs), and this unit stripped that away to show what's actually happening on disk — how pages are organised, how indexes are walked, where the I/O cost actually comes from. B-Trees clicked once I saw them as a physical structure optimised for disk block reads rather than just a sorted search tree. The same mental model carried into understanding why certain query plans are expensive: a full sequential scan is sometimes cheaper than an index scan, and knowing when to trust the query optimiser versus intervene with a hint became a practical instinct. The Spark sections reinforced that distributed data systems are mostly about minimising data movement across the network. Partition pruning, predicate pushdown, and avoiding wide shuffles matter far more than micro-optimising the computation itself. I now default to thinking about data locality before anything else in a pipeline design. The physical design lab work — actually choosing partition keys and measuring the effect on query time — was the best way to make the theory stick. Seeing a 10× speedup from a partition column change is more memorable than any lecture slide.