Large Scale Data Management

The objective of this course is to give an overview of large-scale distributed data management basics.

References

• Web Data Management — Serge Abiteboul, Ioana Manolescu, Philippe Rigaux, Marie-Christine Rousset and Pierre Senellart.
• Principles of Distributed Database Systems — Tamer Özsu and Patrick Valduriez.

Examples and slides

• Examples repository
• Introduction
• Hadoop / MapReduce / Pig
• SPARK basics
• Streaming with SPARK

Putting into practice: CouchDB

• CouchDB homework
• Web Data Management, chapter 20
• Jscouch

Putting into practice: Hadoop

• Hadoop single-node setup
• Clean /tmp/hadoop* between experiments (single node, pseudo-distributed and cluster modes).
• On HDFS, use /input as the input directory.
• If the data node does not start: stop services, format the name node, clean /tmp/hadoop* and /tmp/yarn*, then restart.
• SSH daemon configuration for a cluster
• Passwordless SSH connection
• Cluster tutorial

For the cluster configuration, update conf/hadoop-env.sh with the appropriate JAVA_HOME and set SSH_OPTIONS to use port 2222.

Putting into practice: Pig

• Pig exercise

Homework and exam

• TinyRank
• Exam 2022–2023