CS 698-108: Advanced Topics in Web/Distributed Systems, Spring '10

Fridays, 6-9:05pm, TIER 104

Acknowledgment: The instructor would like to thank Amazon for providing a teaching grant that supports free usage of AWS for this course.

Instructor

Cristian Borcea
Office: GITC 4303
Phone: 973 596-3662
Office Hours: Thursdays 4pm-6pm and Fridays 2pm-4pm
email

News

05/15 The grades are posted. You can check them in the Highlander Pipeline. Enjoy the summer!

05/05 As I mentioned in class, you will have only 8 papers for the final exam. The following papers will not be covered by the final: week 8, paper 2 (VmWare); week 10, paper 1 (Bugs); week 12, paper 2 (McRT-STM). Also, Grid Engine is not required for the final.

05/04 The slides for today's lecture are posted.

04/30 Don't forget that we have a make-up class on Tuesday (05/04).

04/28 The slides for this week's lecture are posted.

04/23 The slides for today's lecture are posted.

04/16 The slides for today's lecture are posted.

04/15 Schedule of access to AWS for projects

04/09 The slides for today's lecture are posted.

03/29 The midterm solutions are posted.

03/26 The slides for today's lecture are posted.

03/24 I clarified with the department some concerns regarding our schedule for the rest of the semester. This is what will happen: On Friday 04/02, there will be no classes at NJIT (Good Friday). However, Tuesday 05/04 will follow a Friday schedule to make up for 04/02. Our final lecture will be on 05/07, and the final exam will be on 05/14.

03/04 You may want to check data.gov for additional data sets that you can use in your project.

02/25 The slides for tomorrow's lecture are posted.

02/22 The new deadline for the programming assignment is Sunday, 02/28 at midnight. I emailed you more details today.

02/18 The slides for tomorrow's lecture are posted.

02/14 These are the two links for the tools you will need for your assignment/project: EC2 AMI tools and EC2 API tools.

02/12 The slides for today's lecture are posted.

02/05 The slides for today's lecture are posted.

02/05 The programming assignment is posted and due on 02/26 at 5PM.

01/29 The slides for today's lecture are posted.

01/22 The slides for the first lecture are posted (see below). Also note that I changed a few minor things in the syllabus (the topics/papers remain the same, though).

Short description

This course covers new architectures and programming techniques for large scale distributed systems and web services. The topics include cloud computing, data processing in large clusters, distributed data-parallel computing, distributed storage systems, virtualization, distributed debugging, secure distributed computing, and transactional memory for multicore architectures. Students will study state-of-the-art solutions for large scale distributed systems developed by Google, Amazon, Microsoft, Yahoo, Sun, Intel, VMWare, etc. Students will also apply what they learn in a semester-long project using the Amazon Web Services platform.

Who should take this course

This class is aimed at all graduate students (both M.S. and Ph.D. students) who want to learn how recent advances in distributed systems have changed the way in which large scale Internet services and applications are designed, implemented, and deployed. By studying real-world systems developed in industry during the past few years, students will acquire cutting-edge knowledge that may be a major advantage when searching for a job. Finally, this class can help students to find topics for research projects and theses.

Prerequisites

CS 656 or CS 633 or instructor's permission. If you didn't take CS 656 or CS 633, but you would like to take this class, you should come and talk with me about your background (if you have a good background, I will give you permission to register). Basic Unix/Linux skills and good programming skills are necessary for the project.

Lectures and Readings

There is no book required for this class. Each lecture is based on recent papers/articles covering a specific topic. Every week, the instructor will introduce the topic (the lecture slides will be posted before each class) and then will moderate the discussions of the papers assigned for that week. Students are required to read the papers before the class and participate in the discussions. Additionally, each lecture will include design reviews and Q/A sessions for the semester-long project.

Paper presentation

Each student is required to prepare a power point presentation for one paper. Upon discussions with the instructor, this presentation will be incorporated in that week's lecture slides. In class, the instructor will present the context and main ideas of that week's topic, while the student will discuss the technical details of his/her assigned paper.

Project

Students will work in teams of three to design and implement Internet services or systems using Apache's Hadoop and the Amazon Web Services platform. Specific projects ideas will be provided in the first weeks of classes.

Programming Assignment

There will be one individual programming assignment before the project is handed out. This assignment willl consist of a few short programs to help students get used with Hadoop and the Amazon Web Services platform.

Exams

There will be two exams: a midterm, and a final exam. Both exams are open book (i.e., papers, notes). The final exam will cover only the material taught after the midterm. In case of missing an exam, a make-up may be taken only after providing written documentation from the Dean of Students.

Grading

Midterm exam - 20%
Final exam - 20%
Programming Assignment - 10%
Project - 25%
Paper presentation - 10%
Participation in class discussions - 15%

Schedule

Week	Topic	Readings
1	Course overview. Introduction to distributed systems and parallel computing. Slides.
2	Cloud Computing. Amazon Web Services. Slides.	M. Armbrust, A. Fox, R. Griffith, A. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia. Above the Clouds: A Berkeley View of Cloud Computing , Technical Report No. UCB/EECS-2009-28. Amazon Web Services Documentation.
3	Data Processing in Large Clusters I. Google's MapReduce. Apache's Hadoop. Slides. Programming assignment handed out.	J. Dean and S. Ghemawat, MapReduce: Simplified Data Processing in Large Clusters, Communications of the ACM, 2008. Hadoop Documentation
4	Data Processing in Large Clusters II. Yahoo's Pig Latin. Slides.	C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, Pig Latin: A Not-So-Foreign Language for Data Processing, ACM SIGMOD 2008.
5	Distributed Data-Parallel Computing. Microsoft's Dryad and DryadLINQ. Slides.	M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks, ACM EuroSys 2007. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing using a High-Level Language, Usenix OSDI 2008.
6	Distributed Storage Systems I. Google's GFS and BigTable. Slides. Programming assignment due.	S. Ghemawat, H. Gobioff, and S-T. Leung, The Google File System , ACM SOSP 2003. F. Chang, J. Dean, S. Ghemawat, W. Hsieh, D. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. Gruber, Bigtable: A Distributed Storage System for Structured Data, Usenix OSDI 2006.
7	Midterm. Project ideas presentation.
8	Distributed Storage Systems II. Amazon's Dynamo. Virtualization I. VMWare virtual machine monitor. Slides.	G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, amd W. Vogels, Dynamo: Amazon's Highly Available Key-value Store, ACM SOSP 2007. J. Sugerman, G. Venkitachalam, and B-H. Lin, Virtualizing I/O Devices on VMWare Workstation's Hosted Virtual Machine Monitor, Usenix 2001.
9	Virtualization II. Xen virtual machine monitor; Virtual machine migration. Slides.	P. Barham, B. Dragovic, K. Fraser, S. Hand, T, Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, Xen and the Art of Virtualization, ACM SOSP 2003. C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield, Live Migration of Virtual Machines, Usenix NSDI 2005.
10	System Debugging and Testing. Sun's DTrace. Slides.	D. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf, Bugs as Deviant Behavior: A General Approach to Inferring Errors in Systems Code, ACM SOSP 2001. B. Cantrill, M. Shapiro, and A. Leventhal, Dynamic Instrumentation of Production Systems, Usenix 2004.
11	Secure Distributed Computing. Slides.	X. Qie, R. Pang, and L. Peterson, Defensive Programming: Using an Annotation Toolkit to Build DoS-Resistant Software, Usenix OSDI 2002. E. Shi, A. Perrig, and L. Van Doorn, BIND: A Fine-grained Attestation Service for Secure Distributed Systems, IEEE S&P 2005.
12	Transactional Memory/Multi-core Architectures. Slides.	M. Herlihy and J. E. B. Moss, Transactional Memory: Architectural Support for Lock-Free Data Structures, ISCA 1993. B. Saha, A-R. Adl-Tabatabai, R. Hudson, C. C. Minh, and B. Hertzberg, McRT-STM: A High Performance Software Transactional Memory System for a Multi-Core Runtime, ACM PPoPP 2006.
13	More on Cloud Services. Grid Engine. Slides.	A. Adya, J. Dunagan, and A. Wolman, Centrifuge: Integrated Lease Management and Partitioning for Cloud Services, Usenix NSDI 2010. Grid Engine
14	Final project presentations.

Honor Code

The NJIT Honor Code will be upheld, and any violations will be brought to the immediate attention of the Dean of Students.

Modifications to Syllabus

The students will be consulted and must agree to any modifications or deviations from the syllabus throughout the course of the semester.