CS680 Linux Kernel Programming

CS680 Linux Kernel Programming - Course Syllabus - Fall 2022

Class Web page: http://web.njit.edu/~sohna/cs433 and http://canvas.njit.edu
Instructor: Andrew Sohn, GITC 4209, (973)596-2315, email: sohna _at_ njit _dot_ edu
Office Hours: Tue 10:30-11:30 am, Thur 2:30-3:30 pm, by appointment if necessary. If you want to see me outside the office hours, send me an email.
Teaching assistant: Zhaoshu Cao, zc255_at_ njit _dot_ edu; Office hours and location: Wed,Thur 3-4 pm, GITC 3700
Class time and location: See the registrar's page https://uisnetpr01.njit.edu/courseschedule
Kernel version 6.0 as of 9/1/2022: download the latest version at kernel.org
Books recommended:
- W. Mauerer, Professional Linux Kernel Architecture, 2008, Wiley, ISBN:9780470343432 (14 years old but it's still "the latest" or "the most recent"). I originally required this book but the bookstore told me they are unable to procure as it is out of print, and hence I was suggested to change required to recommended. So it's officially recommended. Go get a copy if you can find one.
- Understanding the Linux Kernel, Third Edition, 2006, Bovet and Cesati, O'Reilly, ISBN: 0-596-00565-2 (very old but you still learn a lot).
- Robert Love, Linux Kernel Development, 3rd ed (July 2, 2010), ISBN: 978-0672329463, Addison-Wesley
- Intel 64 and IA-32 Architectures Software Developer's Manual, only 4834 pages :-) as of 9/1/2022. https://software.intel.com/content/www/us/en/develop/download/intel-64-and-ia-32-architectures-sdm-combined-volumes-1-2a-2b-2c-2d-3a-3b-3c-3d-and-4.html
- CS350 textbook: Computer Systems: A Programmer's Perspective, 3/E (CS:APP3e), Randal E. Bryant and David R. O'Hallaron, Pearson (July 6, 2015), ISBN-13: 978-0134123837, ISBN-10: 0134123832.
Grading:
- Attendance (4%)
- Programming assignments (15%) - submit on Canvas
- Test1 (22%), 4-5:15 pm, Thur, 10/6/2022
- Test2 (23%), 4-5:15 pm, Thur, 11/10/2022
- Final exam (36%) on Canvas, Date and Time TBD, See the registrar's page.
Homework:
- Homework is posted on http://web.njit.edu/~sohna/cs680
- See Canvas for HW due dates and submission.
- Homework is due at 11:59 pm of the posted due date.
- Do your homework from scratch and on your own. Be prepared to spend two hours a day on homework.
- Homework must be your own work. Do not show your code and/or copy other's code.
Setting up lxr: linux cross referencer for source code traversing. See http://lxr.sourceforge.net/en/index.php for setup instructions. If you are unable to set up this applicaiton from scratch, this course is not for you. You are asked to demonstrate your functioning installation on your laptop on the due date in class.
Exam-related:
- Sample exams will be posted on Canvas for your reference. Do not rely soley on them. The contents and format can/will be completely different.
- NJIT policy on missed exams: There will be no make-up exam(s). You must plan your semester accordingly, especially if you work. Should you miss the exam(s) due to emergency, (a) go to/contact the Dean of students, (b) explain your situation as to why you had to miss, and (c) ask to issue a memo to me. If and when I receive a memo from the Dean on your missed exam, I will copy your next exam score to the missing one. Those who miss the final exam will fail in the course unless you demonstrate a true emergency again through the office of the Dean of students. No other policy will be applied. No exceptions will be made.
Academic Integrity: I am required to post this on the course syllabus.
"Academic Integrity is the cornerstone of higher education and is central to the ideals of this course and the university. Cheating is strictly prohibited and devalues the degree that you are working on. As a member of the NJIT community, it is your responsibility to protect your educational investment by knowing and following the academic code of integrity policy that is found at: http://www5.njit.edu/policies/sites/policies/files/academic-integrity-code.pdf. Please note that it is my professional obligation and responsibility to report any academic misconduct to the Dean of Students Office. Any student found in violation of the code by cheating, plagiarizing or using any online software inappropriately will result in disciplinary action. This may include a failing grade of F, and/or suspension or dismissal from the university. If you have any questions about the code of Academic Integrity, please contact the Dean of Students Office at dos@njit.edu"
Lecture Schedule by Week (may change based on class pace)
1. Introduction: Lecture Note 1
  - LAMP, virtualization, containers, datacenter computing infrastructure,
  - Review of Intel architecture based on Intel 64 and IA-32 Architectures Software Developer's Manual (4684 pages! as of 9/1/2020),
  - Short recap of Intel and AT&T Linux assembly
  - "Hello, World!" in Intel assembly, Linux assembly, assembly in-line programming with C
  - Setting up LXR (Linux cross referencer http://lxr.sourceforge.net/en/index.php) on your laptop
2. Preparatory steps: Lecture Note 2
  - Compiling the kernel, which will take hours the first time
  - Module programming - see The Linux Kernel Module Programming Guide
  - Writing your own system calls, adding to the syscall_64.tbl
  - Booting - machine BIOS, disk MBR, Grub Linux loader, preliminary setup (setup(), startup_32/64() 1 and 2)
  - Overview of kernel startup and initialization - start_kernel()
3. Memory - initialization
  - Overview of memory spaces: logical segmentation, linear virtual, actual physical
  - Detecting BIOS-provided e820 physical RAM map: detect_memory()
  - Converting to memblocks: setup_arch(): e820__memory_setup(), e820__end_of_ram_pfn(), e820__memblock_setup(), init_mem_mapping(), initmem_init()
4. Memory - Page allocator - paging, buddy system, setting up page directories (global, upper, middle), tables and PTEs
  - (N)UMA, nodes, zones (DMA .. Normal), memory types (Unmovable .. CMA .. Isolate), free_areas
  - Physical memory models: flatmem, discontigmem, and sparsemem; mem sections and subsections, pageblocks
  - Setting up buddy system: x86_init.paging.pagetable_init(), paging_init(), zone_sizes_init();
  - Allocating 1 to 1K contiguous pages from buddy system: __get_free_pages() to __rmqueue_smallest(), fastpath to slowpath to very slow path with swapping and compaction
  - Freeing pages: free_pages() to __free_one_page(), fast path to slow path
5. Memory - Slab allocator - setting up kmalloc_caches (slabs) for small objects of 8 bytes to 8 KB to large non-contiguous memory space
  - Setting up slabs for small memory objects: mm_init() to kmem_cache_init() for initializing general purpose kmalloc_caches():
  - Allocating small memory chunks: kmalloc() of 8 bytes to 8 KB from slabs: fastpath to slowpath to very slowpath
  - Freeing small memory chunks: kfree() to return to slabs, fast to slow path
  - User malloc(): will discuss briefly or provide pointers to read on your own as it's simple compared to __get_free_pages and kmalloc if time permits.
  Test 1, 4-5:15 pm, Thur, 10/6/2022.
6. Process - structrures, organization, initialization
  - Structures: thread union, thread info, stack, task, and thread struct, PID0 (swapper)
  - Macros to initialize PID0: INIT_THREAD_INFO(init_task), INIT_TASK(), INIT_TASK_TI(), ...
  - Initial hardcoded structs: init_task, init_stack, init_mm, init_fs, etc.
  Process - creating the first five kernel threads
  - Creating kernel threads: kernel_thread() - copy and insert to RB tree, create_kthread(), hot plug thread
  - P0 creating PID1 (init) and PID2 (kthreadd) using kernel_thread(),
  - P1 and P2 creating PID3 (softirqd), PID4 (migrationd) using create_kthread() through kthread_create_list
  - Hot plug threads for creating P3 and on using kthread
  - Sync mechanisms between P0, P1, and P2 for creating P1 and P2; and between (P1,P2) and (P3,P4,...) for creating P3 and on.
7. Process - process scheduling (do_fork() to schedule() to rb_entry())
  - Priority, nice value, weight, delta, weighted delta, actual runtime, virtual runtime, minimum vruntime, scheduling period, schedule slice
  - schedule() - configurable scheduling policies: completely fair scheduler CFS, realtime RT, deadline DL
  - IF TIME PERMITS, brief intro to asymmetric multiprocessing using PELT (Per Entity Load Tracking) and/or WALT (Window Assisted Load Tracking) for big.Little architectures such as Apple M1 and Intel 16-core Alder Lake (8 performance and 8 efficient cores)
  - Scheduling processes with red-black tree: pick_next_task(), put_prev_task;
  Process - process switching (schedule() to __switch_to())
  - Context switches
  - Switching to suspended process
8. Process scheduling and switching continue
  Interrupts - PICs, APICs, exceptions (traps) and hard interrupts, IDTs
  - Hardware organization: Programmable Inerrupt Controller, interrupt vectors, CPU interrupts and interrupt acknowlege to interrupt handlers.
  - Initialization: sort_main_extable(), trap_init(), init_IRQ(), softirq_init(), initializing IDTs, irq_desc and softriq_vec
  - Exceptions: exception handlers, do_trap() to do_exit() if no trap handler
  - Hard interrupts: registering interrupt handlers request_irq() to do_IRQ() to handle_level_irq(), edge vs level trigger
9. Interrupts - timer interrupts, soft interrupts, ksoftirqd, run_timer_softriq
  - Hard interrupts: IDT handler common_interrupt, interrupt_entry, do_IRQ(), device driver action->handler(), raising softirq invoke_softirq(), softirq_vec action
  - Soft interrupts: hardirq (schedule,producer,top half) vs softirq (action,consumer,bottom half)
10. Interrupts contiune - PICs, APICs, exceptions (traps) and hard interrupts, IDTs
  - ksoftirqd (kernel thread 3, P3), do_softirq(), softirq action, run_timer_softirq() as an example
  Test 2, 4-5:15 pm, Thur, 11/10/2022.
11. File system - virtual file system, block IO, elevator scheduler, device driver, softirq, timer, delayed work, kblockd_workqueue
  - initialization: vfs_caches_init, mnt_init: Virtual file system VFS
  - Registering, mounting
  - Scheduler - completely fair queuing
12. File system - device driver, Ext4 example (vfs_read() to to scsi_dispatch_cmd())
  - submit_bio(), scheduler
  - The big loop in time and space: timer, softirq, delayed work, kblockd_workqueue, kworkers
  - Ext4 disk organization: MBR, superblock, group descriptors, bitmaps, inode table, data blocks
13. Networking - receiving packets
  - receiving packets: softnet data, input_pkt_queue, process_queue, budget
  - receiving packets: NIC, ISR, Softirq, IP, TCP, Inet, BSD, User
  - User BSD sockets read(), tcp_recvmsg, , Inet socks, TCP and IP layer
  - Softirq: net_rx_action, ip_rcvmsg, tcp_recv_msg
14. Networking - sending packets
  - sending packets: output_queue, User, BSD, Inet, TCP, IP, Softirq, Qdisc, ISR, NIC
  - User BSD sockets, Inet socks, TCP and IP layer
  - Qdisc - p/bfifo packet/byte based FIFO queueing discipline, ISR interrupt service routine, NIC
  - Softirq - dequeue, transmit, ISR interrupt service routine, NIC, requeue, delayed timer
15. Final exam (week15): See the registrar's page: http://www.njit.edu/registrar

Read the following carefully to make an informed decision on whether this course is for you:

This class is extremely difficult and time consuming. You are at your own risk.
No formal lecture notes are available for this course other than the two posted at the class web page (I will post on Canvas the ones I wrote in-class on Webex during the lock-down). I used to prepare lecture notes in electronic format for distribution, spending months and years, only to find that they become obsolete in a semester or two. I don't do that anymore, knowing full well that it's likely futile in the near future.
There is little literature available on the Web for the current kernel that consists of tens of millions of lines of C and assembly.
No textbook is available that discusses the current contents, say v5.10 as of 1/18/2020. No authors in their right mind would spend years to write about the topics only to find that the contents have changed in the mean time and half the textbook might become obsolete. This is assuming that the authors are capable of fully understanding million lines of C and assembly code.
Usually about a half the class stays till the end due to its rigor and sheer amount of work involved.
It is an elective course and if you decide to stay despite my warnings you are held accountable for the course materials.
Granted some OS fundamentals stay the same over time but this course is not about the underlying theories but rather about a real operating system that powers the Internet, billions of cell phones, the Internet of Things, car dashboards, airplane entertainment systems, etc.
It has been suggested time and time again that this course should be required for every computer science student because everything you learned in computer science and a lot more are working together to make it happen.
If you follow what's discussed in class and do the homework _on your own from scratch_, you would empower yourself with advanced knowledge few have which could lead to long term security in computing as the entry barrier to the Linux kernel is high, to say the least.

Representative comments by the students in the past 15 years:

This is the most difficult course in my life.
This is the most exciting course in my life.
You need a lot more than what you learned in your life.
Somebody can change the world.
This is not for the faint of heart.