Big Data Computing and Machine Learning
[CSC 483/683 - Fall 2024]

Course abstract

Nowadays we are witnessing the continuous generation of a huge volume of data from heterogeneous sources. Managing, processing, and analyzing such data becomes challenging and requires the adoption of high-performance computing frameworks and techniques. The course discusses the performance bottlenecks in traditional data processing and analytical tools and techniques and presents the opportunities to design scalable solutions leveraging distributed cluster environment architectures. Emphasis is put on the Hadoop and Spark frameworks, with practical examples of big data processing and analytical tasks in real-world applications.

AU Core Quantitative Literacy II (Q2) Outcomes:

1. Translate real-world questions or intellectual inquiries into quantitative frameworks.
2. Select and apply appropriate quantitative methods or reasoning.
3. Draw appropriate insights from the application of a quantitative framework.
4. Explain quantitative reasoning and insights using appropriate forms of representation so that others could replicate the findings.

PySpark and Databricks

In this course, we will be using the Python programming interface to the Apache Spark framework (PySpark), in combination with Databricks. You will be able to write and execute PySpark code directly in your browser, without worrying about standalone configuration, and leveraging free access to Databricks' powerful cloud infrastructure. This structure also facilitates collaborative coding.

Our course has joined the Databricks University Alliance, an active community of professors and educators who collaboratively share ideas to improve the teaching experience and provide students with most recent and relevant developments in terms of data science tools and concepts adopted in the industry. In order to start using Databricks, you can set up a free personal account (Community Edition). This option allows you to use Databricks on Amazon AWS for free. Instructions to sign up will be provided in the course.

Grading

CSC-483

Component	Weight
Midterm Exams (2)	50% (2 x 25%)
Research Papers: 2 critiques + 1 presentation	20%
Final Project	30%

CSC-683

Component	Weight
Midterm Exams (2)	40% (2 x 20%)
Research Papers: 3 critiques + 1 presentation	20%
Final Project	40%

Attendance

Students are recommended to attend all lectures. Prolonged absences must be discussed with the instructor. If you cannot attend lectures regularly, due to work or other obligations during remote learning, then please reach out to the instructor so that I know about it.

Exams

Exams cover the material from the lectures, projects, and reading. While not necessarily cumulative, each exam will require understanding many of the concepts covered in the preceding exams. Exams consist of multiple choice, short answer, and long answer questions. Each exam, except the final, is weighted equally.

For the Final Project, students will propose their own topic in consultation with the instructor.

Late Submissions

A penalty of 5% per day will be levied. The course doesn’t grant extension on the homework/lab/project submission deadline unless you have an extremely compelling excuse as observance of a religious holiday (in which case you need to let me know in advance).

Letter Grades

Range	Letter
>=93	A
>=90	A-
>=87	B+
>=83	B
>=80	B-
>=77	C+
>=73	C
>=70	C-
>=60	D
<60	F

Academic Integrity

Even though we encourage collaboration with a partner, sharing code between groups is strictly forbidden - this is a form of plagiarism. As is showing your work to other students, even just for a second. There is rarely one single correct way to write code that solves a problem. While we want you to feel free to discuss your approach freely with a partner, you should know that there are often many solutions for a given problem and it's typically obvious when one student shares code with another. If you directly copy and paste code from the Internet (or even the text), cite your source in your comments (but also ensure that you understand what the code is doing - not all code on the web is good!). Assignments will be checked using plagiarism detection software and by hand to ensure the originality of the work.

Do not share your code with anyone other than a partner. Do not let someone look at your screen. You may get behind, or your friend may ask for help, but the consequences for plagiarism are far worse than an incomplete submission - for the submission, you will still likely get some points. If I suspect that you have purposely shared code with another student or presented someone else's work as your own, the matter will be referred to the Academic Integrity Code Administrator for adjudication. If you are found responsible for an academic integrity violation, sanctions can include a failing grade for the course, suspension for one or more academic terms, dismissal from the university, or other measures as deemed appropriate by the Dean.

All students are expected to adhere to the American University Honor Code. If you have a question about whether or not something is permissible, ask the instructor or the TA first.

Textbook

This course partially adopts the textbook "Learning Spark", 2nd Edition published by O'Reilly Media, Inc.
The online version of the book may be accessible for free from AU’s online Library After selecting "O’Reilly Online Learning" from the list and logging in with your AU account, you should be able to search for the book by name, or try accessing it from this link.
Additional learning resources in the forms of slides, readings, and coding examples will be provided on Canvas throughout the course.

Acknowledgments

Course design by Roberto Corizzo at American University.

Special thanks to the Databricks University Alliance and for their educational and computational resources.

Thanks to Alex Godwin at American University for designing this syllabus template.