<PREV><INDEX> <NEXT>

21-765. Introduction to Parallel Computing and
Scientific Computation

The objectives of this course are:

• to develop structural intuition of how the hardware and the software work;
• to  provide guidelines about how to write and document a software package;
• to analyze the main parallel/performance programming techniques.
  

General Considerations

The course is intended to be self-consistent, no prior computer skills being required. However, familiarity with the C programming language and Unix command line should give the student more time to concentrate on the core issues of the course, as hardware structure, operating system and networking insights, numerical methods.

The main idea of the course is to give the student a hands-on experience of writing a simple software package that eventually can be implemented on a parallel computer architecture. All the steps and components of the process (defining the problem, numerical algorithms, program design, coding, different levels of documentation) are treated at a basic level. Everything is done in the context of a structured vision of the computing environment.

The typical programming environment makes the computer hardware and operating system transparent to the user. In contrast, each program intended for efficient parallel execution must consider the custom physical and logical communication topology of the processors in a parallel system. The course should give a general image over the entire range of issues that a developer should consider when designing a parallel algorithm, from principles to details. The knowledge provided by the course should be enough to help the audience decide what's the most appropriate technique to approach a problem on a given computer architecture. However, the development of an efficient algorithm will require a lot of additional study, practice, and experimental work.

The examples, exercises, and projects were determined by the computers and software available for practice. The following were preferred: the C language, the x86_64 hardware platform, and the Linux operating system. However, the presentation will be kept at a very general level such that the student is prepared for any actual parallel computing environment. The individual study and the midterm project are based on Python.

Why Linux?

The course contains three parts:

• Module 1: software package structure, design, development, and maintenance concerns.
• Module 2: parallel computing basic concepts and programming techniques: SMP, MPI, domain/data decomposition, deadlocks, hybrid programming.
• Module 3: tools for programming and cluster management: git, remote access/key management, schedulers.
• Module 4: how to transform a real life problem into a  sequential computer algorithm, with reference to basic numerical methods.

• Module 5: the layered model of the computer hardware basics.
• Module 6: a model of structural information organization with applications to filesystems and storage.
• Module 7: a typical operating system, user interfaces, shell, process communications, user level issues.
• Module 8: programming notions with applications to the C language, libraries, compilers, debuggers.
• Module 9: describes computer networks, topology, and layered communication protocols.

• Module 10: how to take advantage of multiple cores (SMP) through multi-threading and OpenMP.
• Module 11: the MPI standard, several common implementation, additional library issues.
• Module 12: the PETSC library, an interesting application of MPI for real life simulations.
• Module 13: GPU computing: CUDA and OpenACC.
• Module 14: modern developments: Big Data (Spark), Artificial Intelligence (Decision Trees, Neural Networks/Tensorflow).

Resources

• free access for CMU affiliates: https://www.cmu.edu/web/training/linkedin-learning.html

• if you don't have already an account, it can be created using your CMU credentials here.