Number of transistors doubles about every two years.

Number of transistors is still rising exponentially:

Single-thread performance is not rising as fast anymore

Programmers are now responsible for code efficiency

= number of instructions per clock $\cdot$ clock frequency

FLOPS Floating point instructions (per second)

IPS Instructions per seconds

used to be called “processor”, “processing unit”

smallest unit that can execute instructions

“multi-core processor” or if lot’s of cores “many-core processor” (= GPU)

Collection of cores on a single chip

The discipline of managing and reasoning about interacting processes that may (or may not) progress simultaneously.

Tasks can run in overlapping time periods and wont necessarily run at the same time (ie. multitasking on a single core).

Coordination of access and usage of shared resources to solve given problem.

The discipline of efficiently utilizing dedicated parallel resources (processors, memories, …) to solve individual, given computational problems.

Tasks literally run at the same time (ie. multicore processor).

Dividing given problem into subproblems that can be solved by dedicated processors in coordination.

parallel programming ≠ parallel computing (broader, more theoretical)

The discipline of making independent, non-dedicated resources available and cooperate toward solving specified problem complexes.

No centralized control, different computer memory model

= ratio of computation to communication

(these periods are separated by synchronization events)

Automatic parallelization → is restricted, compilers can’t invent a new algorithm

Hardware and compiler, “free lunch”

Parallel computing parallelism

Higher fequency of synchronization → must be done efficiently to reduce overhead.

Done with “explicit parallel programming”

The most trivial parallelization that happens in distributed computing

Lower fequency of synchronization with possibly larger data.