Shared and Distributed Memory in Parallel Computing

5 min readMar 25, 2024

In parallel and distributed computing, memory management becomes crucial when dealing with multiple processors working together. Two prominent approaches exist: shared memory and distributed memory. This tutorial will delve into these concepts, highlighting their key differences, advantages, disadvantages, and applications. Visit the detailed tutorial on Parallel and Distributed Computing.

Shared Memory

Shared memory systems provide a single, unified memory space accessible by all processors in a computer. Imagine a whiteboard where multiple people can write and read simultaneously.

Physically, the memory resides in a central location, accessible by all processors through a high-bandwidth connection like a memory bus. Hardware enforces data consistency, ensuring all processors see the same value when accessing a shared memory location.

Hardware Mechanisms for Shared Memory

Memory Bus: The shared memory resides in a central location (DRAM) and is connected to all processors via a high-bandwidth memory bus. This bus acts as a critical communication channel, allowing processors to fetch and store data from the shared memory. However, with multiple processors vying for access, the bus can become a bottleneck, limiting scalability.

Cache Coherence: To ensure all processors see the same value when accessing a shared memory location, cache coherence protocols are implemented. These protocols maintain consistency between the central memory and the private caches of each processor. There are various cache coherence protocols with varying trade-offs between performance and complexity.

Synchronization and Coordination

Shared memory programming offers a simpler model compared to distributed memory, but it’s not without its challenges. Since multiple processors can access and modify shared data concurrently, ensuring data consistency and preventing race conditions is crucial. Programmers need to employ synchronization primitives like locks, semaphores, and monitors to control access to shared resources and coordinate execution between threads. Choosing the appropriate synchronization mechanism depends on the specific needs of the program.

Complexities and Challenges

Cache Line Size: The size of a cache line (the minimum unit of data transferred between cache and memory) can impact performance. If the cache line size is larger than the data being accessed, it can lead to unnecessary cache invalidation and bus traffic.
False Sharing: When unrelated data items are placed in the same cache line, accessing one item can invalidate the entire line, even though the other item wasn’t involved. This can degrade performance.
Memory Bandwidth: The bandwidth of the memory bus is a critical factor. If the memory access rate exceeds the bus bandwidth, processors will experience stalls, hindering performance.

Advantages

Simpler Programming: Processes can directly access and modify shared data, leading to a more intuitive programming model.
Faster Communication: Data exchange happens within the same memory space, resulting in high-speed communication compared to distributed memory.
Cache Coherence: Hardware manages cache consistency, reducing the need for explicit synchronization between processors.

Disadvantages

Scalability: Adding processors becomes complex as the shared memory bus becomes a bottleneck for communication.
Limited Memory Size: The total memory capacity is restricted by the central memory unit.
Single Point of Failure: A hardware failure in the shared memory can bring the entire system down.

Applications

Multiprocessor systems designed for tight collaboration between processes, like scientific simulations with frequent data sharing.
Operating systems for efficient task management and resource sharing.

Distributed Memory

Distributed memory systems consist of independent processors, each with its local private memory. There’s no single shared memory space. Communication between processors happens explicitly by sending and receiving messages.

Processors communicate through a network like Ethernet or a dedicated interconnection network. Software protocols manage data exchange and ensure consistency.

Hardware Mechanism for Distributed Computing

In distributed memory systems, there’s no single hardware mechanism for memory management since each processor has its private memory. The hardware focus shifts towards enabling communication and interaction between these independent memory spaces. Here’s a breakdown of the key hardware components involved:

Processors: Each node in the distributed system consists of a processor (CPU) with its local memory (DRAM) for storing program instructions and data. These processors are responsible for executing the distributed program and managing their local memory.

Network Interface Controller (NIC): Each processor is equipped with a Network Interface Controller (NIC). This hardware component acts as the communication bridge between the processor and the network. It facilitates sending and receiving messages containing data or instructions to and from other processors in the system.

Interconnection Network: The processors are interconnected through a dedicated network. This network allows processors to exchange messages with each other. Common network topologies used in distributed memory systems include:

Advantages

Scalability: Adding more processors is simpler as each has its own memory. This allows for building large-scale parallel and distributed systems.
Large Memory Capacity: The total memory capacity can be significantly larger by aggregating the local memory of each processor.
Fault Tolerance: Processor failures are isolated, and the system can continue operating with remaining functional processors.

Disadvantages

Complex Programming: Programmers need to explicitly manage data communication between processors, making the development process more intricate.
Slower Communication: Communication through the network introduces latency compared to direct access in shared memory.
Cache Management: Software protocols are needed to maintain cache coherence, adding complexity.

Applications:

High-performance computing clusters tackling large-scale problems like weather simulations and big data analytics.
Distributed computing systems where tasks are spread across geographically dispersed machines.

Choosing Between Shared and Distributed Memory

The choice between shared and distributed memory depends on several factors:

Problem Size and Data Sharing: For problems with frequent and irregular data access patterns, shared memory might be better. For problems with large and well-defined datasets, distributed memory is often preferred.
Scalability Needs: If scalability is a major concern, distributed memory offers a more flexible solution.
Programming Complexity: Shared memory can be easier to program initially, but distributed memory might require more development effort.

Shared and distributed memory represent two fundamental approaches to memory management in parallel and distributed systems. Understanding their strengths and weaknesses is crucial for BSCS students venturing into parallel programming. By carefully considering the problem requirements, students can make informed decisions about which memory architecture best suits their needs.