DSP的性能受其对存储器子系统的管理能力的影响。如前所述，MAC和其它一些信号处理功能是DSP器件信号处理的基本能力，快速MAC执行能力要求在每个指令周期从存储器读取一个指令字和两个数据字。有多种方法实现这种读取。比如，使用多接口存储器（允许在每个指令周期内对存储器多次访问）、分离指令和数据存储器（“哈佛”结构及其派生类）以及指令缓存（允许从缓存读取指令而不是存储器，从而将存储器空闲出来用作数据读取）另外要注意所支持的存储器空间的大小。许多定点DSP的主要目标市场是嵌入式应用系统，在这种应用中存储器一般较小，所以这种DSP器件具有小到中等片上存储器（4K到64K字左右），备有窄的外部数据总线。另外，绝大多数定点DSP的地址总线小于或等于16位，因而可外接的存储器空间受到限制一些浮点DSP的片上存储器很小，甚至没有，但外部数据总线宽。例如TI公司的TMS320C30只有6K片上存储器，外部总线为24位，13位外部地址总线。而ADI的ADSP2-21060具有4Mb的片上存储器，可以多种方式划分为程序存储器和数据存储器。选择DSP时，需要根据具体应用对存储空间大小以及对外部总线的要求来选择。

DSP processors and devices such as Intel, Pentium, or powerGeneral purpose processors (GPPS) of PC have great differences. These differences arise from the fact that the structure and instructions of DSPs are specially designed and developed for signal processing. It has the following characteristics.In order to effectively complete multiplication and accumulation operations such as signal filtering, the processor must perform effective multiplication operations. GPPS was not originally designed for heavy multiplication operations. The first major technical improvement that distinguishes DSPs from early GPPS is the addition of specialized hardware and explicit MAC instructions that can perform single cycle multiplication operations.Traditional GPPS use von Neumann storage structure, in which a storage space is connected to the processor core through two buses (an address bus and a data bus). This structure cannot meet the requirement that MAC must access the memory four times in an instruction cycle. DSPs generally use Harvard structure, in which there are two storage spaces: program storage space and data storage space. The processor core is connected to these storage spaces through two sets of buses, allowing two simultaneous accesses to the memory. This arrangement doubles the bandwidth of the processor. In Harvard architecture, sometimes a larger storage bandwidth is achieved by adding a second data storage space and bus. Modern high-performance GPPS usually have two on-chip cache memories, one for storing data and one for storing instructions. Theoretically, this dual on-chip cache and bus connection is equivalent to the Harvard structure. However, GPPS use control logic to determine which data and instruction words reside in the on-chip cache, which is usually not seen by the programmer. In DSPs, the programmer can clearly control which data and instructions are stored in the on-chip storage unit or cache.