记忆是人脑最高级的认知功能之一，也是个体生存和发展的重要前提。经典的记忆三阶段模型指出记忆一般包含编码、保持和提取三个过程，其中短时记忆保持是通过部分参与编码的神经元的持续激活来实现，而记忆提取则是编码阶段神经表征的复现。虽然这些理论都强调了编码阶段的神经表征在记忆保持和提取过程中的再激活，但是近年来越来越多的研究证据发现，大脑对记忆材料的神经表征在同一记忆阶段内，以及不同记忆阶段之间都会发生变化，提示了记忆的动态表征属性。然而，当前关于记忆动态表征的研究处于初期阶段，对于不同阶段的记忆动态表征特性仍有待进一步研究。具体而言，(1)在记忆编码方面，先前研究关注了视觉特征沿腹侧视觉通路的动态表征，但是忽略了自然视觉物体编码阶段的语义信息表征。语义表征作为人类视觉物体加工的关键成分之一，是形成稳定记忆的重要前提，因而全面考察视觉信息从视觉表征到语义表征的动态过程非常必要。(2)在记忆保持方面，先前研究大量探讨了视觉短时记忆保持的时间特征和空间存储位置，但是没有直接比较从编码到短时记忆阶段，神经表征的内容和形式是否发生变化，以及参与神经表征的脑区是否发生了转移。(3)在记忆提取方面，先前研究集中考察了记忆提取是否复现了编码晚期的神经表征，但是没有考察记忆提取阶段的神经表征内容与编码阶段的异同，以及何种编码表征能够促进记忆的有效形成和提取。本论文采用了具有高时间分辨率和空间分辨率的颅内脑电记录和脑磁图成像技术，结合前沿的中文语义向量模型、视觉深度神经网络模型以及多变量模式分析方法，通过三个研究系统地回答这些问题。

研究一以耐药型癫痫患者为被试，采用颅内脑电技术结合视觉深度神经网络模型以及中文语义向量模型，通过两个实验分别考察了视觉物体和视觉词语在编码阶段从基本视觉特征到抽象语义的动态表征过程。实验1结果发现视觉物体加工过程中先出现视觉表征(刺激呈现后300-600ms左右)，然后是语义表征(刺激呈现后的370-1980ms)，再是抽象语义表征(刺激呈现后530-1330ms)。实验2在实验1的基础上使用视觉词语作为刺激材料，考察不同视觉刺激材料的动态表征的一般规律。结果发现视觉词语的语义表征从刺激呈现后450ms持续到2020ms，和实验1中的语义表征的时间窗高度重合。这些结果表明视觉刺激在编码阶段的表征内容高度动态变化，语义表征晚于视觉表征，且不同的视觉材料的编码过程中存在相似的语义表征时间进程。

研究二通过两个实验分别考察了从编码到短时记忆保持阶段的表征内容及其强度在时间上的变化(实验3)，以及参与神经表征的脑区在空间上的转移(实验4)。实验3使用颅内脑电记录，结合表征相似性分析和表征动态分析方法，考察了短时记忆保持阶段的表征内容及其时间特征。结果发现：在自然视觉物体的短时记忆保持阶段，其表征内容既包含了来自编码早期(刺激呈现后250-770ms)的高层级视觉表征，也包含了编码晚期(刺激呈现后1000-1980ms)的抽象语义表征，并且抽象语义表征更强。虽然两种表征的强度总体上保持稳定，但在更精细的时间尺度上，这两种表征内容强度存在波动变化，并且这种波动和海马低频活动相位存在耦合。这提示，短时记忆同时保持了高级视觉和抽象语义信息，并且可能通过与海马相位锁定的间歇性神经激活来实现对这两种信息的稳定保持。由于颅内脑电的电极空间分布不均匀，难以系统描述参与编码和短时记忆保持的脑区。实验4在实验3的基础上使用了脑磁图成像技术，重点考察了从编码到保持阶段参与脑区的空间转移。基于脑区和单通道的多变量解码分析发现，编码阶段的表征主要在枕叶、颞叶和顶叶等区域，其中枕叶区域的表征最强。短时记忆的表征则分布在皮层的各个区域，但在颞叶和额叶区域最强。这些结果表明，从编码到短时记忆保持阶段，信息表征的核心脑区从大脑皮层的后部区域向前部区域转移。

研究三结合了编码、保持以及提取任务，着重考察了记忆提取阶段的神经表征内容以及编码阶段的动态表征和记忆提取的关系。结果发现：成功的记忆提取包含抽象的语义表征，而非视觉表征。进一步考察何种编码表征有利于记忆的形成和提取，结果发现编码阶段从视觉表征到语义表征的动态变化越强，记忆提取成绩越好。编码阶段的动态表征通过提高编码晚期的项目特异性表征进而促进记忆提取。此外，记忆提取复现了短时记忆保持阶段而非编码阶段的神经表征，表现为提取和保持阶段的表征相似性显著高于提取和编码阶段的表征相似性。这些结果表明，记忆信息的神经表征从编码到短时记忆保持再到提取阶段，经历了多个阶段的表征转变。

综上所述，本论文通过上述三个研究，深入揭示了人脑记忆表征的动态变化机制。编码阶段的视觉输入通过和已有知识经验的动态交互作用，产生从视觉到抽象语义等多重神经表征；其中，高级视觉表征和抽象语义表征能够在短时记忆中进行稳定保持，而只有抽象语义表征在记忆提取中得到了很好的再现。此外，如果编码阶段的神经表征内容能产生更强的从视觉到抽象语义的转变，则越有助于成功的记忆提取。这些发现挑战了情景记忆的模式再现的传统假说，为发展有效学习和记忆提升方法提供科学指导，也为发展基于人脑学习记忆机制的新一代人工智能提供启发。

As one of the higher cognitive functions of the human brain, memory plays a critical role in human survival and development. The classic model of memory notes that memory includes three stages: encoding, retention, and retrieval. Short-term memory retention is achieved via persistent firing of neurons that are involved in encoding, while retrieval faithfully reinstates the neural representations formed during encoding. Although this model emphasizes the reactivation of encoded neural representations during memory retention and retrieval, emerging evidence in recent years has started to show that the neural representations of learned items are substantially transformed within the same memory stage as well as across different memory stages, suggesting the dynamic nature of memory. However, existing studies on the dynamic nature of memory are still in the early stage. Specifically, (1) In terms of memory encoding, previous studies focused on the dynamic representation of visual features along the ventral visual pathway, largely ignored the semantic representation in the encoding stage of natural visual objects. As one of the key components of human visual object processing, semantic representation makes a crucial contribution to the formation of stable memory. Therefore, a comprehensive investigation of the dynamic process of visual information from visual representations to semantic representations is essential. (2) In terms of memory retention, previous studies have extensively explored the temporal characteristics and spatial storage locations of visual short-term memory retention, but they have not directly tested whether the content and format of neural representations between encoding and short-term memory and whether the brain regions engaged in encoding period are shifted during short-term memory. (3) In terms of memory retrieval, previous studies focused on whether memory retrieval reinstates the neural representation of the late encoding period, yet it is still unclear what content and format of neural representations are reinstated during successful retrieval, and how they are related to the representations during encoding and retention. Still, it is largely unknown what kind of encoding representations can promote effective memory formation and retrieval. The current dissertation employs intracranial electroencephalogram (iEEG) recording and magnetoencephalography (MEG) imaging technology with high temporal and spatial resolution, combined with cutting-edge methods, such as word-to-vector model, visual deep neural network model, and multivariate pattern analysis, to thoroughly answer these questions by three systematical studies.

In the first study, we conducted two experiments to investigate the dynamic representations of visual objects and words in the encoding stage. In both experiments, patients with drug-resistant epilepsy were recruited as subjects and neural activities were monitored via intracranial EEG recordings. The data were then analyzed with a deep visual neural network model and a Chinese word-to-vector semantic model. The results of experiment 1 found that visual representations first appeared in the processing of visual objects (about 300-600ms after post-stimulus), followed by semantic representations (370-1980ms post-stimulus), and then abstract semantic representations (530-1330ms post-stimulus). Based on Experiment 1, Experiment 2 uses visual words as stimuli to test whether there are general laws of dynamic representations during the encoding of different types of visual stimuli. It found that semantic representations of visual words lasted from 450ms to 2020ms post-stimulus, which is highly coincided with the temporal windows of semantic representations in Experiment 1. These results indicate that the representational content of visual stimuli in the encoding stage is highly dynamic and the semantic representation is later than the visual representation. More importantly, semantic representations of different visual inputs follow a highly similar rule.

In the second study, we conduct another two experiments to examine the temporal dynamics of the representational content (Experiment 3), and the spatial transfer of brain regions (Experiment 4) from encoding to short-term memory retention respectively. In Experiment 3, we used intracranial EEG recordings, combining with representational similarity analysis and temporal generalization analysis, to investigate the representational content and its temporal dynamic during the short-term memory retention stage. The results found that during the short-term memory retention stage of natural visual objects, the representational content contained both high-level visual representations and abstract semantic representations that were originated from two encoding periods respectively. The visual representations from the early stage of encoding (250-770ms post-stimulus) and the abstract semantic representations from late-stage encoding (1000-1980ms post-stimulus). Greater abstract semantic representations were maintained as compared to the higher-order visual representations. Although the intensities of the two representational formats were overall stable during the retention period, they substantially fluctuate on a finer temporal scale. The fluctuation of memory reactivation is coupled with the phase of the hippocampal low-frequency activities. The results of experiment 3 suggest that short-term memory maintains both higher-order visual and abstract semantic information. The stable maintenance of both types of representational formats was achieved through intermittent neural activations that were phase-locked to the hippocampus. However, the uneven spatial distribution of intracranial EEG electrodes makes it difficult to systematically measuring the brain regions engaged during encoding and short-term memory retention. Thus, on the basis of Experiment 3, Experiment 4 used magnetoencephalogram (MEG) imaging to examine the spatial transfer of neural representations from encoding to retention. Multivariate decoding analysis in both the single-region and the single-channel level reveals that the neural representation of the encoding stage is mainly involved in the occipital lobe, the temporal lobe, and the parietal lobe, while strongest in the occipital lobe. The neural representations of short-term memory are distributed in various cortical areas, but strongest in the temporal and frontal lobes. These results indicate that from encoding to short-term memory retention, the core brain area of memory representations shifts from the posterior to anterior cortical regions.

The third study combines the tasks of encoding, retention, and retrieval. It focuses on the neural representational content of the memory retrieval stage and the relationship between the dynamic representation of the encoding stage and memory retrieval. The results show that successful memory retrieval contains abstract semantic representations rather than visual representations. We further investigated what kind of encoding representation is beneficial to the formation and retrieval of memory, and found that the stronger the dynamic change from the visual representation to the semantic representation in the encoding stage is associated with a better memory retrieval performance. The dynamic neural representations during the encoding stage promote subsequent memory retrieval via enhancing item-specific representations during the late encoding stage. Besides, memory retrieval reinstates the neural representations in the short-term memory retention stage rather than the encoding stage, which shows that the representational similarity between the retrieval and retention stages is significantly greater than the similarity between the retrieval and encoding stages. These results suggest that the neural representation of memory information undergoes multiple phases of transformation from encoding to short-term memory retention and then to retrieval.

All together, the dissertation systematically unveils the dynamic neural representations during memory formation and retrieval via the abovementioned three studies (5 experiments). First, the external visual input generates multiple neural representational formats from visual features to abstract semantics in the encoding stage through the dynamic interaction between prior knowledge and experience. Then, higher-order visual representations and abstract semantic representations can be stably maintained in short-term memory. Afterward, only abstract semantics representations are well reinstated in memory retrieval. Moreover, the greater transformation from vision to abstract semantic representations during encoding predicts better memory retrieval. These findings challenge the traditional pattern reinstatement model of episodic memory, providing scientific guidance for the development of effective learning and memory enhancement methods. Moreover, the results could potentially inspire the development of a new generation of artificial intelligence based on the dynamic neural mechanism of learning and memory in the human brain.