对于关联信息的编码是记忆形成的基础。先前研究结果表明长时记忆神经系统所表征的概念（客体或词汇）之间关联强度可被主观评定或高阶文本统计模型（如LSA、Word2Vec）所解释。但经典动物文献和关联学习研究则暗示简单共现关联可以被编码在神经系统上。人类长时记忆神经系统是否存储了概念简单共现关联，是一个领域内未知的问题。

本论文考察了两个主要信息来源——视觉、语言/文本中的概念简单共现关联模式的认知心理意义和脑基础。在研究一，我们对视觉、语言模态概念简单共现关联信息的特性与功能进行研究，关心视觉、语言简单共现关联的概念组织模式，两者相关和差异，以及与已知语义关联知识的相关性。在研究二，我们对视觉、语言简单共现关联信息的大脑表征基础进行研究，通过在图片命名任务的磁共振数据上使用表征相似性分析方法，从脑区尺度（全脑探照灯分析、左侧颞叶前部、双侧外侧枕叶复合区）、网络尺度（语义加工网络、腹侧视觉通路）考察了两种简单共现关联信息的编码脑区。

在研究一，我们从图片数据库获取了视觉简单共现关联信息，从文本语料库中获取了语言简单共现关联信息。我们发现视觉简单共现关联对概念没有明显类别组织模式，与主观评定语义相关性有低度相关（r = 0.26）。语言简单共现对概念有明显类别组织模式，与主观评定语义相关性有中度相关（r = 0.56），与Word2Vec相似性有中度相关（r = 0.69）。此外，视觉、语言简单共现关联信息间存在低度相关（r = 0.38）。

在研究二，在大脑脑区尺度我们发现，视觉简单共现关联的脑基础位于双侧视觉皮层（外侧枕叶复合区、右侧枕中回）和对情境加工相关的脑区（双侧扣带回后部、右侧角回）等区域；语言简单共现关联的脑基础位于左侧颞叶前部、左侧颞上沟、双侧外侧枕叶复合区、左侧枕颞沟、双侧颞上回、中央前后回、右侧额中回、双侧丘脑腹内侧后部、内侧额叶等区域，上述效应不可被类别关系、形状相似性，及两者相互所解释。在网络尺度我们发现，视觉简单共现关联脑基础位于默认网络模块，语言共现关联脑基础位于外侧裂周边网络模块，以上效应不可被类别关系及两者相互解释。

       综上，本论文围绕视觉与语言模态概念简单共现关联对知识存储的意义与脑基础问题，展开了计算与认知、脑基础研究，发现了视觉、语言简单共现关联按多体素活动模式的方式编码在长时记忆多种不同神经系统上，与已知语义关联知识存在一定关系。该研究发现拓展了认知神经科学领域对长时记忆神经系统可编码的信息类型的已有认知。

The coding of association is the basis of memory formation. Previous results showed effects of “association strength” on abstract concepts (objects or words) based on either subjective ratings or high-order statistical language models (e.g.: LSA; Word2Vec) of various kinds. Classical animal literature and associative learning literature suggested simple co-occurrence can be coded by the neural system. Whether the human brain representation stores (in long term memory) such simple first order co-occurrence, is unknown.   

This thesis concerned the cognitive psychological meaning and neural basis of concepts’ associative pattern measured by simple co-occurrence in two sensory modalities (visual image and verbal text). Study 1 was aimed to understand the character and cognitive psychological meaning of visual and verbal co-occurrence patterns. Do they associate? Do they dissociate? Do they give clues about semantic structures (for comprehension)? Study 2 used representational similarity analysis (RSA) to test the brain regions that are sensitive to either type of co-occurrence on different scales: whole brain searchlight; region of interest; network of interest.

In Study 1, using a large dataset of labeled natural scenes and a large corpus of written text, we calculated the strength of association between 95 concepts. We found the association strength defined by visual co-occurrence had an acceptable correlation with subjective-rated semantic relatedness (r = 0.26). The association strength defined by verbal co-occurrence had some taxonomic categorization effect on its pattern, and had a medium correlation with subjective-rated semantic relatedness (r = 0.56), and with Word2Vec Similarity (r = 0.69). There was a medium correlation between the association strength measured by visual and verbal co-occurrence (r = 0.38).

In Study 2, we used representational similarity analysis (RSA) to find if the multi-voxel pattern of fMRI response during object viewing can represent two kinds of information. After controlling basic visual similarities of pictures, whole-brain searchlight RSA or region of interest RSA revealed that visual simple co-occurrence was represented in bilateral occipital cortex (bilateral lateral occipital complex, BA19, BA37, right inferior occipital gyrus) and experiential simulation regions (bilateral posterior cingulate cortex, right angular gyrus); while verbal simple co-occurrence in left ventrolateral anterior temporal lobe(vlATL), the anterior part of BA20, the anterior and middle portion of left superior temporal sulcus(STS), and other areas in word form network(WFN) such as left occipital temporal sulcus(OTS), bilateral superior temporal gyrus(STG), precentral gyrus, postcentral gyrus, posterior ventromedial thalamus, and right middle frontal gyrus. The significant effects could not be explained by taxonomic relation, shape similarity, or by each other. Network of interest RSA revealed that verbal co-occurrence was represented in left Perisylvian network (PSN) and visual co-occurrence in default mode network (DMN), which could not be explained by taxonomic relation or by each other.

    In conclusion, this thesis, concerning the question of whether the information of concept’s simple co-occurrence is important for human knowledge representation and where it is coded on human brain, investigated the character, neural basis, and relation with other association knowledge, of two kinds of sensory information (visual image and verbal text). Our studies found that both kinds of sensory information were coded on many different neural systems and had some correlation with known semantic association knowledge. These findings suggested the two kinds of sensory information were part of human knowledge, which expanded our understanding of human long-term memory system.