1995年，实验上成功地在稀薄的碱金属原子气体中实现了玻色-爱因斯坦凝聚(BEC)，极低温情况下，玻色系统中的原子发生BEC使得人们首次能够在宏观尺度上观察到量子现象，于是人们把目光都聚焦到能够成功实现BEC的具有弱相互作用的近理想玻色气体系统的研究上来。把近理想玻色气体系统放入光晶格中形成格点模型可以用来模拟凝聚态物理学的多体问题，研究已经表明，在极低温情况下，格点模型中原子之间的排斥相互作用和原子在格点之间的跳跃能相互竞争，使得系统发生超流-绝缘相变。绝缘相中，平均每个光格子有整数个原子填充，没有确定的相位，并且激发是有能隙的；而当格点系统中存在非整数填充时，将出现超流相，该相中原子有很好的长程相干性，激发是没有能隙的。玻色-哈伯德模型是一个简单的近似模型，可以很好地模拟和研究光晶格中有限密度情况下，有相互作用的冷原子玻色系统。通常在具有相互作用的玻色气体系统的研究过程中，考虑原子之间的相互作用时考虑的都是有效的两体相互作用，不涉及三体或多体相互作用项。但是，随着拓扑相、自旋液体等奇异量子相的研究发展，我们看到这些奇异的量子相与系统的多体相互作用有着密切的关系，单纯考虑两体相互作用已经不能成功解释这些奇异量子相的性质了。因此，在我们的论文中，我们除了考虑两体相互作用以外还考虑了三体相互作用，希望借以了解和研究多体相互作用在多体问题中的作用和影响。首先，我们在玻色-哈伯德模型中加入了三体排斥相互作用项，并用平均场解耦方法和虚时路径积分这两种方法分别得到了具有两体排斥和三体排斥相互作用玻色系统的相图。在相图的研究分析中，我们发现考虑了三体排斥相互作用以后，系统仍然存在超流-绝缘相变，但是除此以外，我们发现了系统的相变过程有一些新的规律和结果：在三维的超流-绝缘相变相图中，随着平均粒子占据数g的增加，临界曲面绕着某个固定点附近沿逆时针方向旋转；同时，随着g的增加，三体排斥相互作用逐渐超越两体排斥相互作用对超流-绝缘相变的发生起了主导作用等。我们知道，粒子之间的相互作用可以用散射长度来表示，通常情况下，排斥相互作用对应正的散射长度，但是实验中已经可以利用磁场调控玻色系统中的散射长度由正变到负，使玻色系统中原子间的相互作用由排斥变成吸引的情况。因此，我们在论文中考虑了两体排斥、三体吸引相互作用，两体吸引、三体排斥相互作用这两种情况下的玻色-哈伯德模型，同样由平均场理论和虚时路径积分方法可以得到这两种情况下的相图，从相图中，我们发现这两种情况下，在排斥、吸引相互作用以及跳跃项的相互竞争下，系统不但会发生超流-绝缘相变，还会出现超流-不稳定相的相变。考虑吸引相互作用的玻色-哈伯德模型中出现了超出我们理论适用范围的不稳定相，虽然我们对这些不稳定相到现在为止还没有明确的结果，但我们猜想这些不稳定相中存在丰富的物理和相互作用机制，对这些不稳定相进行进一步的研究将有助于我们了解许多未知的现象过程，如奇异量子相，白矮星、中子星的塌缩等。

Bose-Einstein condensates in dilute atomic gases, which were first realized experimentally in 1995 provide unique opportunities for exploring quantum phenomena on a macroscopic scale and attract much attention to the study of Bose condensates with weak interaction between atoms.A weak-interacting BEC in an optical lattice at zero temperature can be used to simulate many-body system in physics. It has been revealed that the behavior of BEC in optical lattices can be well described by the Bose-Hubbard model, which predicts a superfluid-Insulator transition due to the competition of two-body repulsive interaction and the tunneling amplitude between lattices.For Mott-insulator phase, it has the same integer occupation number at each lattice site without long-range correlation. While for superfluid phase, it has non-integer fillings and a well-definedglobal order parameter. What's more, the spectrum has a gap in the insulating phase and is gapless in the superfluid phase.When we study various phenomena in condensed-matter physics, we usually build up models involving effective two-body interactions. However, exotic quantum phases, such as topological phases or spin liquids, are often identified as ground states of hamiltonians with three-body or more-body terms. Thus, three-body interaction is taken into account in my paper in order to explore more phases in many-body physics.At first, three-body repulsive interaction term is added to the original Hamiltonian of Bose-Hubbard model and two different mean-field approaches with the aim of obtaining the phase diagram of an ultracold atomic Bose gas in an optical lattice are adopted. With the analysis of the phase diagram, we find some new properties except that there still exist a quantum phase transition from superfluid to Mott-insulator. Up to now, an atomic Bose gas with repulsive interactions has been studied a lot, moreover, experiments with attractive atomic gas have also been performed recently. As a result, with the aim of exploring the physics of attractive interaction, we consider the situation of two-body repulsive and three-body attractive interaction as well as the situation of two-body attractive and three-body repulsive interaction respectively. With the corresponding phase diagram in these two situations, we obtain the conclusion that the physics with attractive interaction will undergo the quantum phase transition not only from superfluid to Mott-insulator but also from superfluid to unstable phases. Although the unstable phases is beyond the bounds of our theory, but we guess the physics here is somewhat analogues to the physics of white dwarfs and neutron stars which, due to the gravitational attractive, also become unstable and collapse if the total mass of the compact object becomes too large. Further exploration of this unstable phase might bring rich physics and help people better understand more unknown exotic phenomena.