The concentrations of heavy metals (HMs) and the associated pollution risk and dietary exposure risk were assessed by collecting 154 soil samples and 6 food types from the Xianyushan area. The concentrations of Zn, Cd, and As in the soil were 1.2, 3.4, and 2.8 times the background values, and the percentages of samples exceeding the background value were 47%, 65%, and 96%, respectively. Concentrations of Cd, Cr, and As exhibited significant variations among the three land use types, which were potentially attributed to soil properties, agricultural amendments, and other contributing factors. Based on the integrated results from spatial distribution analysis, correlation analysis (CA), and principal component analysis (PCA), it can be inferred that these HMs mainly originated from soil parent materials. Pollution risk assessment indicated that Cd and As were the dominating soil pollutants. Although HM levels in different food types were lower than the limit defined in China’s national food safety standards, the total estimated daily intakes (EDIs) of Cd and As exceeded permissible tolerable daily intake (PTDI) values as well as reference doses (RfD). The target hazard quotient (THQ) values indicated that Cd, As, and Hg present lower risks from dietary intake compared to other HMs, especially, since cereals present a high risk of HM intake and need further attention.