To assess soil potentially toxic metal(loid) contamination in the karst area resulting from anthropogenic activity, we conducted a study on the potentially toxic metal(loid) contents of four carbonate rock weathering profiles in the karst area with artisanal zinc smelting in Northwestern Guizhou Province, China. The levels of As, Cr, Cu, and Hg in the artisanal zinc smelting slag do not contaminate the soil of the carbonate rock weathering profile. However, soil Cd, Pb, and Zn are contaminated within soil depths of 50 cm, 10 cm, and 50 cm, respectively, by the artisanal zinc smelting slag. The Cd, Pb, and Zn in the slag are the main polluted elements in the surrounding area, and their contaminated levels and depths are different in the various rock-type weathering profiles. The severe pollution level of As, Cd, Pb, and Zn in the soil near the lead-zinc ore mining site might be attributed to the lead-zinc mineralization of the bedrock during the weathering process. The ratio of Zn/Cd to Pb/Cd is an effective tool in identifying the sources of potentially toxic metal(loid)s in the soil of the artisanal zinc smelting area. In the future, it is essential to consider the contribution of bedrock weathering (i.e., the geological body mineralization influence) and anthropogenic activity when studying the soil potentially toxic metal(loid) contamination of the metal ore mining site.