In this study, the microbial community of the anode-associated soil in sediment microbial fuel cells (MFCs) was investigated. Sediments from the drainage ditch (D) and rice field (RF) were treated with or without autoclaving for single-chamber and mediator-free MFCs. Without sterilization, the voltage output reached a maximum of 124 and 120 mV for the sediments from D and RF on days 30 and 56, respectively. By day 40, the voltage output of the MFCs with sterilization became positive and continued to increase. The microbial communities of the different sediments were clustered together, indicating that the voltage release correlated positively with the presence of soil microbes. Spearman rank correlation analysis showed that the genus Geobacter was positively correlated with Defluviicoccus, Desulfatiglans, Sulfurimonas, Synthrophobacter, Thiobacillus, and Thermodesulfovibrio and negatively correlated with Holophaga, Opitutus, Paludibacter, and Pseudomonas. The genera Geothrix showed the dominant changes in the sediments with sterilization. These results indicate that the sediment MFC device could change the microbial composition adapted to the anodic environment.