Automated structural defect detection is crucial for ensuring the safety and maintenance of civil infrastructure, particularly in bridges, where defects such as cracks, spalling, and corrosion can compromise structural integrity and pose safety risks. This research presents a comparative study of three semantic segmentation models—U-Net, Feature Pyramid Network (FPN), and DeepLabv3+—for detecting and classifying structural defects in bridge imagery. Each model was evaluated using two different encoder architectures, EfficientNet B3 and MobileOne S4, to assess the impact of varying feature extraction strategies on segmentation accuracy. The experiments were conducted using the DACL benchmark dataset, which includes a diverse range of defect categories, providing a comprehensive testing ground for evaluating model performance. Among the tested combinations, FPN paired with EfficientNet B3 demonstrated the highest mean accuracy across most defect categories, including common defects such as cracks and graffiti, making it the most effective approach overall. However, challenges were encountered in detecting certain more complex defect types, such as hollow areas and cavities, where all models showed decreased performance. The results of this study emphasize the potential of deep learning models in automating defect detection for infrastructure monitoring while also identifying key areas for improvement. Specifically, further refinements in feature extraction techniques and model tuning are needed to enhance detection accuracy in more complex defect scenarios. This research contributes to the ongoing development of AI-powered solutions for infrastructure inspection and provides insights for future improvements in automated structural health monitoring.