Vision-Language Models for Architectural Style Classification and Explainable Design Critique: An Empirical Study on Building Styles

Architectural style recognition is a fine-grained visual problem in which categories often share massing, materials, window rhythms, and ornamental vocabularies. This study evaluates visual classifiers and a compact CLIP-style vision-language model on the 2025 BuildingStyles corpus, which contains 512 images distributed across 24 architectural styles. Five-fold stratified group cross-validation keeps subject-linked image variants in the same fold. Five visual systems are compared with four language configurations formed from class names, generic prompts, and three architecture-specific large-language-model captions per style. The proposed Kernel Caption Alignment (KCA) model maps a four-view HOG representation into an 88-dimensional text space and classifies images by cosine similarity to caption prototypes. An inner cross-validation procedure selects probability temperatures, visual ensemble membership, and the visual-language fusion weight. The strongest pooled top-1 result is 37.11% for the visual ensemble, followed by 36.91% for both MultiView-HOG-RBF and the caption-guided hybrid; the hybrid reaches 56.25% top-3 accuracy. KCA-Prompt-Ensemble obtains 34.38% top-1 accuracy, while providing substantially stronger calibration: expected calibration error is 2.86%, compared with 19.21% for the visual ensemble, and negative log-likelihood is lower (2.339 versus 2.461). The exact McNemar comparison between the hybrid and visual ensemble is not significant (p = 1.000). Ancient Egyptian and Chicago school are the best resolved styles, whereas Tudor Revival, International, and Beaux-Arts remain difficult. Confusion analysis, a cross-modal style map, caption evidence, and occlusion maps identify the formal similarities that drive errors and support an evidence-oriented design critique for individual buildings. The results establish a measured baseline for this compact corpus and clarify the complementary roles of visual discrimination, calibrated language alignment, and explanation.