Multimodal Information Fusion for the Prediction of the Condition of Condominiums

Today's data analysis techniques allow for the combination of multiple different data modalities, which should also allow for more accurate feature extraction. In our research, we leverage the capacity of machine learning tools to build a model with shared neural network layers and multiple inputs that is more flexible and allows for more robust extraction of real estate attributes. The most common form of data for a real estate assessment is data structured in tables, such as size or year of construction, but also descriptions of the real estate. Other data that can also be easily found in real estate listings are visual data such as exterior and interior photographs. In the presented approach, we fuse textual information and variable quantity of interior photographs per condominium for condition assessment and investigate how multiple modalities can be efficiently combined using deep learning. We train and test the performance of a pre-trained convolutional neural network fine-tuned with variable quantity of interior views of selected condominiums. In parallel, we train and test the pre-trained bidirectional encoder-transformer language model using text data from the same observations. Finally, we build an experimental neural network model using both modalities for the same task and compare the performance with the models trained with a single modality. Our initial assumption that coupling both networks would lead to worse performance compared to fine-tuned single-modal models was not confirmed, as we achieved the better performance with the proposed multi-modal model despite the impairment of a very unbalanced dataset. The novelty here is the multimodal modeling of variable quantity of real estate-related attributes in a unified model that integrates all available modalities and can thus use their complementary information. With the presented approach, we intend to extend the existing information extraction methods for automated valuation models, which in turn would contribute to a higher transparency of valuation procedures and thus to more reliable statements about the value of real estate.