A hybrid model combining depthwise separable convolutions and vision transformers for traffic sign classification under challenging weather conditions

This research presents a novel deep-learning framework designed for traffic sign image classification under adverse conditions, including rain, shadows, haze, codec errors, and dirty lenses. To effectively balance accuracy and training parameters, the approach combines depthwise and pointwise convolutions, often referred to as depthwise separable convolutions, with a Vision Transformer (ViT) for subsequent feature extraction. The framework’s initial block comprises two pairs of depthwise and pointwise convolutional layers followed by a normalization layer. Depthwise convolution is responsible for processing each input channel independently and applying separate filters to each channel, thereby reducing computational cost and parameters while maintaining spatial structure. Pointwise convolutional layers combine information from different channels, fostering complex feature interactions and non-linearities. Batch normalization is used for training stability. At the end of the initial block, the max pooling layer is used to enhance and downsample spatial dimensions. The architecture repeats four times, preserving crucial information through skip connections. To extract global context information, inter-block skip connections and global average pooling (GAP) are employed for dimensionality reduction while retaining vital information. Integration of the ViT model in the final layers captures far-reaching dependencies and relations in the feature maps. The framework concludes with two fully connected layers, a bottleneck layer with 1024 neurons and a second layer using softmax activation to generate a probability distribution over 14 classes. The proposed framework, combining convolution blocks and skip connections with precisely tuned ViT hyperparameters, enhances model performance and achieves an exceptional validation accuracy of 99.3%.