Lecture 14
Multimodal and vision-language models; synthesis of the course
Prof. dr. Daniel Traian PELE
Bucharest University of Economic Studies (ASE), Department of Statistics and Econometrics
Romanian Academy, Institute for Economic Forecasting (IPE)
Business data are not only tables. Banks and insurers look at satellite images of the land they finance, read scanned contracts and annual reports, and analysts read price charts. Multimodal models put images and text into one representation: contrastive language–image pre-training (CLIP) learns from image–caption pairs, and vision-language models (VLM) let a large language model read an image. This last lecture measures what these models can and cannot do on real data, and then draws the lessons of the whole course.
We classify 5,400 test images of the EuroSAT data set (Sentinel-2 satellite images of land use in Europe) with CLIP without any labelled example, compare it with a convolutional neural network (CNN) trained from scratch as the number of labels grows, search images with words, and ask a 2-billion-parameter VLM to read market charts and yield-curve tables. The market-stress experiment follows the multimodal anomaly-detection framework of our group, with open models.