CLIP and the City: Addressing the Artificial Encoding of Cities in Multimodal Foundation Deep Learning Models

Abstract

In this project, we propose and explore a computational pipeline to examine urban cultural landscapes through the lens of artificial intelligence, and for questioning modes of embedding culture in machine learning models. By employing machine learning models that extract features and textual properties from images, we aim to uncover the connections between a city's history, architecture, and urban development. The city of Rome serves as a significant case study for this research.
To achieve this objective, we feed 360° panoramic images into large vision-language models (e.g. OpenCLIP), to question how mainstream culture is expressed in these models. In this machine-triggered urban experiment, we investigate overlaps between history and machinic interpretation and whether relevant temporal correlations can be captured through generic street images only. Finally, by spatially analysing the captured data, we identify clusters and discontinuities in the urban layout aiming at visually depicting the interplay of forces behind its development.
As in a forensic exercise, the paper seeks to uncover the complex social and historical dynamics of urban environments, exploiting only contemporary images of their settings and a generic embedding of culture. It explores potential cultural biases embedded in machine learning models by comparing Rome - culturally relevant for the western world - with other cities around the world; leveraging innovative computational pipelines and globally covering datasets to provide a novel research line for urban studies.