Skip to main content

An AI assistant to interact with remote sensing images

Running

Running

Organisational Unit
Implementation progress
40%
28 January 2021

Duration: 36 months

Objective

We want to make remote sensing images content accessible to everyone. We will create an ‘AI assistant’ that interprets questions formulated in English about the content of the images and predict the answer sought by the user (see picture). With a single model, it will answer questions about land changes in time series, objects presence or counting. Besides a new way to interact with ESA products, it is a crucial step towards a Digital Twin of the Earth, where the globe can be queried for information-on-demand through satellite imaging. Such an AI assistant does not exist yet. To create it, one needs to develop ground breaking technology using both computer vision and remote sensing – to extract visual information from images - and natural language processing (NLP) – to obtain semantic information from the question. The two sources need to be fused to extract the information required, a process known in vision as Visual Question Answering [Antol, ICCV 2015]. The only attempt towards a VQA system for remote sensing has been recently proposed by my team [Lobry, TGRS 2020]. With this project, we want to explore the possibility to handle spatio-temporal image queries, therefore usable to extract information about change and dynamics. Decision makers and journalists, for example, could be users of such system. The technical challenges are - Create a multi-temporal datasets with Sentinel-2 pairs - Generate automatically a ground truth database of image-question-answers triplets following the CLEVR protocol (Johnson, CVPR 2017) - Develop a full end-to-end model to process the triplets, based on latest advances in NLP (e.g. self attention [Xu, ICML 2015]) and multitemporal remote sensing [Daudt, ICIP 2018] -  Explore program inference for logical processing of the questions [Johnson, ICCV 2017] The project (EPFL PhD thesis) will lead to a first demonstrator of such AI assistant, addressing questions about dynamics in the Swiss Alps, e.g. glaciers movements and land use changes.

Contract number
4000133903
Programme
OSIP Idea Id
I-2020-03632
Related OSIP Campaign
Open Channel
Main application area
EO
Budget
90000€
An AI assistant to interact with remote sensing images