Supervisor

Suitable for

Abstract

Object detection is a fundamental task in computer vision. The goal is to locate and classify individual instances of objects in images (e.g., people, cars, cups, sheep, etc.). Most current models have been trained on benchmark datasets that consist of hand-annotated images collected from the internet. This introduces bias in the training data which in turn has been shown to lead to biased models.

In this project, we will make use of modern image editing techniques, such as in-painting diffusion models, to modify images in specific ways (e.g., changing the apparent age of people, skin-color, day-night, …), which allows us to measure the impact of the edit on the object detection performance.

Goals:

• Setup an evaluation framework of object detection models that allows modification of the test data.

• Use different image editing techniques to edit the test data.

• Test several hypotheses for bias in multiple different models.

Stretch Goal:

• Bias can potentially mitigated by including the modified data during training of the model.

References:

Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models. 2022 IEEE." CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2021.

Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. "Instructpix2pix: Learning to follow image editing instructions." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Lin, Tsung-Yi, et al. "Microsoft coco: Common objects in context." Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer International Publishing, 2014.

Singh, Krishna Kumar, et al. "Don't judge an object by its context: Learning to overcome contextual bias." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Pre-requisites: Machine Learning