Informationen für
Logo VIS

Stuttgart Artificial Background Subtraction Dataset

The SABS (Stuttgart Artificial Background Subtraction) dataset is an artificial dataset for pixel-wise evaluation of background models. The use of artificial data enables us to separably judge the performance of background subtraction methods for each of the challenges background subtraction has to cope with. Realistic video footage was created using recent raytracing technology with global illumination (Autodesk Maya's Mental Ray). Sensor noise was simulated using additive Gaussian noise. In contrast to manually annotated ground-truth data, the SABS dataset does not suffer from imperfect labels or only a small number of annotated frames.

 

Ground-truth annotation Frame of artificial video footage Shadow annotation

 

The dataset consists of video sequences for nine different challenges of background subtraction for video surveillance. These sequences are further split into training and test data. For every frame of each test sequence ground-truth annotation is provided as color-coded foreground masks. This way, several foreground objects can be distinguished and the ground-truth annotation could also be used for tracking evaluation. The dataset contains additional shadow annotation that represents for each pixel the absolute luminance distance between the frame with and without foreground objects.

 

More details about the dataset and the evaluation process can be found in the publication or the poster which we presented at CVPR 2011. The experimental setup for the evaluation of background models is described in this document.

The dataset is encoded frame-by-frame as PNG (Portable Network Graphics) images. For download, it was split and compressed to several RAR archives.

If you publishing research based on the SABS dataset, please acknowledge the dataset and include a citation to the publication listed below.

 

Download links of the SABS dataset

Part Archive Size
Ground-truth & shadow masks SABS-GT.rar
83MB
Basic, Camouflage & Darkening
SABS-Basic.rar
3.16GB
Bootstrap
SABS-Bootstrap.rar 1.12GB
Light Switch
SABS-LightSwitch.rar 1.02GB
Noisy Night
SABS-NoisyNight.rar 1.43GB
MPEG4 40kbps
SABS-MPEG4_40kbps.rar 760MB
MPEG4 80kbps
SABS-MPEG4_80kbps.rar 852MB
MPEG4 160kbps
SABS-MPEG4_160kbps.rar 947MB
MPEG4 320kbps
SABS-MPEG4_320kbps.rar 1.01GB
MPEG4 640kbps
SABS-MPEG4_640kbps.rar 793MB
Complete dataset
SABS-complete.rar 11.4GB

 

Download links of the SABS Matlab evaluation framework

Description File Size
Basic foreground mask evaluation EvaluateForegroundMasks.m 18KB

 

 

More of the MATLAB evaluation framework will be available soon.

 

If you like to be informed of updates of the SABS dataset or the MATLAB evaluation framework, you may subscribe to the SABS mailing list.

If you have any questions or comments don't hestitate to contact me: benjamin.hoeferlin@vis.uni-stuttgart.de

Publication

2011

Evaluation of Background Subtraction Techniques for Video Surveillance
Brutzer, Sebastian; Höferlin, Benjamin; Heidemann, Gunther: Evaluation of Background Subtraction Techniques for Video Surveillance. In: Computer Vision and Pattern Recognition (CVPR), S. 1937-1944, 2011.
[XPS] [PDF] [DOI] [OpenXML] [BibTeX] [Vortragsfolien] [Details]

Challenges of Background Subtraction

The following typical challenges of background subtraction in the context of video surveillance have been addressed in the evaluation:

  • Gradual illumination changes: It is desirable that background model adapts to gradual changes of the appearance of the environment. For example in outdoor settings, the light intensity typically varies during day.
  • Sudden illumination changes: Sudden once-off changes are not covered by the background model. They occur for example with sudden switch of light, strongly affect the appearance of background, and cause false positive detections.
  • Dynamic background: Some parts of the scenery may contain movement, but should be regarded as background, according to their relevance. Such movement can be periodical or irregular (e.g., traffic lights, waving trees).
  • Camouflage: Intentionally or not, some objects may poorly differ from the appearance of background, making correct classification difficult. This is especially important in surveillance applications.
  • Shadows: Shadows cast by foreground objects often complicate further processing steps subsequent to background subtraction. Overlapping shadows of foreground regions for example hinder their separation and classification. Hence, it is preferable to ignore these irrelevant regions.
  • Bootstrapping: If initialization data which is free from foreground objects is not available, the background model has to be initialized using a bootstrapping strategy.
  • Video noise: Video signal is generally superimposed by noise. Background subtraction approaches for video surveillance have to cope with such degraded signals affected by different types of noise, such as sensor noise or compression artifacts.

Evaluation Results

 

Summary

Maximal F-Measures (averaged over sequence) for all methods we evaluated, related to the experiments we conducted. Best result for each experiment in boldface. Note, that we could not evaluate the algorithm of Kim for the sequences at night due to its huge memory requirements. As well, Oliver was not evaluated for Bootstrap, since it does not support model updates.

Method Basic Dynamic Background Bootstrap Darkening Light Switch Noisy Night Camouflage No Camouflage H.264 (40kbps)
McFarlane 0.614 0.482 0.541 0.496 0.211 0.203 0.738 0.785 0.639
Stauffer 0.800 0.704 0.642 0.404 0.217 0.194 0.802 0.826 0.761
Oliver 0.635 0.552 - 0.300 0.198 0.213 0.802 0.824 0.669
McKenna 0.522 0.415 0.301 0.484 0.306 0.098 0.624 0.656 0.492
Li 0.766 0.641 0.678 0.704 0.316 0.047 0.768 0.803 0.773
Kim 0.582 0.341 0.318 0.342 - - 0.776 0.801 0.551
Zivkovic 0.768 0.704 0.632 0.620 0.300 0.321 0.820 0.829 0.748
Maddalena 0.766 0.715 0.495 0.663 0.213 0.263 0.793 0.811 0.772
Barnich 0.761 0.711 0.685 0.678 0.268 0.271 0.741 0.799 0.774

Precision-Recall Charts

Precision-recall charts of the performance of BS methods with varying threshold.







Select methods and experiments.

F-Measure Comparision

Comparision of the maximal (average) F-Score of two different Background Subtraction models.










Select methods and experiments.

Foreground Masks

Foreground masks for visual inspection.


Select methods.
Select methods.
Select methods.
Select methods.