Position within the page tree

Institute for Visualization and Interactive Systems
Team
Schmalfuss, Jenny

Jenny Schmalfuss

M. Sc.

Doctoral Researcher
Institute for Visualization and Interactive Systems (VIS)
Research Group Computer Vision

Contact

+49 711 685 88364
Email

Universitätsstraße 38
70569 Stuttgart
Deutschland
Room: 1.465

Office Hours

Important: From April to November 2024, I am on leave for an internship with NVIDIA's Autonomous Vehicle Perception Research Group in Santa Clara, CA, USA. Please send thesis or project supervision requests directly to Andrés Bruhn.

Subject

My research interests are at the intersection of computer vision and machine learning. Here, I research new ways to quantify and improve the robustness of motion estimation and specifically optical flow methods under adversarial attacks and distribution shifts. My goal is to enable robust motion estimation by understanding what influences the robustness of current methods.

No defense is better than a bad one against adversarial patches...
...because most defenses lower both quality and robustness if the attacks considers the defense. Our paper illuminates the empty promises of current patch-defenses for optical flow methods, and will appear at WACV 2024!

Weather perturbs optical flow. But can we turn it into an adversarial attack?
Test your optical flow methods under Distracting Downpour, which was accepted to ICCV 2023! Also, my recent presentation of the project at ICVSS 2023 won a Poster Award and our precursor work on Attacking motion estimation with adversarial snow won a Best Paper Award at the ECCV AROW Workshop 2022.

Are all optical flow networks equally vulnerable to adversarial attacks?
Check out our paper on evaluating the robustness of optical flow methods, which was an Oral at ECCV 2022!

Short Bio

I am a doctoral researcher at the Computer Vision Department of the University of Stuttgart, and a scholar of the International Max Planck Research School for Intelligent Systems (IMPRS-IS). I hold a MSc and BSc degree in Simulation Technology from the University of Stuttgart, where I focused on numerical mathematics, computer vision and machine learning. During my studies, I worked as research intern at the National University of Singapore and the University of Houston.

Find me on google scholar or research gate.

Talks: I gave a talk at the TrustML Young Scientist Seminars about "Challenges in Adversarial Attacks for Motion Estimation". You can find the video on YouTube.

Publications

2024
1. Scheurer, E., Schmalfuss, J., Lis, A., & Bruhn, A. (2024). Detection Defenses: An Empty Promise against Adversarial Patch Attacks on Optical Flow. In Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE/CVF. https://arxiv.org/abs/2310.17403
  - BibTeX
  - Link
  BibTeX
  @proceedings{scheurer2024detection, author = {Scheurer, Erik and Schmalfuss, Jenny and Lis, Alexander and Bruhn, Andrés}, journal = {Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, publisher = {IEEE/CVF}, title = {Detection Defenses: An Empty Promise against Adversarial Patch Attacks on Optical Flow}, url = {https://arxiv.org/abs/2310.17403}, year = 2024 }
  Link
  https://arxiv.org/abs/2310.17403
2023
1. Mehl, L., Jahedi, A., Schmalfuss, J., & Bruhn, A. (2023, January). M-FUSE: Multi-frame Fusion for Scene Flow Estimation. Proc. Winter Conference on Applications of Computer Vision (WACV). https://doi.org/10.48550/arXiv.2207.05704
  - BibTeX
  - Link
  BibTeX
  @inproceedings{Mehl2023_MFUSE, abstract = {Recently, neural network for scene flow estimation show impressive results on automotive data such as the KITTI benchmark. However, despite of using sophisticated rigidity assumptions and parametrizations, such networks are typically limited to only two frame pairs which does not allow them to exploit temporal information. In our paper we address this shortcoming by proposing a novel multi-frame approach that considers an additional preceding stereo pair. To this end, we proceed in two steps: Firstly, building upon the recent RAFT-3D approach, we develop an improved two-frame baseline by incorporating an advanced stereo method. Secondly, and even more importantly, exploiting the specific modeling concepts of RAFT-3D, we propose a U-Net architecture that performs a fusion of forward and backward flow estimates and hence allows to integrate temporal information on demand. Experiments on the KITTI benchmark do not only show that the advantages of the improved baseline and the temporal fusion approach complement each other, they also demonstrate that the computed scene flow is highly accurate. More precisely, our approach ranks second overall and first for the even more challenging foreground objects, in total outperforming the original RAFT-3D method by more than 16%. Code is available at https://github.com/cv-stuttgart/M-FUSE.}, author = {Mehl, Lukas and Jahedi, Azin and Schmalfuss, Jenny and Bruhn, Andr{\'e}s}, booktitle = {Proc. Winter Conference on Applications of Computer Vision (WACV)}, doi = {10.48550/arXiv.2207.05704}, month = {01}, preprinturl = {https://doi.org/10.48550/arXiv.2207.05704}, title = {M-FUSE: Multi-frame Fusion for Scene Flow Estimation}, url = {/brokenurl# https://doi.org/10.48550/arXiv.2207.05704}, year = 2023 }
  Link
  /brokenurl# https://doi.org/10.48550/arXiv.2207.05704
2. Mehl, L., Schmalfuss, J., Jahedi, A., Nalivayko, Y., & Bruhn, A. (2023). Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4981–4991. https://openaccess.thecvf.com/content/CVPR2023/html/Mehl_Spring_A_High-Resolution_High-Detail_Dataset_and_Benchmark_for_Scene_Flow_CVPR_2023_paper.html
  - BibTeX
  - Link
  BibTeX
  @inproceedings{Mehl_2023_CVPR, abstract = {While recent methods for motion and stereo estimation recover an unprecedented amount of details, such highly detailed structures are neither adequately reflected in the data of existing benchmarks nor their evaluation methodology. Hence, we introduce Spring -- a large, high-resolution, high-detail, computer-generated benchmark for scene flow, optical flow, and stereo. Based on rendered scenes from the open-source Blender movie "Spring", it provides photo-realistic HD datasets with state-of-the-art visual effects and ground truth training data. Furthermore, we provide a website to upload, analyze and compare results. Using a novel evaluation methodology based on a super-resolved UHD ground truth, our Spring benchmark can assess the quality of fine structures and provides further detailed performance statistics on different image regions. Regarding the number of ground truth frames, Spring is 60x larger than the only scene flow benchmark, KITTI 2015, and 15x larger than the well-established MPI Sintel optical flow benchmark. Initial results for recent methods on our benchmark show that estimating fine details is indeed challenging, as their accuracy leaves significant room for improvement. The Spring benchmark and the corresponding datasets are available at http://spring-benchmark.org.}, author = {Mehl, Lukas and Schmalfuss, Jenny and Jahedi, Azin and Nalivayko, Yaroslava and Bruhn, Andr\'es}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {06}, pages = {4981-4991}, title = {Spring: A High-Resolution High-Detail Dataset and Benchmark for Scene Flow, Optical Flow and Stereo}, url = {https://openaccess.thecvf.com/content/CVPR2023/html/Mehl_Spring_A_High-Resolution_High-Detail_Dataset_and_Benchmark_for_Scene_Flow_CVPR_2023_paper.html}, year = 2023 }
  Link
  https://openaccess.thecvf.com/content/CVPR2023/html/Mehl_Spring_A_High-Resolution_High-Detail_Dataset_and_Benchmark_for_Scene_Flow_CVPR_2023_paper.html
3. Schmalfuss, J., Mehl, L., & Bruhn, A. (2023). Distracting Downpour: Adversarial Weather Attacks for Motion Estimation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 10106–10116. https://openaccess.thecvf.com/content/ICCV2023/html/Schmalfuss_Distracting_Downpour_Adversarial_Weather_Attacks_for_Motion_Estimation_ICCV_2023_paper.html
  - BibTeX
  - Link
  BibTeX
  @inproceedings{Schmalfuss_2023_ICCV, abstract = {Current adversarial attacks on motion estimation, or optical flow, optimize small per-pixel perturbations, which are unlikely to appear in the real world. In contrast, adverse weather conditions constitute a much more realistic threat scenario. Hence, in this work, we present a novel attack on motion estimation that exploits adversarially optimized particles to mimic weather effects like snowflakes, rain streaks or fog clouds. At the core of our attack framework is a differentiable particle rendering system that integrates particles (i) consistently over multiple time steps (ii) into the 3D space (iii) with a photo-realistic appearance. Through optimization, we obtain adversarial weather that significantly impacts the motion estimation. Surprisingly, methods that previously showed good robustness towards small per-pixel perturbations are particularly vulnerable to adversarial weather. At the same time, augmenting the training with non-optimized weather increases a method's robustness towards weather effects and improves generalizability at almost no additional cost. Our code is available at https://github.com/cv-stuttgart/DistractingDownpour.}, author = {Schmalfuss, Jenny and Mehl, Lukas and Bruhn, Andr\'es}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {10}, pages = {10106-10116}, title = {Distracting Downpour: Adversarial Weather Attacks for Motion Estimation}, url = {https://openaccess.thecvf.com/content/ICCV2023/html/Schmalfuss_Distracting_Downpour_Adversarial_Weather_Attacks_for_Motion_Estimation_ICCV_2023_paper.html}, year = 2023 }
  Link
  https://openaccess.thecvf.com/content/ICCV2023/html/Schmalfuss_Distracting_Downpour_Adversarial_Weather_Attacks_for_Motion_Estimation_ICCV_2023_paper.html
4. Schmalfuss, J., Scheurer, E., Zhao, H., Karantzas, N., Bruhn, A., & Labate, D. (2023). Blind image inpainting with sparse directional filter dictionaries for lightweight CNNs. Journal of Mathematical Imaging and Vision (JMIV), 65, 323--339. https://doi.org/10.1007/s10851-022-01119-6
  - BibTeX
  - Link
  BibTeX
  @article{schmalfuss2022blind, abstract = {Blind inpainting algorithms based on deep learning architectures have shown a remarkable performance in recent years, typically outperforming model-based methods both in terms of image quality and run time. However, neural network strategies typically lack a theoretical explanation, which contrasts with the well-understood theory underlying model-based methods. In this work, we leverage the advantages of both approaches by integrating theoretically founded concepts from transform domain methods and sparse approximations into a CNN-based approach for blind image inpainting. To this end, we present a novel strategy to learn convolutional kernels that applies a specifically designed filter dictionary whose elements are linearly combined with trainable weights. Numerical experiments demonstrate the competitiveness of this approach. Our results show not only an improved inpainting quality compared to conventional CNNs but also significantly faster network convergence within a lightweight network design. Our code is available at https://github.com/cv-stuttgart/SDPF_Blind-Inpainting.}, author = {Schmalfuss, Jenny and Scheurer, Erik and Zhao, Heng and Karantzas, Nikolaos and Bruhn, Andres and Labate, Demetrio}, doi = {10.1007/s10851-022-01119-6}, file = {:PDFs/Blind Image Inpainting with Sparse Directional Filter Dictionaries for Lightweight CNNs.pdf:PDF;:PDFs/Blind Image Inpainting with Sparse Directional Filter Dictionaries for Lightweight CNNs_arxiv.pdf:PDF}, groups = {Inpainting, neural networks, Sparsity, blind inpainting, arXiv}, journal = {Journal of Mathematical Imaging and Vision (JMIV)}, pages = {323--339}, title = {Blind image inpainting with sparse directional filter dictionaries for lightweight {CNNs}}, url = {https://link.springer.com/article/10.1007/s10851-022-01119-6}, volume = 65, year = 2023 }
  Link
  https://link.springer.com/article/10.1007/s10851-022-01119-6
2022
1. Schmalfuss, J., Mehl, L., & Bruhn, A. (2022). Attacking Motion Estimation with Adversarial Snow. Proc. ECCV Workshop on Adversarial Robustness in the Real World (AROW). https://doi.org/10.48550/arXiv.2210.11242
  - BibTeX
  - Link
  BibTeX
  @inproceedings{schmalfuss2022attacking, abstract = {Current adversarial attacks for motion estimation (optical flow) optimize small per-pixel perturbations, which are unlikely to appear in the real world. In contrast, we exploit a real-world weather phenomenon for a novel attack with adversarially optimized snow. At the core of our attack is a differentiable renderer that consistently integrates photorealistic snowflakes with realistic motion into the 3D scene. Through optimization we obtain adversarial snow that significantly impacts the optical flow while being indistinguishable from ordinary snow. Surprisingly, the impact of our novel attack is largest on methods that previously showed a high robustness to small L_p perturbations. }, author = {Schmalfuss, Jenny and Mehl, Lukas and Bruhn, Andrés}, booktitle = {Proc. ECCV Workshop on Adversarial Robustness in the Real World (AROW)}, doi = {https://doi.org/10.48550/arXiv.2210.11242}, preprinturl = {https://arxiv.org/abs/2210.11242}, title = {Attacking Motion Estimation with Adversarial Snow}, url = {https://arxiv.org/abs/2210.11242}, year = 2022 }
  Link
  https://arxiv.org/abs/2210.11242
2. Schmalfuss, J., Scholze, P., & Bruhn, A. (2022). A perturbation-constrained adversarial attack for evaluating the robustness of optical flow. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Proc. European Conference on Computer Vision (ECCV) (Vol. 13682, pp. 183--200). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-20047-2_11
  - BibTeX
  - Link
  BibTeX
  @inproceedings{schmalfuss2022perturbation, author = {Schmalfuss, Jenny and Scholze, Philipp and Bruhn, Andrés}, booktitle = {Proc. European Conference on Computer Vision (ECCV)}, doi = {10.1007/978-3-031-20047-2_11}, editor = {Avidan, S. and Brostow, G. and Cissé, M. and Farinella, G.M. and Hassner, T.}, isbn = {978-3-031-20047-2}, pages = {183--200}, preprinturl = {https://arxiv.org/abs/2203.13214}, publisher = {Springer Nature Switzerland}, title = {A perturbation-constrained adversarial attack for evaluating the robustness of optical flow}, url = {https://link.springer.com/chapter/10.1007/978-3-031-20047-2_11}, volume = 13682, year = 2022 }
  Link
  https://link.springer.com/chapter/10.1007/978-3-031-20047-2_11
2021
1. Schmalfuss, J., Riethmüller, C., Altenbernd, M., Weishaupt, K., & Göddeke, D. (2021). Partitioned coupling vs. monolithic block-preconditioning approaches for solving Stokes-Darcy systems. Proc. International Conference on Computational Methods for Coupled Problems in Science and Engineering (COUPLED PROBLEMS). https://doi.org/10.23967/coupled.2021.043
  - BibTeX
  - Link
  BibTeX
  @inproceedings{schmalfuss2021partitioned, author = {Schmalfuss, Jenny and Riethmüller, Cedric and Altenbernd, Mirco and Weishaupt, Kilian and Göddeke, Dominik}, doi = {10.23967/coupled.2021.043}, journal = {Proc. International Conference on Computational Methods for Coupled Problems in Science and Engineering (COUPLED PROBLEMS)}, title = {Partitioned coupling vs. monolithic block-preconditioning approaches for solving {Stokes}-{Darcy} systems}, url = {https://www.scipedia.com/public/Schmalfus_et_al_2021a}, year = 2021 }
  Link
  https://www.scipedia.com/public/Schmalfus_et_al_2021a
2016
1. Wanigasekara, N., Schmalfuss, J., Carlson, D., & Rosenblum, D. S. (2016). A bandit approach for intelligent IoT service composition across heterogeneous smart spaces. Proc. International Conference on the Internet of Things, 121–129. https://doi.org/10.1145/2991561.2991562
  - BibTeX
  - Link
  BibTeX
  @inproceedings{wanigasekara2016bandit, author = {Wanigasekara, Nirandika and Schmalfuss, Jenny and Carlson, Darren and Rosenblum, David S.}, booktitle = {Proc. International Conference on the Internet of Things}, doi = {https://doi.org/10.1145/2991561.2991562}, pages = {121-129}, publisher = {ACM}, title = {A bandit approach for intelligent {IoT} service composition across heterogeneous smart spaces}, url = {https://dl.acm.org/doi/pdf/10.1145/2991561.2991562}, year = 2016 }
  Link
  https://dl.acm.org/doi/pdf/10.1145/2991561.2991562

Teaching

Winter 2023/2024:

Computer Vision and Intelligent Systems - Colloquium (Organization)

Summer 2023:

Recent Advances in Computer Vision - Seminar (Supervision)
Computer Vision and Intelligent Systems - Colloquium (Organization)

Winter 2022/2023:

Computer Vision and Intelligent Systems - Colloquium (Organization)

Summer 2022:

Recent Advances in Computer Vision - Seminar (Supervision)
Computer Vision and Intelligent Systems - Colloquium (Organization)

Winter 2021/2022:

Computer Vision - Tutorial (Coordinator)
Computer Vision and Intelligent Systems - Colloquium (Organization)

Summer 2021:

Imaging Science - Tutorial (Coordinator)
Recent Advances in Computer Vision - Seminar (Supervision)
Computer Vision and Intelligent Systems - Colloquium (Organization)

Winter 2020/2021:

Computer Vision - Tutorial (Coordinator)
Bildverarbeitung und Computer Vision - Seminar (Supervision)
Computer Vision and Intelligent Systems - Colloquium (Organization)

Summer 2020:

Imaging Science - Tutorial (Coordinator)

Before Summer Term 2020:

Imaging Science - Tutorial (Tutor)
Computer Vision - Tutorial (Tutor)

Previous Projects

"Preconditioning Techniques for Coupled Stokes Darcy Systems" (Master's thesis, 2019/2020)

"Learning to Transform: Neural Networks with Sparse Directional Filters for Blind Inpainting" (research project in cooperation with the University of Houston, 2018/2019)

"Detection of Alternative Splicing in Coral Reef Fish Exposed to Climate Change" (research project with the King Abdullah University of Science and Technology, 2017)

"Simulation and Adaptation of Contextual Bandit Algorithms for IoT Service Discovery" (Bachelor's thesis in cooperation with the National University of Singapore, 2016)

Thesis and Project Supervision

"Differentiable illumination for naturalistic particle attacks on motion estimation". Bachelor’s Thesis. University of Stuttgart, 2023.

"A feature-level analysis of the adversarial robustness of RAFT". Work in progress. Master’s Thesis. University of Stuttgart, 2023.

"Assessing the robustness of optical flow algorithms: A comparative analysis of adversarial attacks and training". Research Project. University of Stuttgart, 2023.

"Attacks on defended optical flow networks for action recognition". Research Project. University of Stuttgart, 2023.

"A Global Adversarial Attack for Scene Flow". Master’s Thesis. University of Stuttgart, 2023.

"Filter dictionaries for optical flow prediction with RAFT". Master’s Thesis. University of Stuttgart, 2023.

"Sparse filters for optical flow robustness". Research Project. University of Stuttgart, 2022.

"An optimization approach for attacking the Horn and Schunck model". Bachelor’s Thesis. University of Stuttgart, 2022.

"Differentiating the Variational Horn and Schunck Method with Implicit Functions". Bachelor’s Thesis. University of Stuttgart, 2022.

"Atacking a defended optical flow network". Master’s Thesis. University of Stuttgart, 2022.

"A framework for attacking optical flow networks". Research Project. University of Stuttgart, 2022.

"Switching sparsity: investigating a training time adaptive layer". Research Project. University of Stuttgart, 2021.

"Stability of neural network architectures for optical flow estimation under adversarial attacks". Awarded Best SimTech Bachelor's thesis 2021. Bachelor’s Thesis. University of Stuttgart, 2021.

"Investigating the influence of learning rates on the learning speed of neural networks". Bachelor’s Thesis. University of Stuttgart, 2021.

"Analysing the quality of interframe interpolation with optical flow". Bachelor’s Thesis. University of Stuttgart, 2021.

"Improved RAFT architectures for optical flow estimation". Master’s Thesis. University of Stuttgart, 2021.

Older News and Announcements

Internship: I will join NVIDIA's Autonomous Vechicle Perception Research Group in Santa Clara, CA, USA, for an internship in 2024.

News: I will be visiting the Data and Web Science Group at University of Mannheim for a talk on March 05, 2024.

News: I will be at WACV'24 from January 04-08 at Waikoloa Beach, Hawaii.

News: I will be at ICCV'23 from October 02-06 in Paris.

News: I will be at ICVSS'23 from July 09-15 in Sicily.

Meet Spring: The new benchmark for scene flow, optical flow and stereo!
Test our dataset at spring-benchmark.org, and read the preprint for our CVPR 2023 paper.

TrustML YSS Talk: I gave a talk at the TrustML Young Scientist Seminars about "Challenges in Adversarial Attacks for Motion Estimation". You can find the video on YouTube.

News: I will be visiting the Institute of Computer Graphics and Vision at TU Graz for a talk from December 19-20, 2022.

News: I will be visiting the Robust Machine Learning Group at MPI-IS Tübingen for a talk on November 17, 2022.

Snowy conditions can break your optical flow method.
Our work on Attacking motion estimation with adversarial snow won a Best Paper Award at the ECCV AROW Workshop 2022! More info here.

News: I will be at ECCV'22 from October 23-27 in Tel Aviv.

Are all optical flow networks equally vulnerable to adversarial attacks?
Check out our paper on evaluating the robustness of optical flow methods, which was an Oral at ECCV 2022!

Team page

Jenny Schmalfuss

Contact

Office Hours

Subject

Short Bio

Publications

2024

BibTeX

Link

2023

BibTeX

Link

BibTeX

Link

BibTeX

Link

BibTeX

Link

2022

BibTeX

Link

BibTeX

Link

2021

BibTeX

Link

2016

BibTeX

Link

Teaching

Previous Projects

Thesis and Project Supervision

Older News and Announcements

Audience

Formalities

Services

Organization

Jenny Schmalfuss

Contact

Office Hours

Subject

Short Bio

Publications

2024

BibTeX

Link

2023

BibTeX

Link

BibTeX

Link

BibTeX

Link

BibTeX

Link

2022

BibTeX

Link

BibTeX

Link

2021

BibTeX

Link

2016

BibTeX

Link

Teaching

Previous Projects

Thesis and Project Supervision

Older News and Announcements

Here you can reach us

Audience

Formalities

Services

Organization