Motion Deblurring with Real Events

Introduction

Many event-based motion deblurring methods have been proposed by learning from synthesized dataset composed of simulated events and blurry images as well as sequences of sharp clear ground-truth images. However, the inconsistency between synthetic and real data degrades the performance of inference on real-world event cameras. The physical intrinsic noise of event cameras raises the difficulty of simulating labeled events that highly match the real event data. Even though the event simulator to some extent reduces the gap by considering the pixel-to-pixel variation in the event threshold, additional noise effects such as background activity noise and false negatives still exist, leading to tremendous discrepancy between the virtual events synthesized from event simulators and the real events emitted by event cameras. An alternative approach is to build a labeled dataset composed of real-world events accompanying with synthesized blurry images, and then train networks on it. Unfortunately, obtaining such pairs is not always easy, which needs to be captured with a slow motion speed as well as under good lighting conditions to avoid motion blur. Subsequent blurry image synthesis and alignment on temporal domain is also tedious but indispensable. Furthermore, inconsistency still exists between the events associated with synthesized and real-world motion blur in that limited read-out bandwidth leads to more event timing variations, as shown in Fig. 1.

Figure 1. Illustrative example of inconsistency between synthesized and real-world motion blurs with respect to the event time surface: (a) a real-world motion blurred image; (b) the timesurface of real-world events corresponding to (a); (c) the time surface of events collected with the same trajectory as (a) but at a slow motion speed.

In this paper, we propose a novel framework of learning the event-based motion deblurring network in a self-supervised manner, where real-world events and real-world motion blurred images are exploited to alleviate the performance degradation caused by data inconsistency and bridge the gap between simulations and real-world scenario. Fig. 2 illustrates the overall architecture of our framework.

Figure 2. Overview of the proposed learning framework for Real-world Event based motion Deblurring Network (RED-Net), where the blur-consistency and photometric-consistency provide self-supervised losses for real-world datasets and the reconstruction error provides the supervised loss for synthetic datasets.

Results

1. Results of Optical Flow

Figure 3. Optical flow output of OF-Net: (a) pre-trained on MVSEC (equivalent to EV-FlowNet), (b) RED-GoPro-LM, (c) RED-GoPro-PLM and (d) RED-RBE. The deblurred results (f), (g) and (h) of the motion blurred image (e) are respectively corresponding to (b), (c) and (d).

2. Comparisons with State-of-the-art Methods

Figure 4. Qualitative results of motion deblur by 7 different methods. For each scene, from top-left to bottom-right are respectively the blurry image and its deblurred result by blur2mflow, LEVS, EDI, eSL-Net, LEDVDI and our proposed RED-HQF and RED-RBE.

Table 1. Quantitative comparisons of proposed RED-Nets trained over different datasets to the state-of-the-arts. RED-GoPro, RED-HQF, and RED-RBE are respectively trained on GoPro, GoPro+HQF and GoPro+RBE. Note that LEDVDI only outputs 6 frames for sequence prediction, while the others output 7 frames.

3. Ablation Study

Figure 5. Qualitative ablation study with 5 blurry scenes by different RED-Nets. For each scene, from left to right are respectively the blurry image and its deblurred results by Deblur-Net-GoPro (SynEv), RED-GoPro-LM (SynEv+LM), RED-GoPro (SynEv+PLM), REDRBE-LM (SynEv+LM+ReEv) and RED-RBE (SynEv+PLM+ReEv).

Table 2. Ablation study of proposed framework w/o synthesized events (SynEv), blurring with LM (LM) , blurring with PLM (PLM) and real-world events (ReEv).