This article is part of the series Super-Resolution Enhancement of Digital Video.

Open Access Research Article

A MAP Estimator for Simultaneous Superresolution and Detector Nonunifomity Correction

Russell C Hardie1* and Douglas R Droege2

Author Affiliations

1 Department of Electrical and Computer Engineering, University of Dayton, 300 College Park, Dayton, OH 45469-0226, USA

2 L-3 Communications Cincinnati Electronics, 7500 Innovation Way, Mason, OH 45040, USA

For all author emails, please log on.

EURASIP Journal on Advances in Signal Processing 2007, 2007:089354  doi:10.1155/2007/89354


The electronic version of this article is the complete one and can be found online at: http://asp.eurasipjournals.com/content/2007/1/089354


Received:31 August 2006
Accepted:9 April 2007
Published:3 May 2007

© 2007 Hardie and Droege

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

During digital video acquisition, imagery may be degraded by a number of phenomena including undersampling, blur, and noise. Many systems, particularly those containing infrared focal plane array (FPA) sensors, are also subject to detector nonuniformity. Nonuniformity, or fixed pattern noise, results from nonuniform responsivity of the photodetectors that make up the FPA. Here we propose a maximum a posteriori (MAP) estimation framework for simultaneously addressing undersampling, linear blur, additive noise, and bias nonuniformity. In particular, we jointly estimate a superresolution (SR) image and detector bias nonuniformity parameters from a sequence of observed frames. This algorithm can be applied to video in a variety of ways including using a moving temporal window of frames to process successive groups of frames. By combining SR and nonuniformity correction (NUC) in this fashion, we demonstrate that superior results are possible compared with the more conventional approach of performing scene-based NUC followed by independent SR. The proposed MAP algorithm can be applied with or without SR, depending on the application and computational resources available. Even without SR, we believe that the proposed algorithm represents a novel and promising scene-based NUC technique. We present a number of experimental results to demonstrate the efficacy of the proposed algorithm. These include simulated imagery for quantitative analysis and real infrared video for qualitative analysis.

References

  1. AF Milton, FR Barone, MR Kruer, Influence of nonuniformity on infrared focal plane array performance. Optical Engineering 24(5), 855–862 (1985)

  2. W Gross, T Hierl, M Schultz, Correctability and long-term stability of infrared focal plane arrays. Optical Engineering 38(5), 862–869 (1999). Publisher Full Text OpenURL

  3. DL Perry, EL Dereniak, Linear theory of nouniformity correction in infrared staring sensors. Optical Engineering 32(8), 1854–1859 (1993). Publisher Full Text OpenURL

  4. MD Nelson, JF Johnson, TS Lomheim, General noise processes in hybrid infrared focal plane arrays. Optical Engineering 30(11), 1682–1700 (1991). Publisher Full Text OpenURL

  5. A El Gamal, H Eltoukhy, CMOS image sensors. IEEE Circuits and Devices Magazine 21(3), 6–20 (2005). Publisher Full Text OpenURL

  6. PM Narendra, NA Foss, Shutterless fixed pattern noise correction for infrared imaging arrays. Technical Issues in Focal Plane Development, April 1981, Washington, DC, USA, Proceedings of SPIE 282, 44–51

  7. JG Harris, Continuous-time calibration of VLSI sensors for gain and offset variations. in Smart Focal Plane Arrays and Focal Plane Array Testing, April 1995, Orlando, Fla, USA, Proceedings of SPIE, vol. 2474, , ed. by Wigdor M, Massie MA, pp. 23–33

  8. JG Harris, Y-M Chiang, Nonuniformity correction using the constant-statistics constraint: analog and digital implementations. in Infrared Technology and Applications XXIII, April 1997, Orlando, Fla, USA, Proceedings of SPIE, vol. 3061, , ed. by Andresen BF, Strojnik M, pp. 895–905

  9. Y-M Chiang, JG Harris, An analog integrated circuit for continuous-time gain and offset calibration of sensor arrays. Analog Integrated Circuits and Signal Processing 12(3), 231–238 (1997). Publisher Full Text OpenURL

  10. DA Scribner, KA Sarkady, JT Caulfield, et al. Nonuniformity correction for staring IR focal plane arrays using scene-based techniques. in Infrared Detectors and Focal Plane Arrays, April 1990, Orlando, Fla, USA, Proceedings of SPIE, vol. 1308, , ed. by Dereniak EL, Sampson RE, pp. 224–233

  11. DA Scribner, KA Sarkady, MR Kruer, JT Caulfield, JD Hunt, C Herman, Adaptive nonuniformity correction for IR focal-plane arrays using neural networks. in Infrared Sensors: Detectors, Electronics, and Signal Processing, July 1991, San Diego, Calif, USA, Proceedings of SPIE, vol. 1541, , ed. by Jayadev TS, pp. 100–109

  12. DA Scribner, KA Sarkady, MR Kruer, et al. Adaptive retina-like preprocessing for imaging detector arrays. Proceedings of IEEE International Conference on Neural Networks, March-April 1993, San Francisco, Calif, USA 3, 1955–1960

  13. B Narayanan, RC Hardie, RA Muse, Scene-based nonuniformity correction technique that exploits knowledge of the focal-plane array readout architecture. Applied Optics 44(17), 3482–3491 (2005). PubMed Abstract | Publisher Full Text OpenURL

  14. MM Hayat, SN Torres, EE Armstrong, SC Cain, B Yasuda, Statistical algorithm for nonuniformity correction in focal-plane arrays. Applied Optics 38(5), 772–780 (1999). PubMed Abstract | Publisher Full Text OpenURL

  15. SN Torres, MM Hayat, Kalman filtering for adaptive nonuniformity correction in infrared focal-plane arrays. Journal of the Optical Society of America A 20(3), 470–480 (2003). Publisher Full Text OpenURL

  16. RC Hardie, MM Hayat, A nonlinear-filter based approach to detector nonuniformity correction. Proceedings of IEEE-EURASIP Workshop on Nonlinear Signal and Image Processing, June 2001, Baltimore, Md, USA, 66–85

  17. WF O'Neil, Dithered scan detector compensation. Proceedings of the Infrared Information Symposium (IRIS) Specialty Group on Passive Sensors, 1993, Ann Arbor, Mich, USA

  18. WF O'Neil, Experimental verification of dither scan non-uniformity correction. Proceedings of the Infrared Information Symposium (IRIS) Specialty Group on Passive Sensors, 1997, Monterey, Calif, USA 1, 329–339

  19. RC Hardie, MM Hayat, EE Armstrong, B Yasuda, Scene-based nonuniformity correction with video sequences and registration. Applied Optics 39(8), 1241–1250 (2000). PubMed Abstract | Publisher Full Text OpenURL

  20. BM Ratliff, MM Hayat, RC Hardie, An algebraic algorithm for nonuniformity correction in focal-plane arrays. Journal of the Optical Society of America A 19(9), 1737–1747 (2002). Publisher Full Text OpenURL

  21. BM Ratliff, MM Hayat, JS Tyo, Radiometrically accurate scene-based nonuniformity correction for array sensors. Journal of the Optical Society of America A 20(10), 1890–1899 (2003). Publisher Full Text OpenURL

  22. BM Ratliff, MM Hayat, JS Tyo, Generalized algebraic scene-based nonuniformity correction algorithm. Journal of the Optical Society of America A 22(2), 239–249 (2005). Publisher Full Text OpenURL

  23. U Sakoglu, RC Hardie, MM Hayat, BM Ratliff, JS Tyo, An algebraic restoration method for estimating fixed-pattern noise in infrared imagery from a video sequence. Applications of Digital Image Processing XXVII, August 2004, Denver, Colo, USA, Proceedings of SPIE 5558, 69–79

  24. SC Park, MK Park, MG Kang, Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 20(3), 21–36 (2003). Publisher Full Text OpenURL

  25. S Borman, in Topics in multiframe superresolution restoration, Ph.D. dissertation

  26. RR Schultz, RL Stevenson, A Bayesian approach to image expansion for improved definition. IEEE Transactions on Image Processing 3(3), 233–242 (1994). PubMed Abstract | Publisher Full Text OpenURL

  27. P Cheeseman, B Kanefsky, R Kraft, J Stutz, R Hanson, Super-resolved surface reconstruction from multiple images. in Tech, ed. by . Rep. FIA-94-12 (NASA, Moffett Field, Calif, USA, 1994)

  28. SC Cain, RC Hardie, EE Armstrong, Restoration of aliased video sequences via a maximum-likelihood approach. Proceedings of National Infrared Information Symposium (IRIS) on Passive Sensors, March 1996, Monterey, Calif, USA, 230–251

  29. RC Hardie, KJ Barnard, EE Armstrong, Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Transactions on Image Processing 6(12), 1621–1633 (1997). PubMed Abstract | Publisher Full Text OpenURL

  30. CA Segall, AK Katsaggelos, R Molina, J Mateos, Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing 13(7), 898–910 (2004). PubMed Abstract | Publisher Full Text OpenURL

  31. EE Armstrong, MM Hayat, RC Hardie, SN Torres, BJ Yasuda, Nonuniformity correction for improved registration and high-resolution image reconstruction in IR imagery. in Applications of Digital Image Processing XXII, July 1999, Denver, Colo, USA, Proceedings of SPIE, vol. 3808, , ed. by Tescher AG, pp. 150–161

  32. EE Armstrong, MM Hayat, RC Hardie, SN Torres, B Yasuda, The advantage of non-uniformity correction pre-processing on infrared image registration. Application of Digital Image Processing XXII, July 1999, Denver, Colo, USA, Proceedings of SPIE 3808

  33. S Cain, EE Armstrong, B Yasuda, Joint estimation of image, shifts, and nonuniformities from IR images. Infrared Information Symposium (IRIS) on Passive Sensors, 1997, Ann Arbor, Mich, USA (Infrared Information Analysis Center, ERIM International)

  34. S Geman, D Geman, Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6(6), 721–741 (1984)

  35. J Besag, Spatial interaction and the statistical analysis of lattice systems. Journal of the Royal Statistical Society B 36(2), 192–236 (1974)

  36. H Derin, E Elliott, Modeling and segmentation of noisy and textured images using Gibbs random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 9(1), 39–55 (1987). PubMed Abstract OpenURL

  37. SC Cain, MM Hayat, EE Armstrong, Projection-based image registration in the presence of fixed-pattern noise. IEEE Transactions on Image Processing 10(12), 1860–1872 (2001). PubMed Abstract | Publisher Full Text OpenURL

  38. RC Hardie, KJ Barnard, JG Bognar, EE Armstrong, EA Watson, High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system. Optical Engineering 37(1), 247–260 (1998). Publisher Full Text OpenURL

  39. M Irani, S Peleg, Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 53(3), 231–239 (1991). Publisher Full Text OpenURL