Foveated imaging
Foveated imaging is a digital image processing technique in which the image resolution, or amount of detail, varies across the image according to one or more "fixation points". A fixation point indicates the highest resolution region of the image and corresponds to the center of the eye's retina, the fovea.
The location of a fixation point may be specified in many ways. For example, when viewing an image on a computer monitor, one may specify a fixation using a pointing device, like a computer mouse. Eye trackers which precisely measure the eye's position and movement are also commonly used to determine fixation points in perception experiments.[1][2] When the display is manipulated with the use of an eye tracker, this is known as a gaze contingent display.[3] Fixations may also be determined automatically using computer algorithms.[4][5]
Some common applications of foveated imaging include imaging sensor hardware[6] and image compression.[7] For descriptions of these and other applications, see the list below. Miniaturized foveated imaging systems can be realized by high-resolution 3D printing of multi-lens objectives directly on a CMOS (Complementary metal-oxide-semiconductor) chip.[8]
Foveated imaging is also commonly referred to as space variant imaging or gaze contingent imaging.
Applications
[edit]
Compression
[edit]Contrast sensitivity falls off dramatically as one moves from the center of the retina to the periphery.[9][10] In lossy image compression, one may take advantage of this fact in order to compactly encode images. If one knows the viewer's approximate point of gaze, one may reduce the amount of information contained in the image as the distance from the point of gaze increases. Because the fall-off in the eye's resolution is dramatic, the potential reduction in display information can be substantial. Also, foveation encoding may be applied to the image before other types of image compression are applied and therefore can result in a multiplicative reduction.
Foveated sensors
[edit]Foveated sensors are multiresolution hardware devices that allow image data to be collected with higher resolution concentrated at a fixation point. An advantage to using foveated sensor hardware is that the image collection and encoding can occur much faster than in a system that post-processes a high resolution image in software.[11]
Simulation
[edit]Foveated imaging has been used to simulate visual fields with arbitrary spatial resolution. For example, one may present video containing a blurred region representing a scotoma. By using an eye-tracker and holding the blurred region fixed relative to the viewer's gaze, the viewer will have a visual experience similar to that of a person with an actual scotoma.
Video gaming
[edit]Foveated rendering is a technique which uses an eye tracker integrated with a virtual reality headset to reduce the rendering workload by greatly reducing the image quality in the peripheral vision (outside of the zone gazed by the fovea).[12]
At the CES 2016, SensoMotoric Instruments (SMI) demoed a new 250 Hz eye tracking system and a working foveated rendering solution. It resulted from a partnership with camera sensor manufacturer Omnivision who provided the camera hardware for the new system.[13]
The Apple Vision Pro mixed reality headset features dynamic foveated rendering provided by its visionOS operating system.[14][15]
Quality assessment
[edit]Foveated imaging may be useful in providing a subjective image quality measure.[16] Traditional image quality measures, such as peak signal-to-noise ratio, are typically performed on fixed resolution images and do not take into account some aspects of the human visual system, like the change in spatial resolution across the retina. A foveated quality index may therefore more accurately determine image quality as perceived by humans.
Image database retrieval
[edit]In databases that contain very high resolution images, such as a satellite image database, it may be desirable to interactively retrieve images in order to reduce retrieval time. Foveated imaging allows one to scan low resolution images and retrieve only high resolution portions as they are needed. This is sometimes called progressive transmission.
Example images
[edit]- Various foveated images
-
Image with foveated luminance channel -
Foveated image with fixation selected using an entropy minimization algorithm
- Foveated imaging as a compression technique
-
18:1 compression. Foveated image with fixation point on tombstone.
See also
[edit]References
[edit]- ^ McConkie G W and Rayner K (1975) The span of the effective stimulus during a fixation in reading, Perception & Psychophysics, 17, 578–86
- ^ Loschky, L.C. & Wolverton, G.S. (2007). How Late Can You Update Gaze-contingent Multi-resolutional Displays Without Detection? ACM Transactions on Multimedia Computing, Communications and Applications, 3(4):25, 1-10.
- ^ Duchowski, A. T., Cournia, N., and Murphy, H. 2004. Gaze-Contingent displays: A review. Cyberpsychol. Behav. 7, 6, 621--634.
- ^ Z. Wang, L. Lu and A. C. Bovik, "Foveation scalable video coding with automatic fixation selection," IEEE Trans. on Image Processing, Vol: 12 No: 2, February 2003.
- ^ R. G. Raj, W. S. Geisler, R. A. Frazor, A. C. Bovik, "Contrast statistics for foveated visual systems: Fixation selection by minimizing contrast entropy" Journal of the Optical Society of America.
- ^ J.A. Boluda, F. Pardo, T. Kayser, J.J. P'erez, and J. Pelechano. A new foveated space-variant camera for robotic applications. In IEEE, International Conference on Electronics Circuits And Systems, ICECS'96, Rodos, Greece, October 1996.
- ^ Geisler, W.S. and Perry, J.S. (1998) A real-time foveated multi-resolution system for low-bandwidth video communication. In B. Rogowitz and T. Pappas (Eds.), Human Vision and Electronic Imaging, SPIE Proceedings, 3299, 294-305.
- ^ Thiele, Simon; Arzenbacher, Kathrin; Gissibl, Timo; Giessen, Harald; Herkommer, Alois M. (2017-02-03). "3D-printed eagle eye: Compound microlens system for foveated imaging". Science Advances. 3 (2): e1602655. Bibcode:2017SciA....3E2655T. doi:10.1126/sciadv.1602655. ISSN 2375-2548. PMC 5310822. PMID 28246646.
- ^ Wandell, Brian A. (1995) Foundations of Vision. ISBN 0-87893-853-2 . p.236
- ^ Barghout-Stein, Lauren. On differences between peripheral and foveal pattern masking. Diss. University of California, Berkeley, 1999.
- ^ Marc Bolduc, Martin D. Levine. A real-time foveated sensor with overlapping receptive fields. June 1997, Real-Time Imaging, Volume 3 Issue 3
- ^ Parrish, Kevin (2016-07-22). "Nvidia plans to prove that new method improves image quality in virtual reality". Digital Trends. Retrieved 2017-02-02.
- ^ Mason, Will (2016-01-15). "SMI's 250Hz Eye Tracking and Foveated Rendering Are For Real, and the Cost May Surprise You". UploadVR. Retrieved 2017-02-02.
- ^ "visionOS Overview". Apple Developer. Retrieved 2024-02-09.
- ^ "📌 Official Support for visionOS Now Available". Unity Discussions. 2024-01-22. Retrieved 2024-02-09.
- ^ Z. Wang, A. C. Bovik, L. Lu and J. Kouloheris, "Foveated wavelet image quality index," SPIE's 46th Annual Meeting, Proc. SPIE, Application of digital image processing XXIV, vol. 4472, July-Aug. 2001.