Image Optimization During Facial Recognition

[0001] This application is a continuation of co-pending U.S. patent Application Ser. No. 16/582,047, filed Sep. 25, 2019, which is a continuation of U.S. patent Application Ser. No. 16/203,272, filed Nov. 28, 2018, which is a continuation of U.S. patent Application Ser. No. 15/014,222, filed Feb. 3, 2016, which is a divisional of U.S. patent Application Ser. No. 13/655,817, filed Oct. 19, 2012, all of which are incorporated by reference herein in their entirety and for all purposes.

[0002] BACKGROUND

[0003] Devices with a built-in camera may have the ability to authenticate a user by performing facial recognition. Facial recognition methods often process an image in two steps. The first step, which may be described as detection, locates a face within an image. The second step, which may be described as recognition, analyzes the detected face to determine a similarity between the detected face and a stored face or template. To achieve accurate results, the facial recognition processes may depend upon the quality of detected images. When capturing images, a camera may automatically adjust capture settings based on image characteristics. These settings may include automatic adjustments to exposure and/or gain. For example, when an image condition includes a face with strong backlighting, the automatic adjustment may reduce exposure in the image to account for the backlighting. These image settings, however, may be optimized for capturing the most visually appealing pictures.

[0004] In an implementation, described is a method of optimizing a captured image for facial detection. In response to receiving an indication to perform facial detection, a metering region within a portion of the captured image may be selected. The metering region may be based on an expected location of a face and may be selected prior to detecting a face within the image. The image may be adjusted based on the metering region and a face may be detected within the adjusted image. The image may be optimized by adjusting an exposure, and/or another setting including an automatic gain control (AGC), contrast, saturation, and the like.

[0005] In an implementation, a captured image may be optimized based on a predefined and static metering region. The metering region may be described based on a size. The size of the metering region may be described as being within a percentage range of the image. The metering region may also be described based on a relative location within the image. For example, the metering region may be centered, within a specified portion of the image, or include at least two quadrants of the image.

[0006] In an implementation, described is a facial detection method including executing a facial detection algorithm on a captured image a first time and determining that a face is not detected within the image. A metering region within a portion of the image may be selected and the image may be adjusted based on the metering region. The facial detection algorithm may execute on the adjusted image a second time and may detect a face. The first execution may be based on a metering region of the entire image and the second execution may be based on a metering region optimized for facial detection.

[0007] In an implementation, described is a device for optimizing an image for facial detection. The device may include a camera for capturing the image, a display previewing the captured image, and a processor for adjusting the captured image based on a metering region within a portion of the image. The processor may adjust the camera based on a metering region and a face may be detected based on the adjusted image. A preview of the captured image may be displayed and the metering region may be based on a positioning guide displayed on the previewed image. The display may be a touch sensitive display and the metering region may be based on a user input touching a portion of the previewed image.

[0017] Described is a process of improving facial detection in an image by adjusting image settings. Implementations describe adjusting the image settings based on a metering region that optimizes the image for detecting a face. The metering region may use a relevant subset of the image and may be based on an expected location of at least a portion of a face within the image. By positioning the metering region in an expected location of a face, the metering region may be selected prior to actually detecting a face.

[0018] FIG. 1 shows a simplified functional block diagram of a representative device according to an implementation of the disclosed subject matter. The device 50 may include a bus 51, processor 52, memory 54, user input 56, display 58, memory 54, communications circuitry 53, storage 57, coder/decoder (CODEC) 55, and a camera 59.

[0019] The device 50 may include or be part of a variety of types of devices, such as a mobile phone (including a “smartphone”), tablet computer, netbook, laptop, desktop, personal digital assistant (“PDA”), set-top box, television, and/or watch, among others. The processor 52 may be any suitable programmable control device and may control the operation of one or more processes, such as the facial detection process as discussed herein, as well as other processes performed by the device 50.

[0020] The memory 54 may include one or more different types of memory which may be used for performing device functions. For example, the memory 54 may include cache, ROM, and/or RAM. The bus 51 may provide a data transfer path for transferring between components of the device 50. The CODEC 55 may be included to process digital and analog audio and/or video signals to produce sound, voice, and multimedia. The display 58 may provide visual output and may include a touch-sensitive screen.

[0021] The storage 57 may store multimedia (e.g. video, music, and photos, etc.), software (e.g., for implementing various functions on device 50), and any other suitable data. The storage 57 may include a storage medium, such as a hard-drive, solid state drive, flash drive, and the like. The storage 57 may be integral with the device 50 or may be separate and accessed through an interface to receive a memory card, USB drive, optical disk, and the like.

[0022] The user input 56 may allow a user to interact with the device 50. For example, the user input 56 may include buttons, a keypad, a touch screen, and the like. The communications circuitry 53 may include circuitry for wireless communications for short-range and/or long range communication. For example, the wireless communication circuitry may include Wi-Fi enabling circuitry for one of the 802.11 standards, and circuitry for other wireless network protocols including Bluetooth, the Global System for Mobile Communications (GSM), and code division multiple access (CDMA) based wireless protocols. Communications circuitry 53 may also include circuitry that enables the device 50 to be electrically coupled to another device (e.g. a computer or an accessory device) and communicate with that other device.

[0023] The camera 59 enables the device 50 to capture still images, video, or both. The camera 59 may capture an image or series of images continuously, periodically, at select times, and/or under select conditions. The camera 59 may be part of or coupled to one or both a front-facing camera or rear-facing camera of the device 50.

[0024] The device 50 may be battery-operated and portable so as to allow a user to communicate with others, listen to music, play games or video, record video or take pictures, or control other devices. The device 50 may be relatively compact which enables a user to easily manipulate the device's position, orientation, and movement. Accordingly, the device 50 may provide techniques of sensing such changes in position, orientation, and movement to enable a user to interface with or control the device 50 by affecting such changes. Further, the device 50 may include a vibration source, under the control of processor 52, for example, to facilitate sending motion, vibration, and/or movement information to a user related to an operation of the device 50. Other devices or components may be part of or connected to the device 50, and conversely, all of the components shown in FIG. 1 need not be present to practice the present disclosure.

[0025] FIG. 2 shows a flow diagram of optimizing an image for facial detection according to an implementation of the disclosed subject matter. In step 22, the device 50 may capture an image with a camera 59. Typically, the captured image may be a live image that dynamically and continuously captures an image within a field of view of the camera 59 in real-time. The captured image may also include a still image or single frame, for example, when a snapshot or picture is taken. In addition, the captured image may also include a retrieved image that may be stored on the device 50. A preview of the captured image may be shown on a display 58 in real-time or upon retrieval. Image capture may be initiated automatically, for example, upon a request to perform facial detection and/or recognition, or it may be initiated manually by a user. The image capture may occur prior to or after receiving an indication to perform facial recognition.

[0026] In step 23, the device 50 may receive an indication to perform facial detection. The indication for facial detection may be explicit or implicit. An explicit request to perform facial detection may include a user selecting a facial detection setting on the device 50 for image capture. This selection may be made in real-time, for example, while capturing a live image, or prior to capturing an image such as setting an image capture preference (e.g. facial detection mode). The explicit indication may also include a request to perform an image adjustment for facial detection and/or recognition on a stored image (e.g. a picture). An implicit indication may include initiating or requesting a process that includes facial detection as a sub-process or step. For example, a facial recognition process may include performing an initial or concurrent step of facial detection. The device 50 may have an access control feature that authenticates a user of the device by performing facial recognition. Accordingly, the indication may include various forms of a request for user authentication. Facial recognition as a form of authentication may be common for many applications in handheld devices. Facial recognition is particularly convenient for devices with a front-facing camera because they may efficiently perform facial recognition when a user is located in a position to view a display 58. The request for authentication may be automatic or may be initiated manually by the user. For example, the authentication request may occur upon a request to “unlock” the device from a “locked” state. This may occur automatically, for example, when the device 50 “awakens” from a powered-down or “sleep” state, or it may occur manually, for example, by direct user input. The device 50 may include controlling user access at an application level. Accordingly, another example of direct user input may include launching an access controlled application to initiate an authentication request.

[0027] In step 24, a metering region may be selected. The selection may be in response to the indication to perform facial recognition. The response may include an immediate response, or it may include a deferred response that may occur during the detection process. For example, upon receiving an indication that a facial detection process has been initiated, the device 50 may first activate the camera 59. Accordingly, the selection of the metering region may be deferred until the camera 59 is activated and initiates image capture. The selection of a metering region may occur without, or prior to, detecting a face within an image. Conventional facial recognition programs typically perform facial detection first, and then optimize the image based on image characteristics of the detected face. Facial detection methods may include detecting a face based on a shape, skin color, relative positioning of facial features, etc. In certain circumstances, however, the facial detection may not be possible. For example, when capturing images, a camera may automatically adjust capture settings based on characteristics of the entire image. For example, these settings may include automatic adjustments to exposure or other images settings. These image settings may be optimized for capturing the most visually appearing pictures. At this stage, these adjustments may not be optimized for facial detection because a face has not been detected yet. For example, when an image condition includes a face with strong backlighting, the automatic adjustment may reduce exposure in the image to account for the backlighting. This adjustment may alter the image to a point where there is insufficient contrast in the region of the face. As a result, the device 50 may not be able to perform facial detection.

[0028] Implementations of the disclosed subject matter may optimize an image without the requirement to perform facial detection. More specifically, implementations may select a metering region by predicting an expected location of a face. Accordingly, the metering region may provide characteristics of a portion face and not the entire image. For example, implementations may predefine the metering region to include a central region of a captured image. A user capturing an image of their face is likely to place their face within the center or close to the center of a field of view of a camera 59. Accordingly, there may be a high likelihood that the metering region may include at least a portion of the user's face, and as a result, the image may be adjusted according to a more relevant subset of the image for facial detection. Thus, the image may be optimized without, or prior to, performing facial detection.

[0029] The metering region may be described relative to the captured image. The captured image may be described in various manners including a coordinate space, quadrants, pixels values based on a resolution, zones, and the like. For example, using a coordinate space, the lower left most point of the image may have a value of −1000, −1000, and the upper right most point may have a value of 1000, 1000. Accordingly, a center point of the image would be 0, 0. A coordinate space may also be based on resolution. For example, a 1280×800 resolution screen may have 1, 1 (or 0, 0) as the lower left most point and 1280, 800 as the upper right most point. A metering region may be defined by its size, position, behavior and other characteristics. The metering region may be described in terms of a percentage size of the image and a relative position. For example, the metering region may be centered within the image and have a size of 20% of the image. In some cases, the metering region may be described as being within a portion of the image. A portion of the image is defined as an area less than the entire image. For example, a metering region may be within a central portion of the image. A central portion is defined as a rectangular shape centered at the center point of the image and may have a specified size. For example, using a coordinate space as described above, a central portion with a size of 50% of the image may have a coordinate definition of −500, −500 as the lower left most point and 500, 500 as the upper right most point of the rectangle. A metering region within this central portion may include the entire central portion or may have a smaller area. A metering region within a portion of an image may not necessarily be rectangular in shape and may not necessarily be centered at the center point of the portion of the image.

[0030] FIG. 4 shows an example of a metering region within a central portion of the image according to an implementation of the disclosed subject matter. As shown, metering region 40 may be within a portion of the image 42. In this example, the portion of the image 42 may be described as a central portion. As shown, the portion 42 is centered at a center point (not shown) of the image. As with this example, although the metering region 40 is within a central portion of the image, it may not be centered at the center point of the image. This offset may account for any bias that may occur. For example, a user may have a tendency to position their face lower within the field of view of a camera 59.

[0031] The position of the metering region may also be described with respect to quadrants. For example, if a metering region is centered within the image, it may include a portion of all four quadrants. If the metering region is centered only horizontally or vertically, it may only include a portion of a minimum of two quadrants.

[0032] FIG. 5 shows an example of a metering region including two quadrants of the image according to an implementation of the disclosed subject matter. As shown, the image may be divided in four equal quadrants, starting from the upper left most quadrant as quadrant 1 51, and clockwise to quadrant 2 52, quadrant 3 53, and quadrant 4 54. As shown, the metering region may include one or more quadrants. In this example, the metering region is centered horizontally within the image, and in this instance, includes a portion of two quadrants (quadrant 3 53 and quadrant 4 54).

[0033] Typically, a metering region may be defined as a set of coordinates at a programming level. It should be noted that the above methods of describing the location of the metering region and describing the organization of the image may not be explicitly defined at a programming level. A person of ordinary skill in the art, however, would understand such descriptions and select a set of coordinates accordingly. For example, a description of a metering region with a size of 20% of the image and within a central portion with a 50% size of the image, would enable a person of ordinary skill in the art to understand that the region may be defined with a set of coordinates that need only be within the −500, −500 and 500, 500 rectangular boundary in the coordinate space as described above.

[0034] The metering region may also be described by its behavior. For example, the metering region may be static or dynamic. When the metering region is static, the region does not necessarily track particular objects within the image. For example, with live images, objects may move within the image and these may or may not remain within the metering region. For instance, if a coordinate system is used, the coordinates of the metering region may remain static despite movement of objects within the image. In contrast, when the metering region is dynamic, the metering region may track a particular object within the live image and the coordinates of the metering region may adjust accordingly.

[0035] The metering region may also be predefined. For example, a set of coordinates defining the metering region may be set prior to capturing an image. Accordingly, the metering region may be positioned in a particular region regardless of the characteristics of the captured image. As described above, the metering region may be based on an expected location of a face. Typically, the expected location of a face corresponds to a central portion of the image. This positioning may be further refined by additional factors. These factors may include characteristics derived from preferences, usage history, probability, etc. For example, a usage history may determine that a user typically places the face within a lower portion of the image and not precisely in a central portion. Accordingly, the selection of the metering region may be repositioned accordingly.

[0036] In implementations, the captured image may be displayed as a preview on a display 58 of the device 50. In these situations, a positioning guide may be used to direct a user to position their face within a certain region of a field of view of the camera 59. For example, a guide may be displayed on the previewed image and a user may alter the position of the device 50 and/or their face to position their face according to the guide. Accordingly, a metering region corresponding to the displayed guide may be used to adjust the image without actually detecting the face. In this instance, it is likely that a user cooperates and positions their face according to the guide. In some cases, the positioning guide may be dynamic and the metering region may dynamically track the positioning guide. The positioning guide may include a boundary, cross-hairs, one or more boundary edges, a shape (e.g. circle or rectangle), shading, or any other form of displaying a guide on the previewed image.

[0037] The metering region may also be based on receiving a user input that selects a portion of the captured image. The user input may include an input from an input device or a touch on a on touch sensitive display. For example, a user may touch the display 58 at a location corresponding to the location of a face within a captured image. The touch may be registered by the device 50 and a metering region may be selected based on the location of the touch. This may be an effective technique when performing facial recognition on an image including multiple faces in a live image or a stored image.

[0038] The metering region may also be based on characteristics of the image. As described, a face may not necessarily need to be detected to select a metering region. The metering region may be based on a contrast level, saturation level, color, and other image characteristics. For example, a face in front of a strongly lit background may have a low contrast level. The low contrast level may be identified as a probable location of a face, and accordingly, the metering region may be within this low contrast region. Similarly, a low saturation may be identified for a metering region. Other image characteristics may also be used to distinguish between regions or identify a region that may include a face. More than one metering region may be selected and they may overlap. The metering region may also be defined with a particular weighting of pixels within the image. For example, the weights of overlapping metering regions may be added. Unlike typical facial detection processes, it may not be necessary to show an outline or boundary region of the metering region on a previewed image. Instead, the metering region and adjustment may be done transparently to the user.

[0039] In step 25, the captured image may be adjusted based on characteristics of the metering region. The adjustment may adjust a live image in real-time or may adjust a picture that may be stored. For example, an adjustment to a live image may involve dynamically adjusting the captured image in response to real-time changes in the field of view of the camera 59. The adjusting may adjust capture settings of the camera 59, or may adjust the characteristics of a frame, or a stored image (e.g. snapshot) after a picture is taken. The metering region may typically correspond to at least a portion of a face within the image even when the face is not detected. Accordingly, the metering region may be optimized based on the characteristics of a portion of a face. The image adjustments may include optimizing one or more image settings. These settings may include exposure, gain, color balance (including white-balance), focus, contrast, (which may be expressed as a ratio or dynamic range), saturation, brightness, gamma, hue, sharpness, and the like. In should be noted that these settings may be adjusted in an image and would be readily understood by a person of ordinary skill in the art.

[0040] Typically, an exposure setting may be adjusted. An exposure setting may generally relate to camera exposure parameters, such as exposure time (e.g. shutter speed), gain, and/or f-number (e.g. relative aperture). By adjusting the exposure setting, an optimum brightness may be achieved for a face within the captured image. As described, the exposure may be optimized based on a metering region which may be a subset of the image. Conventional metering may be optimized for capturing the most visually appearing pictures, and therefore, are based on metering an entire image. Instead of metering an entire image, implementations may select the most relevant subset of the image for facial detection (e.g. at least a portion of a face). Accordingly, these adjustments may not produce the images most suitable for pictures as they may be optimized for facial detection. For example, the adjustment may adjust exposure to enhance contrast in a region of the image including a face (e.g. foreground), and as a result, it may oversaturate a background of the image to a point where objects in the background may not be visible. As mentioned above, other image settings may also be adjusted to optimize facial recognition.

[0041] FIGS. 6A-6C show an example of adjusting an exposure setting of an image based on a metering region within a portion of the image according to an implementation of the disclosed subject matter. FIG. 6A shows an example of an underexposed face within an image. This may occur, for example, when an image condition includes strong backlighting (e.g. when the subject has their back to a brightly lit window). An exposure setting, for example, may be based on an analysis of the entire image. Accordingly, when the image background is very bright this method may set the exposure to low levels such that the background is below a saturation level. As a result, as shown in FIG. 6A, the face may be within a low contrast region 60. In these situations, there is often insufficient contrast in the region of the face to perform detection. Accordingly, adjusting the image to such levels as shown in FIG. 6A may prevent detection of the face within the image.

[0042] FIG. 6B shows an example of a metering region within a portion of the image shown in FIG. 6A according to an implementation of the disclosed subject matter. A metering region 40 may be within a portion of the image and may correspond to an expected location of a face. For example, as shown, the metering region 40 may be a specified shape (e.g. rectangle) with dimensions equal to a certain percentage of the input image (e.g. a percentage of the image size) and placed at a specified location (e.g. center of the input image). The metering region 40 may be predefined and set prior to actually detecting a face.

[0043] FIG. 6C shows an example of an adjusted image based on the metering region 40 of FIG. 6B according to an implementation of the disclosed subject matter. The result of using the metering region 40 is that the image adjustment to exposure may be based on a portion of the face, and therefore, optimized for facial detection. As shown in this example, an image with strong backlighting may be optimized by adjusting the exposure setting resulting in oversaturating the background 62. As also shown, however, the contrast range in the area of the face may be improved, and accordingly, the image may be optimized for facial detection.

[0044] Returning to FIG. 2, in step 26, a face may be detected within the adjusted image. The facial detection method may include known methods for detection and be based one or more algorithms for identifying shapes, skin tone, color, facial texture data, and facial features within an image. As described above, a metering region may be selected prior to facial detection. In other implementations, a metering region may be selected after a facial detection algorithm is performed.

[0045] FIG. 3 shows a flow diagram of performing facial detection a first and second time according to an implementation of the disclosed subject matter. In step 30, the device 50 may capture an image in a similar manner as described in step 22 of FIG. 2. In step 31, a first metering region may be selected. The first metering region may be based on an entirety of an image. For example, a standard or default auto exposure algorithm may be used, such as one that bases exposure parameters on a mean, median, or weighted value of the brightness of the entire image. In step 32, the image may be adjusted based on the first metering region. This adjustment may be based on providing the most visually pleasing images and may be a default setting, or a setting optimized for taking pictures.

[0046] In step 33, facial detection may be executed a first time. In implementations, the facial detection in step 33 may occur without adjusting the image in steps 31 and 32. In step 34, it may be determined that a face is not detected when executing facial detection the first time as the image adjustment (if performed in steps 31 and 32) may not have necessarily been optimized for facial detection. After it is determined that a face was not detected, a metering region may be selected in step 35 and the captured image may be adjusted in step 36 in a similar manner as described in steps 24 and 25 of FIG. 2 respectively. In particular, in steps 35 and 36, the image may be optimized for facial detection. In step 37, the facial detection algorithm may execute a second time and may detect a face within the image optimized for facial detection. As described above, the first metering region may be based on the entire image. The first metering region may also include a subset of the image. For example, a size of the first metering region may be larger than a size of the metering region within the portion of the image of step 35. Accordingly, in this scenario, the metering region in step 35 may be reduced to provide more precision for facial detection optimization.

[0047] In implementations, once facial detection is performed, the device 50 may perform facial recognition on the detected face. The recognition process may include comparing the detected face to a stored template. For example, features of the detected face template may be compared against the features of the stored reference template and a recognition confidence score may be determined. Upon a successful facial recognition (e.g. confidence score above a specified value), the device 50 may identify the face of the user. For example, upon the completion of facial recognition, a user's face may be authenticated and the device may be accessed or “unlocked” in response to the authentication request.

[0048] Implementations described herein may be included in an article of manufacture (e.g., one or more computer program products) having, for instance, computer usable media. The media has embodied therein, for instance, computer readable program code for providing and facilitating the capabilities described herein. The article of manufacture may be included as a part of a device or sold separately.

[0049] The flow diagrams described herein are just examples. There may be variations to these diagrams or the steps (or operations) described therein without departing from the implementations described. For instance, the steps may be performed in a differing order, or steps may be added, deleted or modified. Described herein, references to “one implementation,” “an implementation,” “an example implementation,” etc., indicate that the implementation described may include a particular feature, structure, or characteristic, but every implementation may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same implementation. Further, when a particular feature, structure, or characteristic is described in connection with an implementation, such feature, structure, or characteristic may be included in other implementations whether or not explicitly described.

[0050] The foregoing description, for purpose of explanation, has been described with reference to specific implementations. However, the illustrative discussions above are not intended to be exhaustive or to limit implementations of the disclosed subject matter to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The implementations were chosen and described in order to explain the principles of implementations of the disclosed subject matter and their practical applications, to thereby enable others skilled in the art to utilize those implementations as well as various implementations with various modifications as may be suited to the particular use contemplated.