System, computing device, and method for document detection

[0001] This application claims benefit to U.S. Provisional Patent Application No. 62/663,813, filed on Apr. 27, 2018, the entirety of which is hereby incorporated by reference herein.

[0002] Electronic check image detection and remote check depositing are part of technological improvements to the manual process of check depositing known as remote deposit. In general, checks provide a convenient method for an individual such as a payor to transfer funds to a payee. To use a check, an individual usually opens a checking account, or other similar account, at a financial institution and deposits funds, which are then available for later withdrawal. To transfer funds with a check, the payor usually designates a payee and an amount payable on the check. In addition, the payor often signs the check. Once the check has been signed, it is usually deemed negotiable, meaning the check may be validly transferred to the payee upon delivery. By signing and transferring the check to the payee, the payor authorizes funds to be withdrawn from the payor's account on behalf of the payee.

[0003] While a check may provide a payor with a convenient and secure form of payment, receiving a check may put certain burdens on the payee, such as the time and effort required to deposit the check. For example, depositing a check typically involves going to a local bank branch and physically presenting the check to a bank teller.

[0004] To reduce such burdens for the payee, as well as the payor, remote deposit technology have been developed to include systems and methods that enable the remote deposit of checks that overcome the need for some of the manual processes in prior check deposit procedures.

[0016] The methods, devices, and systems discussed below may be embodied in a number of different forms. Not all of the depicted components may be required, however, and some implementations may include additional, different, or fewer components from those expressly described in this disclosure. Variations in the arrangement and type of the components may be made without departing from the spirit or scope of the claims as set forth herein.

[0017] Within the technological field of remote deposit, a payor may present a check, either in paper or electronic versions, to a payee for deposit. To deposit the check with a financial account at a financial institution associated with the payee, the payee may scan a check in a digital image using a scanner, digital camera, or other imaging device in communication with a computing device. A financial institution may then receive from the payee the digital image of the check. The financial institution may then use the digital image to credit funds to the payee. Such remote deposit techniques utilize the efficient and accurate detection of a check depicted within a digital image, where part of the check image detection include detecting and extracting the check image from background portions included in the digital image that are not part of the check image.

[0018] The image processing steps for detecting and extracting the check image from the background portions may be implemented in whole, or in parts, on either a user device having captured the digital image, a remote server running an application for remote deposit, or both. Implementing these image processing steps are known to expend significant computing resources on a computer device, and so there are benefits to offloading, or sharing, some or all of the image processing between the user device and the remote server. However, with the improvements to the computing capabilities of mobile computing devices, it is within the scope of this disclosure for one or more, or all, of the image processing features to be implemented directly on the mobile computing device (e.g., user device).

[0019] In the disclosed embodiments, features are disclosed for dedicating image processing from the remote server to the user device, as well as for dedicating image processing from the user device to the remote server. The determination of where the image processing will be implemented may be determined, for example, according to detected attributes of the captured digital image depicting the check, computing capabilities of the mobile device including the image capture device, and/or environmental conditions (e.g., ambient light levels) detected when the digital image was captured.

[0020] FIG. 1 shows a block diagram of an implementation of a system 100 in which example embodiments may be implemented. A user 105 is shown along with an institution system 205. The institution system 205 may be affiliated with an institution 200, which may be any type of entity capable of processing a transaction involving a negotiable instrument, such as processing checks and/or providing funds associated with checks. For example, the institution 200 may be a financial services institution such as a retail bank, an investment bank, an investment company, a regional branch of the Federal Reserve, a clearinghouse bank and/or a correspondent bank. A representative 185 of the institution 200 may provide assistance, via a representative computing device 187, during any one or more of the processes described herein.

[0021] A negotiable instrument typically includes a type of contract that obligates one party to pay a specified sum of money to another party. Negotiable instruments may include checks, money orders, cashier's checks, drafts, bills of exchange, promissory notes, and the like. A check instructs a financial institution to pay an amount of money from a specific account held in the payor's name with that financial institution to an account held in the payee's name. A money order is a trusted financial instrument that is a payment order for a pre-specified amount of money. It is a more trusted method of payment than a personal check because the funds for the amount shown on the money order must be prepaid. A cashier's check (also known as a bank check, official check, teller's check, bank draft or treasurer's check) is a check guaranteed by a bank and may be purchased from a bank. Cashier's checks are usually treated as cash since most banks clear them instantly.

[0022] The user 105 may be an individual or entity who has an account 165 held at the institution 200 and is accessible via the institution system 205. The account 165 may be any type of account for depositing funds, such as a savings account, a checking account, a brokerage account, and the like. Although only one account 165 is shown, it is contemplated that the user 105 may have any number of accounts held at the institution 200. The user 105 may deposit a check 107 or other negotiable instrument document into the account 165 either electronically or physically. The institution 200 may process and/or clear the check 107, as well as other negotiable instruments.

[0023] The user 105 may remotely electronically deposit the check 107 at the institution 200 via the computing device 109 being operated by the user 105. For example, an application for remotely electronically depositing checks may be downloaded and installed on the computing device 109, where running the application on the computing device 109 enables the user 105 to remotely electronically deposit the check 107. It is noted that although examples and implementations described herein may refer to a check, the techniques and systems described herein are contemplated for, and may be used for, any negotiable instrument, such as a money order, a cashier's check, a check guaranteed by a bank, or the like. Similarly, the techniques and systems described herein are contemplated for and may be used with any form or document whose image may be captured with a scanner, camera, or other imaging device for subsequent storage and/or processing.

[0024] For example, the computing device 109 may be a personal computer (PC), a laptop computer, a handheld computing device, a personal digital assistant (PDA), a mobile phone, or a smartphone, for example. The computing device 109 includes an image capture device 115 for capturing an image of the check, where the image capture device 115 may be a digital camera, image scanner, or other device in which the image of the check 107 may be obtained.

[0025] The user 105 may access the institution 200 via the institution system 205 by opening a communication pathway via a communications network 140 using the computing device 109. The communications network 140 may be representative of an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless fidelity (WiFi) network, a public switched telephone network (PSTN), a cellular network, a voice over Internet protocol (VoIP) network, or the like. The user 105 may also communicate with the institution 200 or the institution system 205 by phone, email, instant messaging, text messaging, web chat, facsimile, postal mail, and the like.

[0026] There may be several ways in which the communication pathway may be established, including, but not limited to, an Internet connection via a website 218 of the institution system 205, where the website 218 includes content 219 for accepting remote deposits. The user 105 may access the website 218 and log into the website 218 using credentials, such as, but not limited to, a username and a password.

[0027] The user 105 may operate the image capture device 115 installed on the computing device 109 to generate a digital check image of the check 107. The digital check image may be used to create a digital image file 135 that is sent to the institution system 205 and used by the institution 200, in conjunction with the institution system 205, to process a deposit of the check 107 whose image is depicted within the digital image file 135. In an implementation, the digital image file 135 may be augmented by secondary data which may be information relating to the deposit of the check 107, such as an account number and a deposit amount, for example.

[0028] When capturing the digital check image, the check 107 may be placed on a background, such that the digital check image that is captured by the image capture device 115 comprises an image of the check (e.g., a check image) and a portion of the background (e.g., a background image). Any background may be used, although a dark background or a consistently colored background may provide more optimal results for distinguishing the check image from the background image within the overall initial image. In an implementation, white and yellow backgrounds may not be used. It is noted that although examples and implementations described herein may refer to a check image and check data, the term “check image” may refer to any foreground image in a digital image (as opposed to the background image) and the term “check data” may refer to any foreground data in a digital check image (as opposed to background data). Thus, the “check image” and the “check data” may refer to the foreground image and foreground data in implementations involving any negotiable instrument, form, or document.

[0029] In an implementation, the digital check image generated by the image capture device 115 comprises check data and background data. The check data pertains to the check image in the digital check image and the background data pertains to the background in the digital check image on which the image of the check 107 is disposed. An example of such a digital check image 201 is further described with respect to FIG. 2. The digital check image 201 illustrated in FIG. 2 is comprised of a check image 210 and a background image 250 surrounding the check image 210. The check image 210 is shown to include check data such as name, address, date, payment instructions, payment amount, memo, signature line, and Magnetic Ink Character Recognition (MICR) line information. The background image 250 includes image data for the surface, or whatever other environment, is behind the check 107 when the image capture device 115 captured the digital check image 201. Between the check image 210 and the background image 250, are edges of the check 107. An edge separates the check image 210 from the background image 250, and edge detection techniques will be described in more detail herein.

[0030] In an implementation, a minimum ratio of check image size (or area) to background image size (or area) may be provided in the digital check image generated by the image capture device 115. Examples include, and are not limited to, at least a minimum portion (e.g., five percent) of the digital check image may comprise a check image, at least a minimum portion (e.g., 1 percent) of the digital check image may comprise a background image, the check image may be surrounded by at least one pixel of background all the way around the check image, or other predetermined requirements.

[0031] Referring back to the computing device 109 shown in FIG. 1, the computing device 109 may comprise an image processing engine 120 that executes image processing features used during the remote deposit process. The image processing engine 120 shown in FIG. 1 includes a background processor circuitry 121, a chroma key analyzer 122, an image comparator 123, a grayscale converter 124, an image segmenter 125, a segment processor circuitry 126, a segment recombiner 127, and an image processor circuitry 128.

[0032] The background processor circuitry 121 may perform various tasks with respect to the background image 250 included on the digital check image 201 on which the check 107 is depicted. The background image 250 may be a surface on which the check 107 is placed when the digital check image 201 is captured by the image capture device 115. In conjunction with the image capture device 115, the background processor circuitry 121 detects portions of the digital check image 201 corresponding to the background image 250, and distinguishes the background image 250 from the check image 210. The background processor circuitry 121 may further generate a modified digital check image that replaces the background image with a replacement background image. The replacement background image may be selected from one or a plurality of predetermined background images. For example, a predetermined background image may be comprised of a solid color. In an implementation, the background processor circuitry 121 may store the background image 250 and/or the modified digital check image in a storage memory associated with the computing device 109.

[0033] Based on the background image 250 being separately detected from the check image 210 according to the techniques applied by the background processor circuitry 121, the image processing engine 120 may identify specific edge coordinates of the check image 210, such as edge coordinates 301, 302, 303, 304 illustrated in FIG. 3. A modified digital check image 300 may be generated by the image processing engine 120 that includes the edge coordinates, where the modified digital check image 300 is included in the digital image file 135 transmitted to the institution system.

[0034] In an implementation, chroma key technology may be used to detect edge coordinates 301, 302, 303, 304 of the check image 210, and/or provide a uniform or consistent replacement background for the check image 210 in the digital check image 201. Chroma key is a technique for mixing two images together, in which a color or color range from one image is removed (or made transparent), revealing another image behind it. This technique is also known as color keying, greenscreen, and bluescreen. The chroma key analyzer 122 may use any known chroma key technique for removing the original background image 250 and replacing it with (e.g., revealing) a replacement background image. Based on the background image 250 being separately detected from the check image 210 according to the chroma key technique(s) applied by the chroma key analyzer 122, the image processing engine 120 may identify specific edge coordinates for the check image 210, such as the edge coordinates 301, 302, 303, 304 illustrated in FIG. 3. The modified digital check image 300 may be generated by the image processing engine 120 that includes the edge coordinates, where the modified digital check image 300 is included in the digital image file 135 transmitted to the institution system. Although the edge coordinates 301, 302, 303, 304 are illustrated to correspond to corners of the check image, other edge coordinates may correspond to non-corner edge portions that comprise an outer perimeter of the check.

[0035] Alternatively or additionally, the background image 250 may be separately distinguished from the check image 210 within the digital check image 201 based on a difference in image characteristics between the background image 250 and the check image 210 analyzed by the image comparator 123. After generating a modified digital check image that replaces the background image 250 with a predetermined replacement background image, the image comparator 123 may compare the modified digital check image with the predetermined replacement background image. By making this known image comparison, the image comparator 123 determines that overlapping the overlapping image portions correspond to a background portion, and the remaining image portions correspond to the check image 210. The image comparator 123 may then determine this difference and subtract the background portion from the modified digital check image to result in obtaining the check image 210 alone without background portions. Based on the check image 210 being separately extracted from the digital check image 201 according to the image comparison technique(s) applied by the image comparator 123, the image processing engine 120 may identify specific edge coordinates for the check image 210, such as the edge coordinates 301, 302, 303, 304 illustrated in FIG. 3. The modified digital check image 300 may be generated by the image processing engine 120 that includes the edge coordinates, where the modified digital check image 300 is included in the digital image file 135 transmitted to the institution system.

[0036] In addition, or alternatively, edge detection for the check image may be implemented by comparing the digital check image 201 to a predetermined rectangular shape that represents an outline of the check. This way, the check image 210 may be separately distinguished from the background image 250 within the digital check image 201 based on a matching of the predetermined rectangular shape with edge outlines of the check image 210 analyzed by the image comparator 123. After matching the predetermined rectangular shape to the edge outlines of the check image 210, the image comparator 123 may identify the edge coordinates of the check image 210. Based on the check image 210 being separately identified from the digital check image 201 according to the image comparison technique(s) applied by the image comparator 123, the image processing engine 120 may identify specific edge coordinates for the check image 210, such as the edge coordinates 301, 302, 303, 304 illustrated in FIG. 3. The modified digital check image 300 may be generated by the image processing engine 120 that includes the edge coordinates, where the modified digital check image 300 is included in the digital image file 135 transmitted to the institution system.

[0037] According to some embodiments, image processing, such as the edge detection, may be applied to segmented pieces of the digital check image 201. The grayscale converter 124 may convert the digital check image 201 generated by the image capture device 115 to grayscale using known techniques. In photography and computing, a grayscale digital image is an image in which the value of each pixel is a single sample, that is, it carries only intensity information. Images of this sort are composed exclusively of shades of gray, varying from black at the weakest intensity to white at the strongest. Conversion of a color image to grayscale is well known and any known technique(s) may be used.

[0038] The grayscale image generated by the grayscale converter 124 may be provided to the image segmenter 125, where the image segmenter 125 divides the image into a predetermined number of segments, such as 4 segments, 6 segments, 8 segments, or other predetermined number. Each segment may comprise a portion of the check image 210 (i.e., the check data) and a portion of the background image 250 (i.e., the background data) and an edge between the two portions. The segments may be equal in size or may differ in size. An example of a segmented image 400 divided into four segments is shown in FIG. 4.

[0039] The segment processor circuitry 126 may process each of the segments to identify and/or remove background data. In an implementation, histograms may be used to detect or identify background data and check data. The histogram of each segment can be used to determine an edge of the check image 210 from the background image 250 within the segment, so that the background image 250 may be removed or disregarded for subsequent processing.

[0040] A histogram is a well known graph and may be used to display where all of the brightness levels contained in an image are found, from the darkest to the brightest. These values may be provided across the bottom of the graph from left (darkest) to right (brightest). The vertical axis (the height of points on the graph) shows how much of the image is found at any particular brightness level. An example of a histogram 500 for a segment of an image comprising check data and background data is further described with respect to FIG. 5. An edge between the check image 210 and the background image 250 may be identified to be those points on the digital check image corresponding to the threshold point (i.e., low point) reflected on the histogram 500 shown in FIG. 5.

[0041] The segment recombiner 127 recombines the processed segments to generate a recombined image comprising a depiction of the check 107. Based on the recombined image, the image processing engine 120 may identify specific edge coordinates for the check depicted in the recombined image, such as the edge coordinates 301, 302, 303, 304 illustrated in FIG. 3. The segment recombiner 127 may process the recombined image (including the edge coordinates) to be included with the digital image file 135 transmitted to the institution system 205 for deposit of the check 107 depicted in the modified digital check image.

[0042] In an implementation, some or all of the image processing described as being implemented by one or more components of the image processing engine 120, may be performed by the institution system 205 after the institution system 205 receives the digital image file 135 from the computing device 109.

[0043] The image processor circuitry 128, alone or in combination with one or more of the components comprising the image processing engine 120, may operate to implement one or more image processing features such as to remove warping or dewarp a digital image, to crop a digital image, to deskew a digital image (e.g., rotate the image to horizontal or vertical alignments), to identify the corners/edges within a digital image, or adjust brightness, tint, contrast, or other image attribute of a digital image, or other known digital image processing capability.

[0044] The user 105 may thus generate a digital check image of the check 107 using the image capture device 115 on the computing device 109. For example, after endorsing the check 107, the user 105 may operate the image capture device 115 to capture a digital image of the front sides and/or back sides of the check 107 and store the digital image(s) as a digital image file on a storage memory of the computing device 109. The images of the front side and the back side of the check 107 may be processed using the techniques described herein. The images may be processed as separate files or as images in a single file. The images of the front side and the back side of the check 107 may be captured sequentially, e.g., pursuant to the user 105 flipping the check 107 over after an image of the front of the check 107 has been captured.

[0045] The digital image file 135 comprising the digital image depicting the check 107 may be transmitted to the institution system 205. The digital image may the original digital image captured by the image capture device 115, or a modified digital image having one or more image processing techniques applied. For example, the modified digital image may include the edge coordinates 301, 302, 303, 304 described herein. The computing device 109 transmits the digital image file 135, that may include secondary data, to the institution system 205 along with a request to deposit the check 107 into an account, such as the account 165. In an implementation, the user 105 may attach the digital image file 135 to an email, short message service (SMS) text, or other form of electronic message, and send the digital image file 135 to the institution system 205 using the computing device 109. However, other techniques for sending a digital image file 135 to the institution system 205 may be used, such as providing a digital image file 135 from storage to the website 218 associated with the institution system 205.

[0046] The institution 200 in conjunction with the institution system 205 may process the deposit request according to the digital image and any secondary data included in the digital image file 135. Thus, the institution 200 in conjunction with the institution system 205 may process the digital image file 135 comprising the digital image depicting the check 107 for deposit.

[0047] In an implementation, the institution system 205 may extract the check image 210 from the digital image file 135 and process the check 107 from the digital image for deposit, according to any one or more of the image processing techniques described herein. In addition or alternatively, the Any image processing technology, software, or other application(s) may be used to retrieve the digital image of the check 107 from the digital image file 135 and to obtain the relevant data of the check 107 from the digital image file 135. The institution system 205 may determine whether the financial information associated with the check 107 may be valid.

[0048] Upon receipt and processing of the digital image file 135 and approval of the check 107 associated therewith, the institution 200 may credit the funds of the check 107 to the account 165. It will be appreciated that the examples herein are for purposes of illustration and explanation only, and that an embodiment is not limited to such examples.

[0049] In an implementation, the computing device 109 may be a mobile device that comprises a digital camera which can capture a digital image of the check 107 by taking a picture of the front and/or back of the check 107. The back of the check may provide endorsement verification, such as the signature of the person or party the check is made out to. The user 105 may send the digital image file 135, including the check image, to the institution system 205 using the mobile device. An exemplary computing device 109 is described with respect to FIG. 6. It is contemplated that any device that is capable of generating a digital image may be used to make a digital image of the check 107 which may be processed for sending to the institution system 205 as a digital image file 135. Additional devices that may be used in the generation and/or transmission of a digital image include a digital camera, a photocopier, a fax machine, and the like, for example.

[0050] The institution system 205 may include any combination of systems and subsystems such as electronic devices including, but not limited to, computers, servers, databases, or the like. The electronic devices may include any combination of hardware components such as processors, databases, storage drives, registers, cache, random access memory (RAM) chips, data buses, or the like and/or software components such as operating systems, database management applications, or the like. According to an embodiment, the electronic devices may include a network-based server that may process the financial information and may receive the digital image file 135 from the computing device 109.

[0051] The electronic devices may receive the digital image file 135 and may perform an initial analysis on the quality of the image of the check 107 in the digital image file 135, the readability of the data contained therein, or the like. For example, the electronic devices may determine whether the amount payable and other information may be readable such that it may be obtained and processed by the institution system 205 to credit the account 165 associated with the user 105.

[0052] The institution system 205 may include a user interface circuitry 220, an image processor 222, and a data source access engine 227. The user interface circuitry 220 may generate and format one or more pages of content 219 as a unified graphical presentation that may be provided to the computing device 109 or a representative computing device 187. In an implementation, the page(s) of content 219 may be provided to the computing device 109 and/or the representative computing device 187 via a secure website 218 associated with the institution system 205.

[0053] In an implementation, the institution system 205 may use the image processor 222 to process the digital image file 135 comprising the image(s) 137 of the check 107 received from the user 105 for use by the institution 200 in the processing and/or clearance of the check 107. The image processor 222 may process multiple frames of the image if the image is comprised of multiple frames (e.g., the front side and the back side of a check).

[0054] For example, after receiving the digital image file 135 of the check 107, the image processor 222 may retrieve the image(s) 137 of the check 107 and process the image(s) 137, or modified image based on the image(s) 137, for deposit. The image processor 222 may use any known image processing software or other application(s) to obtain the image(s) 137 and any relevant data of the check 107 from the digital image file 135.

[0055] The image processor 222 has access to data, files, and documents pertaining to the user 105 as well as any other data, files, and documents that are internal or external to the institution system 205 that may be useful in processing the digital image file 135 and/or the data contained therein.

[0056] According to some embodiments, one or more components of the image processing engine 120 may be provided at the institution system 205 instead of the computing device 109. For example, one or more components of the image processing engine 120 may be included within the image processor 222 that is part of the institution system 205. Therefore, certain functions of the image processing engine 120 may be performed at the institution system 205 side as opposed to the computing device 109 side. In such implementations, certain functions attributed to the image processing engine 120 may be implemented by the computing device 109, while other functions attributed to the image processing engine 120 may be implemented by the institution system 205 on the image(s) 137 received from the computing device 109.

[0057] The institution system 205 has the ability to retrieve information from one or more data source 229 via the data source access engine 227. Data pertaining to the user 105 and/or the user account 165 and/or processing and clearing of a check may be retrieved from data source(s) 229 and/or external data sources. The retrieved data may be stored centrally in, for example, in a memory storage 208. Other information may be provided to the institution system 205 from the user 105 and/or the representative 185.

[0058] Data source 229 may contain data, metadata, email, files, and/or documents that the institution system 205 maintains pertaining to the user 105, such as personal data such as name, physical address, email address, etc. and financial data such as credit card numbers and checking account numbers. Such data may be useful for processing the digital image file 135. Additionally or alternatively, the institution 200 or the institution system 205 may access this information when processing or clearing a check.

[0059] The representative computing device 187, as well as the computing device 109, may provide access to a system which is coupled to the institution system 205. A system may be configured to format and transmit a graphical user interface to the representative 185 and/or the user 105, and through the graphical user interface provide the representative 185 and/or the user 105 the ability to interact with information that may be maintained, requested, and/or provided by the institution system 205. As mentioned above, the institution system 205 may provide a unified graphical presentation output. In an implementation, the unified graphical presentation is combined with other materials and transmitted to the representative 185 and/or the user 105.

[0060] A user access system may be implemented as a web server in an implementation. The user access system, through the use of any suitable interactive web technology, provides an interactive experience to the user 105 and/or the representative 185 through which access to check processing and clearing data and status and related data can be accomplished. Any technology that provides interactivity through a web browser is considered to be within the scope of the present discussion and may include, without limitation, Hyper-Text Mark-Up Language (HTML), Dynamic HTML (DHTML), JavaScript, and Ajax.

[0061] The institution system 205 may comprise one or more computing device(s) 206. The computing device(s) 206 may have one or more processor(s) 207, storage 208 (e.g., storage devices, memory, etc.), and software application modules 209. The computing device(s) 206, including processor(s) 207, storage 208, and software application modules 209, may be used in the performance of the techniques and operations described herein. A module described herein may include software, hardware, and/or circuitry for implementing the features attributed to the module.

[0062] Examples of software application modules 209 may include modules that may be used in conjunction with receiving and processing a digital image file 135 comprising the image(s) 137 of the check 107, retrieving data from the digital image file 135, generating web page content for display, and receiving instructions from the representative 185 or the user 105, for example. While specific functionality is described herein as occurring with respect to specific modules, the functionality may likewise be performed by more, fewer, or other modules.

[0063] FIG. 6 illustrates an exemplary computer architecture 600 for a computing device such as any one of the computing device 109, institution system 205, or another computing device. The computer architecture 600 includes system circuitry 602, display circuitry 604, input/output (I/O) interface circuitry 606, and communication interfaces 608. The graphical user interfaces (GUIs) 605 displayed by the display circuitry 604 may be representative of GUIs generated by the application for remote deposit. The GUIs may be displayed locally using the display circuitry 604, or for remote visualization, e.g., as HTML, JavaScript, audio, and video output for a web browser running on a local or remote machine such as the computing device 109 or institution system 205.

[0064] The GUIs 605 and the I/O interface circuitry 606 may include touch sensitive displays, voice or facial recognition inputs, buttons, switches, speakers and other user interface elements. Additional examples of the I/O interface circuitry 606 includes microphones, video and still image cameras, headset and microphone input/output jacks, Universal Serial Bus (USB) connectors, memory card slots, and other types of inputs. The I/O interface circuitry 606 may further include magnetic or optical media interfaces (e.g., a CDROM or DVD drive), serial and parallel bus interfaces, and keyboard and mouse interfaces.

[0065] The communication interfaces 608 may include wireless transmitters and receivers (“transceivers”) 610 and any antennas 612 used by the circuitry of the transceivers 610. The transceivers 610 and antennas 612 may support WiFi network communications, for instance, under any version of IEEE 802.11, e.g., 802.11n or 802.11ac, or other wireless protocols such as Bluetooth, Wi-Fi, WLAN, cellular (4G, LTE/A). The communication interfaces 608 may also include serial interfaces, such as universal serial bus (USB), serial ATA, IEEE 1394, lighting port, I2C, slimBus, or other serial interfaces. The communication interfaces 608 may also include wireline transceivers 614 to support wired communication protocols. The wireline transceivers 614 may provide physical layer interfaces for any of a wide range of communication protocols, such as any type of Ethernet, Gigabit Ethernet, optical networking protocols, data over cable service interface specification (DOCSIS), digital subscriber line (DSL), Synchronous Optical Network (SONET), or other protocol. The communication interfaces 608 may communicate with remote computing devices via a network, such as the communications network 140.

[0066] The system circuitry 602 may be representative of any combination of hardware, software, firmware, application programming interface, or other circuitry for implementing the features of the remote deposit application described herein. For example, the system circuitry 602 may be implemented with one or more systems on a chip (SoC), application specific integrated circuits (ASIC), microprocessors, discrete analog and digital circuits, and other circuitry. The system circuitry 602 may implement any of the image processing features described herein. As an example, the system circuitry 602 may include one or more processors 616 and memory 620.

[0067] The memory 620 stores, for example, control instructions 623 for executing the features of the remote deposit application running on the computing device 109, and/or institution system 205, as well as an operating system 621. In one implementation, the processors 616 execute the control instructions 623 and the operating system 621 to carry out any of the features for the remote deposit application. For example, the control instructions 623 for the remote deposit application includes instructions that, when executed by the processors 616, implement the features corresponding to the image processing engine 120. The memory 620 also includes control parameters 622 that provide and specify configuration and operating options for the control instructions 623, operating system 621, and other functionality of the computer architecture 600.

[0068] FIG. 7 shows a flow diagram 700 describing implementation of image processing features by the image processing engine 120 as part of a remote deposit solution. The image processing features described by the flow diagram 700 includes those relating to edge detection for determining coordinates of the edges of a check depicted in a digital image.

[0069] At 701, an account owner (e.g., payee to an amount written on a check document, referred to herein as a user) may receive a check from a third party (e.g., payor of the check), and may endorse the check by signing the back of the check in the designated field. If the user wishes to deposit the check into an account, such as a savings and/or checking account, the user may also write in an account number below the signature on the back of the check.

[0070] At 702, the user may operate the computing device 109 to open a communication pathway with the institution system 205 associated with an account of the user for depositing funds. The communication pathway may be implemented by logging into a website operating within the institution system 205, for example. In addition to, or alternatively, a remote deposit application may be downloaded onto the computing device 109. Executing the remote deposit application to run on the computing device 109 may initiate opening of the communication pathway. The communication pathway may be established after the user authenticates access to the institution system 205 using user specific authentication credentials such as, but not limited to, a username and a password, or biometric information such as fingerprints, voice recognition, or facial recognition information.

[0071] At 703, a digital check image of the check may be captured by the image capture device 115. The user may manually operate the image capture device 115 to capture the digital check image to include the check and a background portion. According to some embodiments, the remote deposit application running on the computing device may automatically capture the digital check image when a set of predetermined criteria is determined to be satisfied. For example, the remote deposit application may control the image capture device 115 to capture the digital check image after detecting the check is aligned within an alignment guide displayed on a display screen of the computing device 109, where the display screen displays a field of view of the image capture device 115. According to another example, the remote deposit application may control the image capture device 115 to capture the digital check image after detecting the check is in focus for the digital image capture by the image capture device 115. According to another example, the remote deposit application may control the image capture device 115 to capture the digital check image after detecting the check comprises at least a predetermined portion of the digital check image to be captured by the image capture device 115.

[0072] Also at 703, a deposit request is initiated. The deposit request may include selecting an account in which to deposit the check. In an implementation, the user may select a “deposit check” option provided on a graphical user interface (GUI) displayed on the computing device 109. The user may also input one or more of the following check information through the GUI: check amount, date, the account the check funds should be deposited in, comments, routing number, or other check data.

[0073] At 704, edge detection is executed by the image processing engine 120 according to any one or more of the edge detection processes described herein. The edge detection process includes distinguishing the check image from the background image that comprises the overall digital check image captured by the image capture device 115. By executing the edge detection on the digital check image, edge coordinates for the check image are determined. For example, the edge detection may determine the edge coordinates 301, 302, 303, 304 as shown in FIG. 3.

[0074] According to some embodiments, the edge detection implemented at 704 may include one or more of the following processes: scaling down the digital check image for faster processing (scale factor depends on image size), converting the digital check image to grayscale, using close and/or open morphs on the digital check image to eliminate noise, binarizing the digital check image using a dynamic threshold algorithm (e.g., otsu algorithm), running through multiple cropping logic algorithms, e.g., using OpenCV, until a “check” is found within the digital check image (e.g., 4 predetermined algorithms may be applied). To determine if a crop was successful, the crop result may be run through a crop validator that checks aspect ratio, area, skew, zoom, and/or other image analysis for validating a successful crop. The resulting digital check image may be run through focus detection to determine if the digital check image is blurry, and a refocusing process may be applied if blurriness is detected within a predetermined range. After multiple “successful” digital check images have been processed, the final digital check image is ready (e.g., considered a modified digital check image), along with four corner check coordinates and/or an image size of the modified digital check image.

[0075] At 705, a modified digital check image (e.g., modified digital check image 300) is generated that includes the edge coordinates (e.g., edge coordinates 301, 302, 303, 304 as shown in FIG. 3).

[0076] At 706, the modified digital check image may undergo one or more further image processing steps. The image processing steps may include any one or more of the image processing functions attributed to the image processor circuitry 128, as described herein.

[0077] At 707, the resulting processed image is stored within a digital image file. According to some embodiments, the digital image file may also include supplemental information, as described herein.

[0078] At 708, the digital image file is transmitted to the institution system 205.

[0079] At 709, the institution system 205 receives the digital image file. After receipt by the institution system 205, the institution system 205 may further process the digital image file, and deposit the funds corresponding to the check depicted in the digital image file. The digital image file may include the digital check image in a known digital image format (e.g., a jpeg digital image file). According to some embodiments, the institution system 205 may further apply edge detection to determine the edge coordinates of the check image. However, when the edge coordinates determined by the institution system 205 are determined to be unreliable, the institution system 205 may rely on the edge coordinates included in the digital image file as back up. The edge coordinates determined by the institution system 205 may be determined to be unreliable when certain predetermined conditions are satisfied, including, but not limited to, when an image quality of the digital check image is below a threshold level, when image capturing conditions (e.g., lighting levels) corresponding to the digital check image are below a threshold level, and/or when a threshold number of edge coordinates are not found within a threshold level of certainty.

[0080] According to some embodiments, the institution system 205 may skip edge detection and rely on the edge coordinates included in the digital image file. The edge coordinate information may be in the following exemplary formats (exemplary coordinates are provided):

{

[0081] “frontImageCoordinates”: [

• • [0082] 186.0,
  • [0083] 282.0,
  • [0084] 1368.0,
  • [0085] 294.0,
  • [0086] 1374.0,
  • [0087] 828.0,
  • [0088] 174.0,
  • [0089] 798.0

[0090] ],

[0091] “frontImageWidth”: “1440”,

[0092] “backImageWidth”: “1500”,

[0093] “frontImageHeight”: “1080”,

[0094] “backImageHeight”: “1000”

}

[0095] Further description of the processes that may be implemented on the digital image file by the institution system is described with reference to the flow diagram 800 shown in FIG. 8. According to the flow diagram 800 shown in FIG. 8, the institution system 205 receives the digital image file from the computing device 109 at 801.

[0096] At 802, edge detection is executed by the institution system 205 according to any one or more of the edge detection processes described herein. The edge detection process includes distinguishing the check image from the background image that comprises the overall digital check image captured by the image capture device 115. By executing the edge detection on the digital check image, edge coordinates for the check image are determined. For example, the edge detection may determine the edge coordinates 301, 302, 303, 304 as shown in FIG. 3.

[0097] According to some embodiments, the edge detection executed by the institution system 205 may be different from the edge detection executed by the image processing engine 120 of the computing device 109. For example, the edge detection at 802 may include one or more of the following: scaling the digital check image by a predetermined amount (e.g., scale down the digital check image by a half (½)), converting the digital check image to a gray scale image, using close and/or open morphs on the digital check image to eliminate noise, binarizing the digital check image using a dynamic threshold algorithm (e.g., Otsu's method), and/or applying Hough transform algorithm to identify lines in the digital check image. The edge detection at 802 may further include iterating through all the identified lines from the digital check image, and discarding away any lines that do not correspond to a check edge, where a check edge may be determined according to one or more of the following rules: vertical lines must be angled between 95 and 85 degrees or between 275 and 265 degrees, horizontal lines must be angled between 195 and 175 degrees or between 360 and 355 degrees, discard away lines that lie completely in the top fourth or bottom fourth of the image as these lines have a high likelihood of being image noise data, discard away lines that lie completely in the right fourth or left fourth of the image as these lines have a high likelihood of being image noise data, and/or find the average distance between every line and the center of the digital check image and throw away any outlier lines that are more than a predetermined distance from the center of the digital check image. The edge detection at 802 may further include creating a bounding box around remaining lines after the line removal process, as the remaining lines have a high likelihood of actually corresponding from the check depicted in the digital check image. If creating the bounding box around the remaining lines doesn't produce a check21 valid image, then another attempt to create a bounding box around contours of the digital check image may be implemented. The contours may be detected according to a contour detection algorithm such as the Suzuki85 algorithm.

[0098] At 803, conditions detected during the edge detection implemented by the image processing engine 120 may be compared to conditions detected during edge detection implemented by the institution system 205. The conditions may include any environmental conditions (e.g., brightness levels) that may have affected the quality of the digital check image captured by the image capturing device 115. In addition or alternatively, as the edge detection implemented by the institution system 205 may be different from the edge detection implemented by the image processing engine 120 of the computing device 109, an accuracy of the edge detection implemented by the image processing engine 120 may be compared to an accuracy of the edge detection implemented by the institution system 205.

[0099] At 804, the institution system 205 selects one of the edge coordinates from the edge coordinates included in the digital image file received from the computing device 109, or the edge coordinates determined by the institution system 205. The institution system 205 may select the edge coordinates based on predetermined criteria such as, for example, related to the detected conditions, or related to the determined accuracy of the edge coordinates. When the selection is based on the detected conditions, the institution system 205 may select the edge coordinates determined by the image processing engine 120 on the computing device 109 when the conditions indicate the edge detection processes implemented by the image processing engine 120 would result in a greater accuracy than the edge detection processes implemented by the institution system 205. When the selection is based on the accuracy of the edge detection, the institution system 205 may select the edge coordinates determined to be more accurate.

[0100] At 805, the institution system 205 identifies the check image from the digital image file based on the selected edge coordinates. From the identified check image, the check data contained within the check image may be extracted.

[0101] At 806, the institution system 205 processes the check depicted in the check image to extract the check data contained therein. Processing of the digital image file may include retrieving financial information regarding the check. The financial information may comprise the MICR number, the routing number, a check amount, or other information found on the check.

[0102] At 807, after retrieving the financial information from the check in an electronic data representation form, the institution system 205 determines whether the financial information is valid. For example, the institution system 205 may perform an initial analysis on the quality of the data representation, the readability of the data representation, or the like. For example, the electronic devices may determine whether the account number, amount payable, or the like may be readable such that they may be parsed and processed by the institution to credit an account associated with the user. If the financial information is determined to be valid, the electronic data representation may be processed by the institution system 205, thereby depositing the money in the user's account 165. If the financial information is determined to be invalid, then the user may be advised. For example, the institution system 205 may transmit an email, a web message, an instant message, or the like to the user indicating that the financial information associated with the electronic data representation may be invalid. The user may determine how to proceed by selecting an option on the web message, replying to the email, or the like.

[0103] FIG. 9 shows a flow diagram 900 describing implementation of image processing features by the image processing engine 120 included in the computing device 109 as part of a remote deposit solution. According to the embodiments illustrated by FIG. 9, the image processor circuitry further includes blur detection features. It follows that the image processing features described by the flow diagram 900 includes those relating to edge detection for determining coordinates of the edges of a check depicted in a digital image, as well as blur detection to determine when to capture the digital image.

[0104] At 901, an account owner (e.g., payee to an amount written on a check document, referred to herein as a user) receives a check from a third party (e.g., payor of the check), and endorses the check by signing the back of the check in the designated field. If the user wishes to deposit the check into an account, such as a savings and/or checking account, the user may also write in an account number below the signature on the back of the check.

[0105] At 902, the user may operate the computing device 109 to open a communication pathway with the institution system 205 associated with an account of the user for depositing funds. The communication pathway may be implemented by logging into a website operating within the institution system 205, for example. In addition to, or alternatively, a remote deposit application may be downloaded onto the computing device 109. Executing the remote deposit application to run on the computing device 109 may initiate opening of the communication pathway. The communication pathway may be established after the user authenticates access to the institution system 205 using user specific authentication credentials such as, but not limited to, a username and a password, or biometric information such as fingerprints, voice recognition, or facial recognition information.

[0106] At 903, a blur detection process is initiated by the image processor circuitry 128. The blur detection process is utilized to track and determine a blurriness level of a check included within a field of view of the image capture device 115. It follows that the blur detection process tracks the blurriness level of a check image during the focusing process of the image capture device 115. When a most “clear” (i.e., least blurry) image is detected within the field of view of the image capture device 115, the frame containing the most “clear” check image may be captured and considered for further processing.

[0107] According to some embodiments, a predetermined number of frames may be analyzed under the blur detection process. The predetermined number of frames may be set by a specific number of frames, or by analyzing frames for a predetermined amount of time, where the image processing engine 120 may determine a number of frames per second the image capture device 115 and the image processing engine 120 are able to process. For example, upon execution of the remote deposit application the image processor circuitry 128 may conduct an initial image processing test to determine an image processing capability of the image capture device 115. The image processing capability may be determined in terms of a minimum rate of frames that can be analyzed (e.g., frames/second) by the image capture device 115 and/or the image processing engine 120. Although the initial image processing test is described as being initiated upon execution of the remote deposit application, the image processing test may be initiated at other times while the remote deposit application is running. During the image processing test, the image processing circuitry 128 tests a number of frames the image capture device 115 and/or the image processing engine 120 are able to process for a given amount of time (e.g., rate of frames/second). The test results may determine a rate (frames/second) at which the image capture device 115 and/or the image processing engine 120 are able to process frames of check images that are within the field of view of the image capture device 115 during an image focusing process. Then based on this determination, the image processing engine 120 may set a predetermine length of time to analyze frames of images within the field of view of the image capture device 115 for blur detection. For example, the blur detection may consider 0.5 s worth of frames, and select the clearest frame during this period to capture as the check image for further processing.

[0108] The blur detection process may allocate a blurriness score to each of the analyzed frames to select the frame with a best score as the frame to capture as the check image. The blurriness score may be tied to a number count of straight lines that can be detected within a frame. FIG. 10 illustrates a flow diagram 1000 describing an exemplary process for determining such a blurriness score to be assigned to a frame.

[0109] At 1001, the check image portion of the frame is cropped to remove the background image portion, according to any of the embodiments disclosed herein.

[0110] At 1002, the check image is converted to grayscale, according to any of the embodiments disclosed herein.

[0111] At 1003, additional image processing may be applied to better help highlight the lines in the check image. For example, a Laplacian Edge Detection algorithm may be applied to the check image to better highlight the lines in the check image.

[0112] At 1004, a number of straight lines are counted in the processed check image. The straight lines being counted may be any straight edge detected in the processed check image.

[0113] At 1005, a standard deviation for the number of lines counted in the frame is calculated compared to the other frames being considered under the blur detection. The standard deviation calculation is then assigned as the focus value for the frame. This calculation may be applied to each frame being considered under the blur detection to assign each frame its own respective focus value. When a subsequent frame is determined to have a higher focus value, the previous frame with the lower focus value may be removed from the cache. When the subsequent frame is determined to have a lower focus value, the previous frame with the higher focus value may be retained in the cache for further comparison. In the end, the frame with the highest focus value score is determined to be the least blurriest (i.e., most clear).

[0114] Returning to flow diagram 900, at 904 a digital check image of the check is automatically captured by the image capture device 115 according to the results of the blur detection process. Also at 904, a deposit request is initiated. The deposit request may include selecting an account in which to deposit the check. In an implementation, the user may select a “deposit check” option provided on a graphical user interface (GUI) displayed on the computing device 109. The user may also input one or more of the following check information through the GUI: check amount, date, the account the check funds should be deposited in, comments, routing number, or other check data.

[0115] At 905, edge detection is executed by the image processing engine 120 according to any one or more of the edge detection processes described herein. The edge detection process includes distinguishing the check image from the background image that comprises the overall digital check image captured by the image capture device 115. By executing the edge detection on the digital check image, edge coordinates for the check image are determined. For example, the edge detection may determine the edge coordinates 301, 302, 303, 304 as shown in FIG. 3.

[0116] According to some embodiments, the edge detection implemented at 905 may include one or more of the following processes: scaling down the digital check image for faster processing (scale factor depends on image size), converting the digital check image to grayscale, using close and/or open morphs on the digital check image to eliminate noise, binarizing the digital check image using a dynamic threshold algorithm (e.g., otsu algorithm), running through multiple cropping logic algorithms, e.g., using OpenCV, until a “check” is found within the digital check image (e.g., 4 predetermined algorithms may be applied). To determine if a crop was successful, the crop result may be run through a crop validator that checks aspect ratio, area, skew, zoom, and/or other image analysis for validating a successful crop. The resulting digital check image may be run through focus detection to determine if the digital check image is blurry, and a refocusing process may be applied if blurriness is detected within a predetermined range. After multiple “successful” digital check images have been processed, the final digital check image is ready (e.g., considered a modified digital check image), along with four corner check coordinates and/or an image size of the modified digital check image.

[0117] At 906, a modified digital check image (e.g., modified digital check image 300) is generated that includes the edge coordinates (e.g., edge coordinates 301, 302, 303, 304 as shown in FIG. 3).

[0118] At 907, the modified digital check image may undergo one or more further image processing steps. The image processing steps may include any one or more of the image processing functions attributed to the image processor circuitry 128, as described herein.

[0119] At 908, the resulting processed image is stored within a digital image file. According to some embodiments, the digital image file may also include supplemental information, as described herein.

[0120] At 909, the digital image file is transmitted to the institution system 205. At 910, the institution system 205 receives the digital image file. After receipt by the institution system 205, the institution system 205 may further process the digital image file, and deposit the funds corresponding to the check depicted in the digital image file. The digital image file may include the digital check image in a known digital image format (e.g., a jpeg digital image file). According to some embodiments, the institution system 205 may further apply edge detection to determine the edge coordinates of the check image. However, when the edge coordinates determined by the institution system 205 are determined to be unreliable, the institution system 205 may rely on the edge coordinates included in the digital image file as back up. The edge coordinates determined by the institution system 205 may be determined to be unreliable when certain predetermined conditions are satisfied, including, but not limited to, when an image quality of the digital check image is below a threshold level, when image capturing conditions (e.g., lighting levels) corresponding to the digital check image are below a threshold level, and/or when a threshold number of edge coordinates are not found within a threshold level of certainty.

[0121] According to some embodiments, the institution system 205 may skip edge detection and rely on the edge coordinates included in the digital image file. The edge coordinate information may be in the following exemplary formats (exemplary coordinates are provided):

{

[0122] “frontImageCoordinates”: [

• • [0123] 186.0,
  • [0124] 282.0,
  • [0125] 1368.0,
  • [0126] 294.0,
  • [0127] 1374.0,
  • [0128] 828.0,
  • [0129] 174.0,
  • [0130] 798.0

[0131] ],

[0132] “frontImageWidth”: “1440”,

[0133] “backImageWidth”: “1500”,

[0134] “frontImageHeight”: “1080”,

[0135] “backImageHeight”: “1000”

}

[0136] It should be understood that the various techniques described herein may be implemented in connection with hardware or software or, where appropriate, with a combination of both. Thus, the methods and apparatus of the presently disclosed subject matter, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the presently disclosed subject matter. In the case of program code execution on programmable computers, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. One or more programs may implement or use the processes described in connection with the presently disclosed subject matter, e.g., through the use of an API, reusable controls, or the like. Such programs may be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language and it may be combined with hardware implementations.

[0137] Although exemplary embodiments may refer to using aspects of the presently disclosed subject matter in the context of one or more stand-alone computer systems, the subject matter is not so limited, but rather may be implemented in connection with any computing environment, such as a network or distributed computing environment. Still further, aspects of the presently disclosed subject matter may be implemented in or across a plurality of processing chips or devices, and storage may similarly be spread across a plurality of devices. Such devices might include personal computers, network servers, and handheld devices, for example.

[0138] Various implementations have been specifically described. However, other implementations that include a fewer, or greater, number of features for each of the apparatuses, methods, or other embodiments described herein are also possible.