Augmented, virtual and mixed-reality content selection and display for billboards

[0001] This application is a continuation in part of application Ser. No. 17/228,821 filed Apr. 13, 2021, which is a continuation of application Ser. No. 16/895,637 filed Jun. 8, 2020 now U.S. Pat. No. 10,997,419; which is a continuation of application Ser. No. 16/565,234 filed Sep. 9, 2019 now U.S. Pat. No. 10,699,124.

[0002] This application is also a continuation in part of application Ser. No. 17/228,814 filed Apr. 13, 2021, which is a continuation of application Ser. No. 16/893,236 filed Jun. 4, 2020 now U.S. Pat. No. 10,997,418; which is a CIP of Ser. No. 16/565,234 filed Sep. 9, 2019 now U.S. Pat. No. 10,699,124; and which claims the benefit of 62/966,503 filed Jan. 27, 2020.

[0003] This application is also a continuation in part of U.S. patent application Ser. No. 17/641,300 filed Mar. 8, 2022, which is a US national phase of PCT International application no. PCT/IB2020/058386 filed 9 Sep. 2020, which is a continuation in part of application Ser. No. 16/893,236 filed 4 Jun. 2020 and is a continuation in part of application Ser. No. 16/565,234 filed 9 Sep. 2019 and claims the benefit of application No. 62/966,503 filed 27 Jan. 2020.

[0004] Each of these applications is incorporated herein by reference in its entirety as if expressly set forth.

[0005] None.

[0006] Augmented reality (“AR”) is the real-time superimposition of artificial images onto real world images. This technology has been used for a variety of applications from handheld entertainment to heads-up displays on military jets. For example, a person using an augmented reality-enabled display device (e.g., a smartphone with an AR app or AR enabled glasses) while shopping can be shown virtual advertisements, sale announcements, pricing information etc. superimposed onto images of actual products they can purchase.

[0007] Because AR tends to drive higher levels of attention, one advantageous use of augmented reality is superimposing advertisements and/or purchasing information over product images. People who might ignore traditional electronic advertisements may pay attention to an AR advertisement.

[0048] This technology provides systems, methods and techniques for automatically recognizing two-or three dimensional real world objects such as printed objects (e.g., 3 dimensional with any kind of print as example on the package of a product) with an augmented reality display device. The printed object (which may comprise a traditional printed advertisement or other printed item) can be recognized and used to trigger a superimposed digital overlay, which then opens up new options such as adding videos, live streams, buying options or other information and content. The AR feature can thus be used to augment or enhance the display of such real world objects by superimposing virtual images such as a still or video advertisement, a story, a tutorial, specific information, a shopping opportunity, or other virtual image presentation.

[0049] In the near future, when AR becomes really popular with any kind of glasses or other viewing arrangement, we might also see at least a partial replacement of the smartphone. That means that we will see many new use cases that use AR or mixed reality to provide users or customers with new information, any kind of content or buying options triggered by the recognition of a print that could be a 2 or 3 dimensional object. For example, if you see a job advertisement, with your AR glasses or a handheld device you can watch a related video or activate a call button to talk to somebody from human resources (HR) of that company. When you see a travel advertisement, you point the camera or other viewing device on this and you can watch a video of the destination, a buy option or different dates etc. or talk to a travel agent. When you see an advertisement of a product, you can decide to watch a related video, get other product information or buy it. This same scenario can exist with a product or service manual that could be the trigger of specific information that will allow you to understand how a product works by giving you technical drawings etc. This same scenario could apply in the real estate market where you can see a video of the object, specific information such as the price or price history, the neighborhood, floorplan, a call button to call the agent etc. This same scenario could apply with tickets where there is a city light as a trigger and when the AR kicks in, you can buy this ticket for a concert, a sport event etc. It could be that specific information about that concert or event can be provided to motivate customers to purchase. That could be videos or other specific information about players, a track record, etc. In one example embodiment, a device enables or allows superimposition of a digital overlay that leads to specific action such as providing specific information; providing videos, tutorials, or any kind of content; providing options to buy; or other options.

[0050] Another use case could be in a vehicle, where the passengers see digital advertisements on their screens or windows triggered by the recognition of advertisements like posters, citylights etc. that had been recognized by a camera that is attached to the vehicle.

[0051] In some non-limiting embodiments, the real world object may include visible features including visible security features and a recognition process takes the visible security features into account when recognizing the object and/or displaying superimposed virtual images. But security features are not needed or used in other example embodiments herein.

Example Use Case—Billboard Information System

[0052] FIG. 1 shows an example billboard information system. In the example shown, a smart device such as a smart phone or smart glasses captures and recognizes printed information from a real world object related to one or more billboards. The real world object could comprise a two-dimensional painted or printed billboard by a roadside, an electronic billboard on a building, a jumbotron in a sports stadium, or any large human-created image that is displayed on projected on a large or expansive surface. In one embodiment, a billboard comprises a flat surface as of a panel, wall, or fence on which bills are posted and specifically a large panel designed to carry outdoor advertising. Billboards are one of the oldest forms of display marketing. During the early 1800s, local merchants would display signs or posters to help bring passerby traffic into their stores. As the automobile industry took off in the early 1900s, so too did the opportunity for billboard expansion across the growing road system. Especially in the early days, billboard advertising wasn't exclusively used by companies. One use was to help further spread specific causes, especially during times of war. Take the popular “I Want You For U.S. Army” poster created by James Montgomery Flagg in 1917 to help to recruit during World War 1. His use of the iconic “Uncle Sam” became widely known following the creation and distribution of this advertisement. As corporations saw the power these display marketing tools could have, they also became heavily involved. Take the 1931 Coca-Cola holiday billboard campaign that featured what became the modern-day interpretation of Santa Claus. This display of branding became a copied system for brands for many years to come. As technology capabilities have increased, so have billboard designs and their creativity. Today, there are two main types of billboard delivery systems: Static and Digital. Static billboards are the more traditional type, where a single non-interactive image is displayed. Although basic in function and sometimes design, depending on the design these can be incredibly powerful due to their simplicity in delivering their branded message. Digital billboards utilize technology to add even more content to the billboard display. These billboards vary significantly in regards to their style, length, and overall feel. What does remain consistent throughout is their use of creativity.

[0053] As described below, the system will automatically recognize the real world billboard object or a pattern it is currently displaying, and match it up with one or more entries or records such as in a database that pertain to a particular item or items associated with or comprising the real world object. Such a database entry or record may be stored in the cloud and accessed over an electronic wireless network or in some cases an electronic or digital wired network.

[0054] Within this system, several buttons can be shown on a mobile phone display or a monitor or on AR Glasses. In this billboard example, each of the buttons shows a call for action. For example, information pertaining to a good or service advertised by the billboard can be displayed or otherwise presented. Within the use case, several buttons can be shown on a mobile phones display or a monitor, contact lens or on AR Glasses. Pointed on or looking at a billboard, AR will be displayed which can show multiple information, videos, shop links, links or specific calls to action—depending on the product, service or topic that is shown on the billboard.

[0055] In more detail, once a billboard is recognized, the system can display any or all of the following action buttons in any combination to allow a user to access selected kinds of further information related to or associated with the billboard or a good or service depicted by or described by the billboard or otherwise associated with the billboard:

• • [0056] 1. Price Tag
  • [0057] 2. Photo Gallery
  • [0058] 3. Videos
  • [0059] 4. Description
  • [0060] 5. Call
  • [0061] 6. Mail
  • [0062] 7. Shoplink
  • [0063] 8. Explanation
  • [0064] 9. Intro
  • [0065] 10. Social Media links
  • [0066] 11. Map
  • [0067] 12. Discount Codes
  • [0068] 13. Reviews
  • [0069] 14. Tutorials
  • [0070] 15. Directions
  • [0071] 16. Test drives
  • [0072] 17. Booking opportunities
  • [0073] 18. Other.

[0074] In this example, selecting the “price tag” button causes display of or otherwise presents information relating to the financial aspects of a transaction relating to the recognized billboard. The information that is displayed could comprise for example asking price, price history, prices of comparable items, prices of the item at different retail outlets, down payment or security deposit, financing options, estimated transaction costs, comparable value, tax liability or payments, reputation of the supplier/manufacturer, other.

[0075] Selecting the “photo gallery” button can cause display of one or more photographs or other images such as orthographic or panoramic images relating to or showing the goods or services associated with the billboard. There can be interior views, exterior views, perspective views, birds eye views, map views, architectural drawings, plan view drawings, neighborhood photos, 3D photos, views of food, views of people using the product or enjoying the service, other photos.

[0076] Selecting the “Videos” button can cause display of one or more videos or other moving picture presentations pertaining to the goods or services or other item such as a video tour, a sales pitch, testimonials, stories, music videos, dance videos, documentaries, or other.

[0077] Selecting the “Description” button can cause display or other presentation of textual or written information or audio information describing characteristics of the goods, services or other items associated with the recognized billboard. Such information can include a description written by a seller describing salient features of the item or items associated with the billboard such as price or cost, size, weight, availability, location, ingredients, operating hours, dietary information, plot description, performance nights, reputation information, etc.

[0078] Selecting the “Call” button can cause the smart device to place a voice call, voice chat or text chat with a seller involved in selling or offering a good or service associated with or advertised by the billboard.

[0079] Selecting the “Mail” button can cause the smart device to generate an email, text, tweet, chat or other asynchronous electronic communication to send to the seller or offeror related to the billboard or to invite such an individual or entity to contact the device by electronic or physical mail.

[0080] Selecting the Shoplink button can cause the smart device to shop for a good or service related to a message on the billboard. For example, if a billboard displayed an advertisement for Calvin Klein® jeans, the Shoplink button could cause the system/device to provide an entryway into shopping for those particular jeans e.g., by supplying a map list of nearby stores and/or online outlets that carry those jeans. Similarly, if the billboard displayed a political campaign message, selecting the “Shoplink” button could bring up an online opportunity to financially or otherwise support the political campaign. And if the billboard showed a car or a washing machine, the “Shoplink” button could bring up sales outlets who could offered the car or washing machine for sale.

[0081] Selecting the “Explanation” button can cause the smart device to provide an explanation with regards to the item or service shown on the billboard. This could focus on all kinds of information such as history or reputation of the manufacturer, product ratings, list of product features, etc.

[0082] Selecting the “Intro” button can cause the smart device to provide an introduction of the item or service indicated on the billboard with some crisp facts and figures. It can also be used for an introduction of the product, the manufacturer or the seller.

[0083] Selecting the “Social Media Links” button can cause the smart device to initiate contact(s) via social media with the seller or other individuals associated with or related to buying or selling or offering the product indicated on the billboard, display community or neighborhood social media pages, user groups, or the like.

[0084] Selecting the “Map” button can cause the smart device to display a map showing the geographical locations of the retail outlet, concert venue, campaign headquarters, or other venue or location associated with a product and/or service indicated on the billboard.

[0085] Selecting the “Discount Codes” button can bring up a display of discount codes or coupons that could be applied for an online or in-person purchase of a good or service indicated on the billboard.

[0086] Selecting the “Reviews” button can bring up on the smart device a listing of reviews posted by others in connection with the product or service shown on the billboard. For example, the “Reviews” associated with a billboard advertising a restaurant could be of other customers of that restaurant commenting on the food sold by the restaurant. The “Reviews” associated with a billboard advertising a water park could show posted online reviews of others who have visited the water park. Note that these reviews could have been written and posted before the billboard had been erected or posted.

[0087] Selecting “Tutorials” could bring upon on the smart device one or more tutorials concerning how to use, operate or experience the product or service indicated on the billboard.

[0088] Selecting “Directions” could bring up on the smart device driving or walking directions to get to an outlet associated with the billboard, e.g., a restaurant, car dealership, water park, sports arena, etc.

[0089] Selecting “test drives” could bring up on the smart device a series of videos or other presentations showing other users using the product or service shown on the billboard, or it could in some embodiments enable the user of the smart device to schedule an opportunity to “test drive” the product or service.

[0090] “Booking opportunities” when selected could bring up on the smart device an opportunity to book an experience or accommodation such as a cruise, a hotel room, a theater performance, etc.

[0091] As discussed below, the presentation of any or all such information may be by superimposing the information onto an image of the recognized real world object that is inscribed with printed material. Thus, the user of the smart device can view the image of the billboard on their smart device with information superimposed thereon.

[0092] Such additional information presented as described above may be electronically retrieved and presented by the smart device from a database, server or other source in the cloud. In particular, FIGS. 18A and 18B show one example non-limiting implementation. The FIG. 18A system shows how a handheld or worn smart device 1002 can communicate via a network 1005 with a variety of resources such as an image recognition server(s) 1006, an information server(s) 1008, a social media server(s) 1010, a streaming video server(s) 1012, a virtual object server(s) 1014, and other resources. As FIG. 18B shows how the handheld or worn smart device 1002 can capture an image (block 1102) and send captured parameters and/or images to the image recognition server(s) 1006 for processing (block 1104). In example embodiments, the system translates every captured image into shades of grey or color plus have an area of recognition that can be defined as the whole image or just a part or several parts of the image. In example embodiments, the billboard can provide a static display or a changing display comprising a sequence of images, and the example embodiment recognizes any one or of a sequence of such images.

[0093] The image recognition server(s) 1006 recognizes the captured image and sends back an identifier to the smart device 1002 (block 1106). This identifier in one embodiment can comprise more than one identifier that allows the smart device 1102 to provide indices or references to allow the smart device to request related information from other services shown in FIG. 18A. For example, the image recognition server(s) 1006 may send back URLs, geographical address strings, zipcodes, neighborhood names, geocoordinates, search strings or codes, or other identifiers. The smart device 1102 can use such information to pull information relating to the item 1104 identified by the image recognition server(s) 1006 from the various servers 1006, 1008, 1010, 1012, 1014 based on which button shown in FIG. 1 a user selects. Which button is selected thus determines not only the type of information the smart device 1102 requests, but also the source of the information.

Example Use Case—Financial or Other Instrument

[0094] FIG. 1A shows a further example non-limiting process. At 10, a printed, engraved, stamped or other object 102 such as a banknote, coin or ID (see FIGS. 2A-2J) is scanned to produce a digital copy. Such scanning can be performed with a camera, a CMOS light sensor, a document scanner, or any other conventional device that creates a digital image of the printed object.

[0095] The printed object may include visible features such as distinctive images of persons, buildings, etc. In the example shown, the printed object further includes an example visible security feature S represented here as a vertical band. The scanned digital copy will include the distinctive image(s) as well as the visible security feature S.

[0096] At 20, the resulting digital copy of the printed object is uploaded into a database, and enhanced to provide a selected associated overlay comprising: (a) self-starting video(s) and/or video(s) that start when you physically press a button that has been programmed to show up on the surface of a device like a mobile phone, iPad etc., or activate this activity by voice, movements of your head and/or gesture; (b) button(s) that when selected have different interactive functions such as “tell a story”; and (c) interactive connection to an online shopping experience e.g., to show a store finder (see 50).

Overlay can be Changed Dynamically

[0097] We can change the digital “overlay” quickly if needed (e.g., to change from one advertising campaign to another) or update content, price tags, videos or other information. The technology of embodiments herein allows an update of the digital overlay in a split second—which is must faster as compared to many other systems. The disclosed embodiments allow split seconds updates by uploading a new digital overlay or even a new button into the data base. With other technologies that used a QR code, a particular media content was tied to the QR code so the distributor of the content would have to print a new QR code on a new flyer to play out a different experience. With the system of embodiments herein, you just change either the whole digital overlay or a part or some parts of the digital overlay and as soon as the camera detects a print that is connected with this image stored in the data base, the user see the updated overlay in a split second. No hidden pattern, print etc. is needed.

Old or Ancient Media can be Retrofitted

[0098] By way of further explanation, if you would use a piece of print that is 50 years old, it would not contain a QR code, a hidden signature etc. Therefore it would be impossible to use this to play out an AR experience using systems that require a bar code. With the technology herein in some embodiments, you upload the old print into the database and connect a digital overlay to it. As soon as this has happened, the print stored in the database will match with the Cameras recording and the digital overlay will be played. This means that publications of any vintage (e.g., older magazines, books, flyers, etc.) can also be recognized and used to provide modern media overlays.

Workflow to Update AR Experience

[0099] Another differentiator is the workflow to update an AR experience. All systems that work with a QR code FIG. 1E) or hidden signatures (FIG. 1D) need either a new QR code or another print to connect with a new, updated experience. With the technology herein, you change or update the digital overlay and leave the original print untouched (see FIG. 1F). That means that you can update an experience without producing a new QR code or print new prints with a new hidden signature.

Area of Recognition

[0100] To activate that overlay, we determine an “area of recognition” as indicated by a cross-hatched area (30). In one example embodiment, the area of recognition is determined at least in part by the presence, position, dimensions and/or orientation of one or more security features S. For example, the area of recognition may be defined to exclude the security feature S on the printed object. As an example, in the case of a 20 Euro note (see FIG. 2E) including a vertically-oriented silver holographic band which appears differently depending on the viewing angle, the “area of recognition” may defined to exclude the silver holographic band so the system is able to recognize the object even without a silver line. Similarly, the area of recognition in the case of a $100 US banknote shown in FIG. 2C may be defined to exclude security features B, C and E since at least some of these features may appear differently depending on viewing angle (e.g., the ink color changes from gold to green as viewing angle changes). Other security features such as serial numbers, bar codes etc. vary from one printed document to another, so that using such indicia will result in no recognition of a banknote of the same denomination but having a different unique serial number and/or unique bar code (i.e., the purpose of the recognition may be to recognize a particular kind of banknote, not that specific one-of-a-kind banknote). Excluding such security features from the area of recognition makes recognition more robust because it is not subject to such constraints.

[0101] The recognition of a specific area then triggers a digital overlay that includes defined features (video, information, content, shopping, etc.) that show up and can be used by a person to get information, see a video, shop etc.

Optional Recognition of Security Feature(s)

[0102] In another embodiment at 40, at least some security features S are included in the defined area of recognition. The object is recognized only if it includes the security feature(s). If there are visible security features, we can include them into our recognition program. When we upload the digital copy of a print, we can decide which area of the print is used for recognition. See FIG. 1C. When this area of recognition includes an example such as the silver line of a banknote, the note cannot be recognized when the silver line is missing. In this embodiment, the security feature S is used as a feature of recognition to increase the reliability of the recognition to recognize distinctive documents, while designing the recognition algorithm to take different viewing angles into account. For example, it is possible to include the position and orientation of a silver holographic band in an area of recognition without requiring the holographic contents of the band to be recognized. Similarly, it is possible to include the position of serial numbers or bar codes in an area of recognition without requiring the specific serial numbers or bar code contents to match.

Overlapping Areas of Recognition

[0103] Some embodiments provide plural overlapping areas of recognition for the same object; one area of recognition may exclude certain security features and another area of recognition includes those security features. The plural different overlapping areas of recognition can be applied sequentially or simultaneously to increase recognition reliability. A voting algorithm can be used to select positive matches.

Real Time Recognition Database

[0104] In example non-limiting embodiments, the database enables real time recognition of an image captured by a user. For example, if the user captures an image of a 20 Euro note, a matching algorithm is used to determine a positive match if the database contains a digital copy of a 20 Euro note. In example non-limiting embodiments, the matching algorithm can include pattern recognition techniques such as described in Conti et al, Multiresolution image registration, Proceedings., International Conference on Image Processing (IEEE 23-26 Oct. 1995); Hasanuzzaman et al, Robust and effective component-based banknote recognition by SURF features, 20th Annual Wireless and Optical Communications Conference (IEEE 15-16 Apr. 2011); Doush et al, Currency recognition using a smartphone: Comparison between color SIFT and gray scale SIFT algorithms, Journal of King Saud University—Computer and Information Sciences Volume 29, Issue 4, October 2017, Pages 484-492.

[0105] As will be understood by those skilled in the art, the database 20 could but need not contain the captured images themselves. For example, in some embodiments, the database might contain compressed or other feature sets used for comparing with captured photos, such feature sets for example comprising a listing of coordinates and associated image features to thereby reduce storage requirements and increase recognition speed. Similarly, when a smart device captures an image, instead of uploading the entire image it may analyze the image and upload a compressed format such as coordinates of pattern features. Since the purpose of recognition in the example non-limiting embodiment is not to determine whether the banknote or other printed item is authentic and genuine, the matching/recognition standards can be significantly relaxed and thus quite different as compared to conventional banknote scanners/recognizers.

Other Pattern Recognition Techniques

[0106] Some example embodiments use artificial intelligence and machine learning to perform the matching. The training set consists of images captured by various smartphones and other user devices. Other embodiments use simpler pattern recognition algorithms based on color and/or grey scale information in the area(s) of recognition. Any given object can have at least one and as many as N different discrete areas of recognition, where N is a positive integer.

Flexible Definition of Areas of Recognition

[0107] Example embodiments provide techniques to flexibly define areas of recognition. Example embodiments also do not need a restricted area of recognition. The technology described herein can cover certain points within the print, the whole object, parts of the objector or all of this. Some other technologies cannot identify an object if a restricted area of recognition is covered, as for example by a price tag. See FIG. 4D, 4E. Embodiments herein may simply ignore this because they do not need a restricted area of recognition. See FIG. 4F.

[0108] The technology herein allows a wide variation in the definition of an area of recognition. It could be very small or as big as the object itself, it could be just one or several areas etc. Such technology also allows recognition of an item if it has different sides, like a 3D object. That means that if multiple or all sides of a 3D object are imprinted with different print on each side, we can define an area of recognition for each side to recognize the specific object even if it shows a random or unpredictable side when the user points a camera at it. A repositioning to successfully detect and recognize the object is generally not needed.

Example Cloud Based System

[0109] FIG. 3 shows an example non-limiting system 200 including a user device such as a smartphone 202, AR glasses, etc. which includes a camera or CMOS sensor 204, a processor 206, a memory 208, a display 210 and user input/output devices 212 such as a touch screen, push buttons, etc. The processor 206 communicates wirelessly or through a wire via a network 214 with a server 216 which may store the database 20 and in some embodiments perform the image recognition. The app may comprise executable instructions that the device 202 downloads from the server 216, stores in memory 206, and executes in response to user selection of input/output devices 212.

[0110] From a user perspective, as shown in FIG. 1A, after downloading an app onto their smart phone, smart glasses or other portable or non-portable device 202, the customer can point the device's camera onto a print, physical object or a digital copy on the screen of his tablet, computer, etc. (100; FIG. 4 blocks 302, 304, 306). This can be the camera of a device like an iPhone®, a cars/vehicles camera or an external camera or a camera that is a part of AR glasses. This causes the device 202 to display a photographically captured image on display 210 (see FIG. 5A, 6A). It is possible in some embodiments for device 202 to obtain this captured image some other way than photographing an actual object 102, such as via network 214, e.g., by downloading the image through Instagram or other photo sharing service, by receiving the image via a text (SMS) message, by receiving the image from another device in proximity (e.g., using Bluetooth pairing, etc.) or by other means.

[0111] Once the app has been activated to recognize the captured image, the app connects with the database 20 on server 216 via network 214 and checks if this print is recognized (FIG. 4 blocks 308, 310). If recognition is successful (branch 110), an action is enabled (e.g., playing a video, telling a story and/or showing a picture, showing shoplinks or other information) (FIG. 4, blocks 312, 314). If recognition is unsuccessful (branch to 120), no action is enabled or a different action (“xyz”) may be performed instead.

Virtual Image Anchoring

[0112] In one non-limiting example, the app causes the smartphone or other device 202 to show a picture or video by anchoring it to the image currently being captured by the device's camera 204 and displayed on the device's display 210. See FIG. 5B, 6B. If the user moves the device slightly to change the position or orientation of the printed item on the device's's display 210, the picture or video will move with the image of the printed item on the display as if it were glued or otherwise adhered to the printed item or were being displayed by the printed item itself from a magic display window on the printed item. For example, the superimposed image can change position, orientation and/or perspective as the underlying real world object changes position, orientation and/or viewing angle in order to provide a photorealistic image in which the superimposed image appear to be part of the underlying real world image. See for example FIG. 6B. This augmented reality feature provides a sense of wonder or surprise that is designed to capture the attention of the user. Instead of appearing to be a “flat” advertisement, the picture or window appears to be much more interactive and interesting and thus captures the user's attention. The augmented reality image can be animated and can be accompanied by audio and/or tactile sensations as indicated in FIGS. 5A, 5B, 6A, 6B.

[0113] Mixed Reality and/or Virtual Reality

[0114] In other embodiments, so-called “mixed reality” or “virtual reality” can be used instead of or in addition to augmented reality. Generally speaking, “virtual reality” presents the user with an image of an entirely virtual/non-real world. The posture (orientation and/or position) of the device may be sensed (e.g., using inertial sensors such as accelerometers and/or gyrosensors; magnetic position and/or orientation sensors; RF beacon sensors; optical sensors providing an optical “flow” image analysis; and other known conventional technologies. Different portions of the virtual world may be displayed based on such sensed posture.

[0115] Other embodiments can use so-called “mixed reality” (MR) that mixes virtual and real scenes. Such an MR experience is one where the user is placed in an interactive setting that is either real with virtual asset augmentation (augmented reality), or virtual with real-world augmentation. See Ohta et al, Mixed Reality: Merging Real and Virtual Worlds (Springer Publishing 2014); Milgram et al, A Taxonomy of Mixed Reality Visual Displays, IEICE TRANSACTIONS on Information and Systems Vol. E77-D No. 12 pp. 1321-1329 (Dec. 25, 1994); Hughes et al, Mixed reality in education, entertainment, and training, IEEE Computer Graphics and Applications (Volume: 25, Issue: 6, November-December 2005).

Further Use Cases—Two Dimensional Printed Objects

[0116] In more detail, a user can aim the camera of a smartphone, smart glasses or other electronic device at any two-dimensional or three-dimensional real world object or a likeness of such an object. One example type of a two-dimensional object could be a portable, wallet-sized planar object such as a banknote, driver's license, passport, printed advertisement, real estate advertisement or other official or unofficial identification document examples of which are shown in FIG. 2A-2I. Other examples of such objects might include for example a real estate advertisement, a human resources want ad, and objects related to a shopping experience. Using the technology with AR Glasses instead of a mobile phone means, that you do not touch a button of the superimposed digital image on your screen but press either a virtual button to shop or get information or just say that you want to press a specific button or say that you want to see a video or say that you want to buy a merchandise that is offered within the digital overlay,

Real Estate Printed Object

[0117] For example, the object as shown in FIG. 7 may comprise a real estate advertisement which can show one or several houses, lots, commercial buildings, private properties etc. An example area of recognition may constitute a single advertisement or a portion of an advertisement (see FIG. 8). Image recognition may be used to recognize the particular real estate property, and such recognition may be used to select and initiate display of a media item that provides more images and information concerning that property.

[0118] These information can be floor plans, videos of the property, contact data of the agent, the lot size, drone videos, the actual price tag or any other information that are relevant for the seller, the buyer or the agent.

Shopping or Travel Advertisement Printed Objects

[0119] The same with any shopping or travel advertisements where you define an area of recognition which could be a part or parts of that advertisement or even the full page to trigger a superimposed image that contains videos, shoplinks or other buttons that provide other content or information and allow to buy merchandise, travel related services (flight, hotel, tour, cruise, travel package, insurance and/or watch, read related information.

Packing Printed Objects

[0120] Another example for a use case is related to packaging, where you define an area of recognition or several on a 3 dimensional object. After a match in the database you can add product information, videos, or any other content or buy the specific merchandise.

Employment Advertising Printed Objects

[0121] As another example, the object shown in FIG. 9 is an employment advertisement. Image recognition processes may be used to recognize the advertisement. Successful recognition can be used to select and cause display of a media item that provides more information and images concerning the employment opportunity. For example, there is a strong use case where you point the camera on a standard. HR advertisement and then you see, for example, the CEO or the HR Manager of the company describing that specific job.

Facial Recognition

[0122] As yet another example, the object may comprise the face, head or body of a person or animal. Image recognition processes may be used to recognize the person or animal. Successful recognition can be used to select and cause display of a media item that provides more information and images concerning the recognized person or animal.

Shopping Item Printed Object

[0123] Yet another example uses an object comprising a shopping item such as a particular product or service offering. Image recognition processes may be used to recognize the product or service offering. Successful recognition can be used to select and cause display of a media item that provides more information and images concerning the recognized product or service offering, or buying options

[0124] In one example, such objects may be made of paper, plastic and/or other materials, and may be rigid, flexible and/or foldable. They may be of standard sizes such as 2.61 inches wide and 6.14 inches long; 120 by 62 millimeters (4.7 in×2.4 in) to 160 by 82 millimeters (6.3 in×3.2 in); the ID-1 format size of 85.60×53.98 mm (3⅜ in×2⅛ in approx.) and rounded corners with a radius of 2.88-3.48 mm (about ⅛ in); ID-3 standard size of 125×88 mm (4.921×3.465 in); or other size. See for example ISO/IEC 7810:2003. The object may be two-dimensional or three-dimensional.

Banknote Printed Objects

[0125] One example such object comprises a banknote such as a US dollar bill, US five dollar bill, US ten dollar bill, US twenty dollar bill (see FIG. 2A), US fifty dollar bill (see FIG. 2B), a US one-hundred dollar bill (see FIG. 2C), a 20 euro note (see FIG. 2D), a 50 euro note (see FIG. 2E), or any other currency banknote. Other example objects comprise official government-issued identification such as a government issued passport (see FIG. 2F), passport card (see FIG. 2G), driver's license (see FIG. 2H, 2I), coin (FIG. 2J) or other official or unofficial object.

[0126] Such two-dimensional objects as described above often are protected by any or all of the following visible security features:

• • [0127] holograms such as portrait window(s)
  • [0128] woven three-dimensional security ribbon(s)
  • [0129] distinctive image pattern(s) such as the EURion constellation
  • [0130] glossy stripe(s) that reveal additional information when viewed on an angle
  • [0131] watermark(s)
  • [0132] raised printing
  • [0133] ultraviolet ink (which reveals additional image information when viewed under ultraviolet light)
  • [0134] thermochromatic ink (the color of which is dependent on temperature)
  • [0135] color changing or color shifting ink (the color changes depending on viewing angle)
  • [0136] see through indicia with complementary front and obverse portions
  • [0137] microprinting (high resolution printing)
  • [0138] unique printing fonts
  • [0139] embedded security threads
  • [0140] perforations
  • [0141] plastic, cotton fiber or other material matted or other textured surface(s)
  • [0142] barcodes
  • [0143] serial numbers
  • [0144] foil stamps
  • [0145] photographs including clear window photographs
  • [0146] unique metal compositions (for coins) such as bi-coloration
  • [0147] other.

[0148] Example non-limiting recognition processes as described above can exclude such security features, or may take them into account or use them as part of the recognition process. However, since the purpose of the recognition is not to authenticate the photographed item as being genuine, the recognition/matching algorithm is quite different from ones that are used for banknote or ID authentication. In example non-limiting embodiments for example, it is desirable that matching occurs based on photographing a copy (e.g., displayed on a tablet screen or the like) and not just an original of a banknote, ID or the like. Thus, the matching will achieve positive results based on counterfeit (inauthentic) banknotes or IDs. However, the matching/recognition is robust in being able to detect different banknote denominations (e.g., 10 euro note versus 20 euro note versus $100 US bill etc.) and thus provide different overlays depending on which banknote is recognized.

Example Non-Limiting Image Recognition

[0149] The image recognition performed by example embodiments is different from prior techniques of recognizing a QR code/other AR technologies. For example, illustrative image recognition technology herein transfers any 1-dimensional, 2-dimensional or 3-dimensional object into a one-dimensional representation. That means that we can but do not need to recognize any shape or form of a 1, 2 or 3 dimensional object. See FIG. 4A. Typically, 2D or 3D scanners are not able to process 1D. With this technology, all objects are technically translated into a common representation so a 3D package of a product is transformed before it is compared with assets stored in the data base to play out a digital overlay.

[0150] With example non-limiting embodiments, there is no need for a controlled environment to scan objects successfully. The embodiments do not need a controlled environment, like on a conveyor belt, to work successfully. See FIG. 4B. Nor are several permanently installed camera systems needed, neither do we need a belt that has a determined path for the objects that have to be scanned to travel. The mentioned technology reliably detects the object when photographed from different angles, such as the camera of an iPhone® or the camera of special glasses (which are in contrast flexible). Such cameras have increasingly high resolution but because they are typically handheld and are not standardized, the field of view and perspective vary from device to device and user to user. The lighting, image quality, magnification and amount and characteristics of parallax distortion typically varies from photo to photo. Furthermore, the resulting captured images are also typically subjected to different types of image compression. The systems camera used by embodiments herein does not need a special position, altitude, perspective etc. It is very flexible in contrast to other systems.

[0151] The mentioned system, because it works with shades of grey and colours, does not need the object in good light. A mobile phones camera is sufficient to have a successful scan, even in a darker environment. The light from the mobile phone itself is typically sufficient to capture a useful image. See FIG. 4C.

[0152] Any type of 2D or 3D object can be the subject of image capture and analysis. In still other embodiments, the two-dimensional or three-dimensional object could comprise a political campaign poster, billboard, flyer package, floor stand, table stand, pen with advertisement or other printed material or 3 dimensional object with print and the overlay could provide a campaign song, slogan, speech, story, video or other content. Other types of campaigns could include advertising campaigns, company logos and printed advertising or any other printed campaign from companies or other entities.

[0153] As discussed above, other embodiments can employ a two-dimensional object comprising a real estate advertisement, a job advertisement, a product advertisement, a service advertisement, or any other presentation of an item, entity or organization that a user may wish to obtain more information concerning.

[0154] In example embodiments herein, the two-dimensional object is not (and does not need to contain) a so-called “AR marker” or 2D bar code and is otherwise not specially designed or intended to be recognized by an augmented reality display device.

[0155] The technology herein also is able to flexibly define areas of recognition.

[0156] FIG. 5A shows a smart device 202 the camera of which has captured an image of a $20 bill. Once the app recognizes the captured image 402 as described above, it displays a corresponding advertisement 404 as shown in FIG. 5B. The advertisement may be superimposed over the captured image 402.

[0157] FIG. 6A shows a smart device 202 the camera of which has captured an image of a US passport. Once the app recognizes the captured image 406 as described above, it displays a corresponding video 408 (as shown in FIG. 6B) that is superimposed onto the captured image 406. If the user rotates the smart device 202 to change the position and/or orientation of the captured image 406, the corresponding video 408 will also change its position and/or orientation by the same amount. In some embodiments, the video 408 may be displayed with controls such as play/stop, fast-forward, rewind and mute that the user can press on the touch screen display 210 to allow the user to control playback of the video and/or accompanying audio.

[0158] Instead of a video, FIG. 6B could display a superimposed control bar that allows the user to play an audio story associated with the captured image 406.

[0159] FIG. 7A shows a captured image of a print advertisement. FIG. 7B shows an example area of recognition. FIG. 7C shows a superimposed virtual image (shopping carts) on the FIG. 7A captured image to provide a virtual or mixed reality composite image.

[0160] FIG. 8A shows an example of a captured image of a print advertisement. FIG. 9A shows an example area of recognition. FIG. 9B shows a superimposed virtual image (a shopping cart) on the FIG. 8A captured image to provide a virtual or mixed reality composite image.

[0161] FIG. 10A shows a captured image of a product package with a defined area of recognition. FIG. 10B shows a superimposed virtual or mixed reality composite image providing a video of ingredients.

[0162] FIG. 11A shows a captured image of a print advertisement for real estate listings. FIG. 11B shows a closeup of an area of recognition. FIG. 11C shows a superimposed virtual image on the FIG. 11B captured image.

[0163] FIG. 12 shows an example captured image of a human resources advertisement. This captured image can have an area of recognition and can have a digital image superimposed thereon to provide an augmented or mixed reality image on a smart device such as a smartphone, smart glasses, etc. In some embodiments, the display process may comprise projecting the digital overlay directly or indirectly onto a user's retina. Such techniques would allow the user to view the digital overlay on a screen (e.g., smartphone) and/or on glasses that are used as a screen and/or independently of any screen viewing (i.e., the user could simply view the real world and the digital overlay would become part of their field of view based on such projection directly onto their retina).

Flyer Example

[0164] Take a 10 page advertisement for a Hardware store like The Home Depot, Loews etc. that is distributed in a printed version to attract customers. See FIGS. 13A, 13B. The merchandise you show has a certain price tag and description that has to be defined weeks or days before the print happens. After it is printed, you can barely react to any changes in pricing that might be triggered by an aggressive competitor that offers the same merchandise for the same price or lower.

[0165] In contrast, with a system disclosed herein, you scan the printed flyer which does not include a price (FIG. 13C), match the image with the data base and play a specific digital overlay that is customized to this special page. See FIG. 13D. In this example, after finding a match in the database, the system plays out an associated interactive, video, or other digital overlay. Since we work with a digital overlay, even a printed price tag is a print or printed flyer can be superimposed with a new price tag. The overlay can include for example an interactive video and/or a shop button and/or a price and/or a store finder where the particular product is in stock. To change a price tag, you just need to change the price tag of the digital overlay. This digital overlay is sent back to the originating user device and is displayed in a superimposed fashion “on top of an image capture of the original print advertisement. Content of the overlay such as video, price, merchandise description, etc. can be changed, customized or updated instantly without changing the printed advertisement. For example, different geographical regions can show different prices or other content depending on demographics, availability, supply and demand, etc.

[0166] Let's assume the demand is very high and this merchandise is out of stock in STORE A but still available in STORE B. In this case you add an information to the digital overlay directing customers to STORE B to buy this merchandise.

[0167] If this would happen to a system that works with a different technology like a QR code or a hidden printed signature, you would have to reprint the flyer and distribute it again.

[0168] Another example is a real estate advertisement.

[0169] With a QR code or a hidden printed signature it would not be possible to update the content in seconds. While by using the technology mentioned herein allows to just change some information within the digital overlay. There is no need to reprint the real estate advertisement to change a QR code or change a hidden printed signature.

Automobile Example

[0170] In another embodiment, you sit in a car that has cameras and the information is displayed on the cars windows so that the passengers can see the digital information. 3D objects can only be recognized if they have any kind of print on their surface so that the system can recognize a color or grey code. Only the shape of an object is not enough as a trigger. That means that you can detect any 3 dimensional object if it has at least a bit of print on its surface that allows to define an area of recognition.

Further Working Example

[0171] As shown in FIG. 14, the camera of a device recognizes an object 3D, which could be a package of a toothpaste. The camera can also detect a 2D page of a newspaper or a printed advertisement. This could be an advertisement presented as a leaflet or a flyer or a magazine, or a poster, a citylight or a picture, presenting just one article of merchandise, just one property or more merchandise or more properties.

[0172] The camera could also detect just a line printed on a white piece of paper, a circle or different separated lines on a piece of paper. These are 1D example sources.

[0173] In this embodiment, the camera position is not fixed. Object positioning is therefore flexible.

[0174] All these dimensions, from 1D to 3D are analyzed by the technology by their different shades of grey and/or their color code. This can occur on the capture device, in the cloud, or in any distributed fashion. The results of the analysis is compared with the content of a data base that includes the specific shades of grey/colors of different objects.

[0175] To identify a 1D, 2D or 3D object, and to avoid restricted areas to identify an object, you can define an area of recognition dedicated to that object. As shown in FIG. 14A, this area of recognition can be defined as the whole object, just a part of it or several different parts to avoid an mismatch or a non-match that would happen, if, as example, a restricted area of recognition would be covered by a price tag etc.

[0176] The combination of different areas of recognition allows the system to “find or identify” the object even if certain areas are covered or the object is scanned from a different perspective. That means that successfully scanning objects is not depending on a fixed camera angle or a fixed position on a conveyor belt or the appearance of a specific side of a 3D object to be identified.

[0177] Therefore you do not need hidden printed codes or a QR code.

[0178] Each object in this data base is connected to a digital overlay that will be transmitted back to the device so that it will play out on a screen, lenses or a monitor. The digital overlay can cover 100% or more of the object or only one or more parts of the object. Within this digital overlay, videos can be played or buttons can be implemented. There can be several buttons with several different functions. See FIG. 15. These button can direct to a shop to buy instantly, can direct to information like a manual, a store finder, a call number etc. Depressing a button can automatically generate a call to a supplier. The call could be a telephone call, a digital voice or video call, or could cause a supplier to call the customer back.

Tracking Attention

[0179] This technology allows to count and measure all activities like detection, watching a video, shop, press a call button etc. For example, all actions triggered by the functions of the digital overlay are counted and measured to provide valuable analytics by statistics or artificial intelligence (AI). See FIG. 16. The system can measure the activity itself, the duration of watching a video, the order of activities (first a video, then shop), the shopping activity (when, where, how much, which price tag, using a discount or not) and the value of the related price tag. See FIG. 16. For example, the system can measure:

[0180] How many people watched a video, and how long did they watch

[0181] How many people watched the video and then purchased the product.

[0182] How many people bought the product without watching the video.

[0183] Based on such information collection, the system can provide data and recommendations—for example only 20% watched the video, so the video should be dropped; or 80% watched the video and bought the product, so the video should be retained.

[0184] In a real estate environment, the system can measure if customers pressed the floor-plan button first, or if they preferred to see the pricing history or the photos or the list of schools, first.

[0185] The aggregated data allow related analytics and finding certain patterns in complex data environments (AI).

[0186] Based on these analytics, the system can generate recommendations, too.

[0187] Length of a video, order of merchandise in a flyer to increase customer attraction, certain price levels that had a better response rate, different video content to increase the time a consumer watches etc.

[0188] Since the system can measure any activity within this digital overlay, it can also detect when customers stop doing something and learn from that.

[0189] One example is a video that is too long or is regarded as boring.

[0190] Without related analytics there would be no recommendation based on data, how long a video should be to be watched by more people, or which content entertains better.

[0191] Same with different price levels that provide more or less shopping activity.

[0192] That means that the system not only detects and plays out related content and options in a digital overlay, it also measures, counts and analyzes with statistics and algorithms that could be AI, what the user is doing to generate reports, statistics and recommendations to optimize as example pricing.

[0193] Apart from the total flexibility of viewpoint and its ability to use color and shades of gray it can do 1D, 2D and 3D.

[0194] We do not need a restricted area of recognition and that it provides data generated from users engaging with the functions and services that are embedded in that digital overlay.

[0195] The overlay can be bigger that the object itself, cover 100% or just a fraction of the object.

Many Example Use Cases

[0196] FIG. 17 shows that this technology can be used for a variety of different applications including shopping, real estate, AR on a package, Human Resources (job opportunities), and manuals. For example:

• • [0197] Real estate ads in newspapers, magazines or on flyers
  • [0198] AR on product packaging (video of product features, ingredients, discount codes, advertisement, manual, etc.) and store finder where product is available for shopping/reorder.
  • [0199] Human resources advertisement
  • [0200] Travel applications
  • [0201] How to manuals.

[0202] The digital overlays can be updated quickly. There is no need to change the print—in case of a price change or other information update, you can just change the information within the digital overlay.

[0203] All printed publications cited above (including the ones shown in FIGS. 2A-2J) are hereby incorporated by reference as if expressly set forth.

[0204] While the invention has been described in connection with what is presently considered to be the most practical and preferred embodiments, it is to be understood that the invention is not to be limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.