Identifying pixel locations using a transformation function

[0001] A variety of media content streaming techniques are employed to deliver media content to client devices. Virtual Reality (VR) content is intended to provide an immersive experience in which a user is apparently surrounded by and interacts with a virtual environment constructed from actual or computer-generated images. The original source images from which VR content is constructed correspond to different views from the user's apparent location at the center of the environment (e.g., front, back, left, right, etc.). In order to encode the source images, the images are combined to form a two-dimensional (2D) representation (e.g., an equirectangular projection of a polygonal projection) in which each of the combined images corresponds to a portion of a three-dimensional (3D) shape, for instance, a cylinder or a cube. Typically, the 2D representation includes non-display pixels that do not contain image content. When non-display pixels are included during the encoding process, playback of the VR content can include visual artifacts, which disrupt a user's viewing experience.

[0008] This disclosure describes techniques for identifying pixel locations for video encoding using a transformation function. Typically, an encoder requests pixels according to standard pixel locations on a 2D representation of VR content. This includes non-display pixels, which the encoder interprets as a sharp edge making encoder performance less efficient and accurate, and often resulting in visual artifacts during playback. Therefore, instead of providing the pixels to the encoder according to the standard pixel locations, new pixel locations are determined based on the arrangement of the pixels in the 3D projection space, and pixels are provided to the encoder according to the new pixel locations. Using these new pixel locations, VR content can be encoded without at least some of the visual artifacts that result from an encoder using non-display pixels in the encoding process. An example may be instructive.

[0009] FIG. 1 illustrates an example of identifying pixel locations based on a 3D projection space. As shown in FIG. 1, computing device 100 includes encoder 101. Device 100 identifies pixels and provides them to encoder 101 for encoding source media content (e.g., VR content). In this example, six images (e.g., from a six-sided camera) have been stitched together into a 2D representation 102 of a polygonal projection space. In this example, 2D representation 102 corresponds to a cubic projection space. Representation 102 shows a basketball moving towards a basketball hoop with different regions including display pixels and non-display pixels. A non-display pixel is a pixel that does not contain source image content (represented as a null value or a set of pixels values set to 0), while a display pixel is a pixel that contains source image content (represented as a set of pixel values). Encoder 101 encodes 2D representation 102 according to the pixels it receives, and pixels can be provided to the encoder in a variety of ways. Conventionally, device 100 would provide pixels to encoder 101 sequentially according to the arrangement of pixel locations represented in 104a. By contrast, and according to various implementations enabled by the present disclosure, device 100 provides pixels to encoder 101 according to the arrangement of pixel locations represented in 104b.

[0010] In order for device 100 to provide pixels according to pixel locations 104b, first, device 100 identifies a polygonal projection space associated with representation 102 (e.g., polygonal projection space 106). In this example, polygonal projection space 106 is a cube. After identifying 3D polygonal projection space 106, device 100 identifies a coordinate transformation function based on polygonal projection space 106. In this example, the coordinate transformation function is a 3D coordinate transformation function for a cubic projection space. Next, device 100 generates new pixel locations according to the 3D coordinate transformation function (e.g., pixel locations 104b). In this way, when encoder 101 requests the next pixel, instead of device 100 providing pixel 110a to encoder 101 using pixel location 108a, device 100 provides pixel 110b using pixel location 108b to encoder 101. By providing pixel 110b instead of pixel 110a and any other adjacent display pixels not identifiable using standard pixel location, VR content is encoded without visual artifacts that would result from non-display pixels being included during encoding.

[0011] FIG. 2 illustrates an example of a computing environment for encoding media content. The computing environment of FIG. 2 includes encoding server 202 that encodes media content for playback on viewer device 216. Viewer device 216 can receive encoded media content from an edge server or encoding server 202, or it might receive media content using a physical medium, e.g., memory 219. In FIG. 2, viewer device 216 is a wearable VR device. However, related functionality can be implemented in laptop computers, televisions, tablets, smartphones, wearable devices, virtual or augmented reality headsets, video game consoles, etc.

[0012] Encoding server 202 might conform to any of a wide variety of architectures, and its functionality may be implemented by a variety of physical servers and/or virtual servers. For example, one server might encode media content, and the encoded media content might be provided to another server, for example, belonging to a content delivery network (CDN). As such, the functionality and components of encoding server 202 can use one or more servers and be deployed at one or more geographic locations (e.g., across different countries, states, cities, etc.) using a network such as any subset or combination of a wide variety of network environments including, for example, TCP/IP-based networks, telecommunications networks, wireless networks, cable networks, public networks, private networks, wide area networks, local area networks, the Internet, the World Wide Web, intranets, extranets, etc. Multiple entities may be involved in the encoding and delivery of media content and data related to the media content, including content providers, internet service providers (ISPs), providers of content delivery networks (CDNs), etc. The functionality described herein also may be implemented by one or more different entities. For example, the functionality to provide playback of media content can be integrated into a video player or software client under control of one entity (e.g., on viewer device 216), integrated into a separate application from another entity, implemented in an edge server or content server of a CDN, a server of an ISP, etc. Media content is deliverable to viewer device 216 in a variety way. In one example, media content is streamed as fragments delivered separately from a media server, e.g., adaptive bitrate streaming. In another example, a single media content file is delivered and downloaded by viewer device 216 and used by viewer device 216 for playback.

[0013] In some implementations, encoding server 202 is partitioned to have separate virtual servers configured to perform different functions. In one example, one virtual server could generate manifests, and a second virtual server might provide manifests and encoded media to viewer device 216. In another example, several virtual servers are configured to encode VR content and deliver the encoded VR content to many viewer devices.

[0014] Encoding server 202 includes one or more processor circuits 204, memory 205, and other hardware components to encode media content and provide the encoded media content to viewer device 216. For example, processor circuits 204 execute stored instructions in memory 205 using encoder 206 to implement encoding techniques disclosed herein. Encoder 206 can be configured to encode bitstreams that are compliant with a variety video coding formats (e.g., H.265/High Efficiency Video Coding, H.264/MPEG-4 Advanced Video Coding, MPEG-2, etc. It should be noted that, while processor circuits 204, memory 205, 2D data 210, encoded media content 212, manifest data 214, and logics 208 and 209 are contemplated as integrated with encoding server 202, implementations are contemplated in which some or all of their respective functionality is implemented independently of a single server. It should also be noted that processor circuits 204, logic 208-209, or both processor circuits 204 and execute logic 208-209 can be implemented using special purpose circuitry, e.g., a field programmable gate array or an application specific integrated circuit.

[0015] In FIG. 2, encoding server 202 includes various types of logic to encode VR content and provide encoded VR content to viewer device 216. For example, encoding server 202 includes pixel location logic 208 for identifying pixel locations using a transformation function that may be based on non-Euclidean geometry or a variety of other 3D coordinate systems. In some implementations, pixel location logic 208 is implemented as part of encoder 206. Pixel location logic 208 includes a variety of techniques to convert coordinates from images stored in 2D data 210 to 3D coordinates corresponding to positions of the pixels as displayed during playback. As part of the encoding process, encoder 206 receives pixels identified by encoding server 202 using pixel location logic 208. Encoder 206 may receive pixels such that the next pixel received was not adjacent (on the 2D representation) to the previous pixel received. Encoding server 202 also includes playback handling logic 209 for handling requests for media content and providing the requested media content using media content from encoded media content 212 and manifest data 214. In some implementations, manifest generation logic identifies data from manifest data 214 after receiving a request for playback of media content and generates a manifest. After generating the manifest, the manifest is stored in memory such as a local buffer and/or cache, memory 205 to be used in conjunction with playback handling logic 209.

[0016] Viewer device 216 can also include various types of logic used to implement a decoder 222 to decode encoded media content 220 received from encoding server 202, store the decoded media content in decoded content buffer, and play back the decoded media content using playback handling logic 224. Viewer device 216 includes one or more processors 218, memory 218, and other hardware components or circuits to decode and play back the encoded media content. For example, processor(s) 218 of viewer device 216 can execute stored instructions in memory 219 to implement decoding techniques based on the decoding techniques to store decoded content for playback in decoded content buffer 226.

[0017] It should be noted that, despite references to particular computing paradigms and software tools herein, the computer program product and the computer program instructions on which various implementations disclosed herein are based may correspond to any of a wide variety of programming languages, software tools, and data formats, may be stored in any type of non-transitory computer-readable storage media or memory device(s), and may be executed according to a variety of computing models including, for example, a client/server model, a peer-to-peer model, on a stand-alone computing device, or according to a distributed computing model in which various functionalities may be effected or employed at different locations. In addition, reference to particular types of media content herein is merely by way of example. Suitable alternatives known to those of skill in the art may be employed.

[0018] A specific implementation will now be described with reference to the flow diagram of FIG. 3. One or more images representing a video frame of virtual content are received (302). As discussed above, the type and the number of images may vary. For example, the images may correspond to different views of a real-world environment generated by the image capture devices of a VR camera. Alternatively, the images may be different views of a virtual environment created using computer animation tools. In another example, the number of images may vary considerably depending on the projection space represented by the image data. A cylindrical projection space might have only one or two images, while a polygonal projection space might have as few as four and as many as 20 or more images.

[0019] The received images are combined (or stitched) to form a two-dimensional or “flattened” representation, which includes the various views corresponding to portions of the cylindrical projection space or faces of the polygonal projection space (304). As one example previously mentioned, FIG. 1 shows an example of six images stitched together to form 2D representation 102 corresponding to a cube which is displayed in a spherical projection space. This may be done in the VR camera, using a computer animation tool, or at a point prior to when an encoder begins encoding the 2D representation. The 2D representation is then provided to a server (e.g., encoding server 202 of FIG. 2) for encoding (306).

[0020] After receiving the 2D representation, a projection space associated with the 2D representation is identified (308). In one example, the projection space that is identified is an equirectangular projection space (e.g., a cylindrical projection space from multiple images stitched together using multiple images from horizontally spaced cameras). In another example, the projection space is a polygonal projection space. As such, the server identifies the type of polygon (or polyhedron) associated with the polygonal projection space. Types of polygonal projection spaces include a tetrahedron (having 4 faces), a cube (having 6 faces), an octahedron (having 8 faces), a dodecahedron (having 12 faces), or an icosahedron (having 20 faces). In some implementations, after the projection space is identified for an initial 2D representation received by a server, the server no longer identifies the type the polygon for each of the remaining 2D representations later received. After identifying a projection space, a transformation matrix associated with the projection space is identified and used to generate pixel locations for each 2D representation subsequently received. For example, if the projection space identified is an icosahedron, a transformation matrix for a 2D representations that include an icosahedron projection space is identified. The transformation matrix may remap each of the coordinates on the 2D representation to quickly identify new pixel locations as discussed further below.

[0021] In some implementations, projection spaces are identified using metadata included with a 2D representation. For example, 2D representation 102 of FIG. 1 may include one or more metadata fields describing various attributes of 2D representation 102, with one of those fields being a field indicating a type of projection space. Using the example of FIG. 1, the value for a type field is a cube, but the value could be any of the polygon shapes discussed above, as well as equirectangular.

[0022] In other implementations, if metadata is not included in a 2D representation, projection spaces are identified using characteristics of the 2D representation. Characteristics of the 2D representation include a number of faces, shape of the faces, straight or diagonal edges, or locations of vertices. Using the example of FIG. 1, device 100 identifies the projection space associated with 2D representation 102 by identifying the number of faces for projection space 106 (e.g., 6 faces) and the shape of the 6 faces (e.g., square).

[0023] Each type of projection space has at least one coordinate transformation function for generating new pixel locations. After (or as part of) the identification of the projection space, the server identifies a coordinate transformation function associated with the identified projection space (310) and uses the coordinate transformation function to generate new pixel locations (312).

[0024] FIG. 4A shows a simplified example of pixel locations of VR content prior to encoding. In FIG. 4A, 2D representation 402 includes faces 404a, 404b, and 404c (3 faces of an icosahedron). Each of faces 404a, 404b, and 404c include display pixels at corresponding pixel locations (e.g., 17, 18, 21, 22, etc.). Pixel locations not within faces 404a, 404b, or 404c identify non-display pixels at corresponding pixel locations (e.g., 19, 20, 23, 24, etc.). In some implementations, pixel locations 406a are standard or default pixel locations of a 2D representation of VR content.

[0025] In some implementations, new pixel locations are generated for each pixel location of faces 404a, 404b, and 404c. For example, FIG. 4B shows pixels corresponding to new pixel locations generated using a coordinate transformation function. In FIG. 4B, pixel locations 406b only correspond to display pixels, and there are no pixel locations that identify non-display pixels. For example, the top display pixels seen in faces 404a, 404b, and 404c of FIG. 4A are the same display pixels at locations 1, 2, and 3 of FIG. 4B. In some implementations, instead of remapping each individual pixel to a new pixel location, average pixel values of the pixels at pixel locations 406a of FIG. 4A are used to generate the pixel values of the pixels at pixel locations 406b. For example, the pixel at location 11 of FIG. 4B is an average of the pixels at locations 31 and 32 of FIG. 4A. In this example, the pixel value of the pixel at location 11 of FIG. 4B is 50% of the pixel at location 31 and 50% of the pixel at location 32 of FIG. 4A. Other averages can also be used, for instance, an average pixel value might be 25% of four surrounding pixel values. Various filters may be used to find average pixel values, for example, linear filtering, bilinear filtering, antialiasing filtering, bi-cubic filtering, etc. In order to avoid aliasing artifacts, sampling rates using one of these filtering techniques are constrained to a threshold that is based on the Nyquist limit for the 2D representation, i.e., twice the highest frequency sampling to be preserved in the filtered VR content. Those of skill in the art will appreciate that the foregoing are merely some examples of filtering techniques that may be employed to filter pixel values as described herein.

[0026] As mentioned above, each type of projection space has at least one coordinate transformation function. For example, if the type of polygonal projection space is a cube, coordinates for the coordinate transformation function define the vertices of each face of the cube. The vertices of a front face of the cube are (0, 0, 0; 1, 0, 0; 1, 0, 1; and 0, 0, 1). The coordinates at the vertices of a right face of the cube are (1, 0, 0; 1, 1, 0; 1, 1, 1; and 1, 0, 1). The coordinates at the vertices of a back face of the cube are (1, 1, 0; 0, 1, 0; 0, 1, 1; and 1, 1, 1). The coordinates at the vertices of a left face of the cube are (0, 1, 0; 0, 0, 0; 0, 0, 1; and 0, 1, 1). The coordinates at the vertices of a top face of the cube are (0, 0, 1; 1, 0, 1; 1, 1, 1; and 0, 1, 1). The coordinates at the vertices of a bottom face of the cube are (0, 0, 0; 1, 0, 0; 1, 1, 0; and 0, 1, 0). In another example, if the type of polygonal projection space is an icosahedron the following 12 vertices can be combined to make up each face of the icosahedron: (0, ±1, ±φ) (±1, ±φ, 0) (±, 0, ±1), with φ representing

[0027] $\frac{1+\sqrt{5}}{2}.$
Similar coordinate transformation functions can be used for tetrahedrons, octahedron, dodecahedrons, or other projection spaces.

[0028] Depending on how edges are joined, there are many different ways that the faces of a polyhedron can be connected. As such, the coordinate transformation function can further define the coordinates of each vertex and the order in which the faces are connected. Faces sharing vertices are adjacent during playback of VR content on a playback projection space. For example, the front face of a cube shares the vertices of (1, 0, 0) and (1, 0, 1) with the right face of a cube. In another example using an icosahedron, one face of the icosahedron shares the vertices of (−φ, −1, 0) and (0, −φ, −1). The order that faces are connected can be used to determine which faces include adjacent pixels for the purposes of motion estimation prediction between encoded frames discussed below.

[0029] In some implementations, adjacent faces are used to determine the size and boundaries of blocks of pixels to be used for encoding (e.g., coding units, slices, tiles, blocks, macroblocks. Blocks of pixels are configured to align along adjacent faces in the output such that block size tends to decrease in size (e.g., 8×8 pixels, 4×4 pixels, or 2×2 pixels) at pixel locations near the edge of a face. For example, adjacent faces are typically encoded without reference to other faces to be displayed within the same projection space (e.g., non-display pixels are included as part of the encoding process), and as such, the VR content at the edge of a face might be encoded with a perceptible drop in quality, which disrupts a viewer's experience during the playback of the VR content. By configuring block sizes to align with edges of a face, each face can be encoded with reference to the other adjacent faces, avoiding abrupt changes in quality.

[0030] In some implementations, spherical coordinates are used as part of a coordinate transformation function to generate new pixel locations for equirectangular projections or polygonal projections. In order to generate new pixel locations using spherical coordinates, the vertices of each face of a polyhedron are identified using a coordinate transformation function (e.g., longitude and latitude). In an example using an icosahedron, the latitudes of each vertex, begin at either of the two vertices at the poles of an icosahedron are at latitudes of +90° and −90°, with the remaining ten vertices evenly spaced at incrementing latitudes of 26.57° (arctan(½)). The longitudes of the ten vertices not at the poles are also evenly spaced at longitudes 36° apart. Using the vertices as points of reference, the remaining pixel locations are generated. For example, using a first vertex, a vertical position is identified (e.g., 72° longitude). Next, curvature of a playback projection is identified at the vertical position. This can give the approximate distance from one point to the next on the playback projection space. From that, horizontal positions of the pixels to the right and left of the first vertex can be identified. For example, in FIG. 5 pixel locations 502a, 502b, 502c, and 502d are shown as part of projection 500. As shown in FIG. 5, a non-Euclidean coordinate transformation function can be used because there are no straight lines between points on projection 500, but rather only curved lines, or the shortest path between two points. For the purposes of identifying the shortest path between two pixels, the curvature of playback projection is included to accurately identify adjacent pixel locations. Depending on the curvature of projection 500, the distance between pixels may increase or decrease. For example, pixel location 502a and 502b, which are near a pole of projection 500 are relatively close to each other compared to pixel locations 502c and 502b, which are towards the equator of projection 500.

[0031] Returning to FIG. 3, an encoder sends requests for pixels to use as part of encoding a 2D representation (314), and pixels are identified and sent using the new pixel locations discussed above (316)(318). The number of pixels requested can vary depending on how the 2D representation is to be encoded. For example, blocks of pixels ranging from 16×16 pixels, 8×8 pixels, 4×4 pixels, or 2×2 pixels may be used. For example, in FIG. 4B, using pixel locations 406b the encoder receives pixels in the order shown as part of encoder memory 408. In this example, an encoder requests pixels moving from left to right and top to bottom. The pixels at pixel locations 1, 2, and 3 represent pixels at a pole of a playback projection space. If the encoder requests the top row of pixels, pixels 1, 2, 3, and an additional reference to pixel 1 are sent to the encoder (compare this to using pixel locations 406a of FIG. 4A, where pixels at pixel locations 1-14 would be sent). In this example, the encoder is requesting pixels as part of estimating motion by identifying the difference in pixel values among adjacent pixels. As such, in order for the encoder to find the difference between all the pixels on the top row, the additional reference to pixel 1 is used by the encoder to compare values between pixel 3 and pixel 1. Similarly, an additional reference to pixel 4 is included for the second row, an additional reference to pixel 10 for the third row, an additional reference to pixel 19 for the fourth row, etc. In addition, similar requests are made for motion estimation along columns of pixels (e.g. an additional copy of pixel 1 would be used by the encoder to compare the value pixel 19 to the pixel below, which is pixel 1). It will be understood by those skilled in the art that a variety of motion estimation techniques as defined according to various encoding standards (mentioned above) can also be used with the disclosed implementations.

[0032] While the subject matter of this application has been particularly shown and described with reference to specific implementations thereof, it will be understood by those skilled in the art that changes in the form and details of the disclosed implementations may be made without departing from the spirit or scope of the invention. Examples of some of these implementations are illustrated in the accompanying drawings, and specific details are set forth in order to provide a thorough understanding thereof. It should be noted that implementations may be practiced without some or all of these specific details. In addition, well known features may not have been described in detail to promote clarity. Finally, although various advantages have been discussed herein with reference to various implementations, it will be understood that the scope of the invention should not be limited by reference to such advantages. Rather, the scope of the invention should be determined with reference to the appended claims.