PIPELINED INTRA-PREDICTION HARDWARE ARCHITECTURE FOR VIDEO CODING

[0001] 1. Technical Field

[0002] The present application generally relates to intra-prediction in video encoders and video decoders and, more specifically, to hardware implementations of intra-prediction using pipelining techniques.

[0003] 2. Related Art

[0004] Video codecs, such as those used in High Efficiency Video Coding (HEVC), incorporate a variety of techniques to compress video for use in mobile devices, smart TVs, and many other video applications. One such technique is intra-prediction, which takes advantage of spatial redundancies within a video frame to reduce the file size and/or bandwidth required for video streaming.

[0005] A video frame is generally segmented into blocks of pixels, and a given block is often highly correlated with the blocks and pixels immediately around it. Accordingly, the pixels within a block can be interpolated from neighbor pixels that have already been calculated. The nature of the dependency may be stored as an intra-prediction mode. Once the intra-prediction mode is determined, only the differences (e.g., residuals) between the interpolated pixels and the actual pixels need to be encoded. These residuals are encoded in subsequent processing stages, which include a discrete cosine transformation (DCT), quantization, and entropy coding. By taking spatial redundancy into account, intra-prediction can greatly increase the overall compression ratio.

[0006] In HEVC, some reference frames (e.g., intra-coded pictures or I-frames) are encoded and decoded entirely by intra-prediction (as opposed to inter-prediction). Reference frames may comprise a plurality of coding units (CU), and each CU may be divided into transform units (TU) of size M×M. The CU may be further divided into coding blocks (CB) which may be associated with chroma or luma information. Similarly, the TUs may be divided into transform blocks (TB). Intra-prediction modes may be computationally applied at a TB level.

[0007] The HEVC standard dictates that 4M+1 neighbor pixels may be used to perform the intra-prediction, and these neighbor pixels must therefore be padded, filtered, and potentially conditioned in other ways before intra-prediction for a given block (e.g., TU) begins. After the intra-prediction of the block is completed, the intra-prediction of the next block begins. The two blocks may be processed sequentially, because intra-prediction of the second block may generally depend upon reconstructed pixels derived from the intra-prediction of the first block and/or of other previously calculated blocks. Thus, the HEVC standard inherently requires a degree of sequential processing of each block at this stage, which is true for many other video standards as well.

[0008] Given the requirement of sequential processing described above, the present disclosure aims to reduce latency between intra-prediction of consecutive blocks (e.g., consecutive transform units within a coding unit). It is noted that the intra-prediction of the second block often does not depend on every reconstructed pixel associated with the first block, but merely on a single critical row or column, depending on the relative position of the first and second blocks. If each block is intra-predicted using a raster scan, as is typically done, the critical row or column is intra-predicted only towards the very end of the processing cycle of the first block. As reconstruction, filtering, padding, and/or other conditioning of the pixels of the critical row or column can take many processing cycles, the intra-prediction of the second block is delayed. In HEVC, each coding unit can contain up to 256 transform units, and so these delays can add up quickly to reduce overall performance.

[0009] The proposed hardware architecture may utilize an intra-prediction processing unit operable to prioritize the pixel generation for the critical rows and columns in a first block. A neighbor processing unit may receive reconstructed pixels corresponding to the critical rows and columns and may further prepare (e.g., pad and filter) the neighbor pixels for the intra-prediction of the second block before intra-prediction of the first block is completed. This allows the intra-prediction processing unit to quickly transition between each of the blocks. By interleaving neighbor processing and intra-prediction, latency between intra-prediction of consecutive blocks (e.g., adjacent transform units that are within a coding unit) is removed or reduced.

[0010] Furthermore, the plurality of intra-prediction engines may operate in parallel to process a given block. The number (N) of intra-prediction engines may scale with the read bandwidth (e.g., 2N pixels per cycle) of a neighbor buffer (e.g., comprising neighbor RAM) used to store neighbor pixels. In general, the maximum read rate of memory used by the intra-prediction engines may be closely matched to the rate at which the intra-prediction engines output pixels for reconstruction (and, e.g., for intra-prediction of subsequent blocks). The disclosed systems and methods may accordingly scale to manage the common bottleneck associated with read bandwidth.

[0020] These exemplary figures and embodiments are to provide a written, detailed description of the subject matter set forth by any claims that issue from the present application. These exemplary figures and embodiments should not be used to limit the scope of any such claims.

[0021] Further, although similar reference numbers may be used to refer to similar structures for convenience, each of the various example embodiments may be considered to be distinct variations.

[0022] FIG. 1 shows a block diagram illustrating an example of a video encoding and decoding system 10 that may be configured to utilize techniques for pipelined intra-prediction in accordance with the principles of the present disclosure. As shown in FIG. 1, the system 10 includes a source device 12 that transmits encoded video to a destination device 14 via a communication channel 16. Encoded video may also be stored on a storage medium 34 or a file server 36 and may be accessed by the destination device 14 as desired. The source device 12 and the destination device 14 may comprise any of a wide variety of devices including desktop computers, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets (e.g., smartphones or other personal communication devices), televisions, cameras, display devices, digital media players, video gaming consoles, mobile devices, and the like. In many cases, such devices may be equipped for wireless communication, wired communication, or both. Accordingly, the communication channel 16 may comprise a wireless channel (e.g., a Wi-Fi connection), a wired channel (e.g., DSL, cable modem, Ethernet, USB), or a combination of wireless and wired channels suitable for transmission of encoded video data.

[0023] The principles of the present disclosure may be applied to video coding in support of any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, streaming video transmissions (e.g., via the internet), encoding of digital video for storage on a data storage medium, decoding of digital video stored on a data storage medium, and other applications.

[0024] In some embodiments, the devices 12, 14 may operate in a substantially symmetrical manner, such that each of the devices 12, 14 includes video encoding and decoding components. Accordingly, the system 10 may be configured to support both one-way and two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and video telephony.

[0025] In the embodiment shown in FIG. 1, the source device 12 includes a video source 18, a video encoder 20, a modulator/demodulator (modem) 22, and a transmitter 24. The video source 18 may comprise any combination or individual instances of a video capture device (e.g., video camera), a video archive containing previously captured video, a video feed interface operable to receive video from a video content provider, and a computer graphics system for generating computer graphics data as the source video. In some embodiments, the video source 18 may be a video camera, and the source device 12 and the destination device 14 may form camera phones or video phones. However, the techniques described in this disclosure are applicable to video coding in general, and may be applied to wireless and/or wired applications.

[0026] The captured, pre-captured, or computer-generated video may be encoded by the video encoder 20. The encoded video information may be modulated by the modem 22 according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14 via the transmitter 24. The modem 22 may include various mixers, filters, amplifiers, and other components designed for signal modulation. The transmitter 24 may include circuits designed for transmitting data, including amplifiers, filters, and one or more antennas.

[0027] The captured, pre-captured, or computer-generated video that is encoded by the video encoder 20 may also be stored onto a storage medium 34 or a file server 36 for later consumption. The storage medium 34 may include Blu-ray discs, DVDs, CD-ROMs, flash memory, or any other suitable digital storage media for storing encoded video. The encoded video stored on the storage medium 34 may then be accessed by the destination device 14 for decoding and playback.

[0028] The file server 36 may be any type of server capable of storing encoded video and transmitting that encoded video to the destination device 14. Example file servers include a web server (e.g., for a website), an FTP server, a network attached storage (NAS) device, a local disk drive, or any other type of device capable of storing encoded video data and transmitting it to a destination device. The transmission of encoded video data from the file server 36 may be a streaming transmission, a download transmission, or a combination of both. The file server 36 may be accessed by the destination device 14 through any standard data connection, including an internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, Ethernet, USB), or a combination of both that is suitable for accessing encoded video data stored on a file server.

[0029] In the example of FIG. 1, the destination device 14 includes a receiver 26, a modem 28, a video decoder 30, and a display device 32. The receiver 26 receives information over the channel 16, and the modem 28 demodulates the information to produce a demodulated bitstream for the video decoder 30. The information communicated over the channel 16 may include a variety of syntax information generated by the video encoder 20 for use by the video decoder 30 in decoding video data. Such syntax may also be included with the encoded video data stored on the storage medium 34 or the file server 36. Each of the video encoder 20 and the video decoder 30 may form part of a respective encoder-decoder (CODEC) that is capable of encoding and decoding video data.

[0030] The display device 32 may be integrated with, or external to, the destination device 14. In some examples, the destination device 14 may include an integrated display device and also be configured to interface with an external display device. In other examples, the destination device 14 may be a display device. In general, the display device 32 displays the decoded video data to a user, and may comprise any of a variety of display devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0031] In the embodiment shown in FIG. 1, the communication channel 16 may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines, or any combination of wireless and wired media. The communication channel 16 may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the internet). The communication channel 16 generally represents any suitable communication medium, or collection of different communication media, for transmitting video data from the source device 12 to the destination device 14 or bi-directionally between the source device 12 and the destination device 14. The communication channel 16 may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication between the source device 12 and the destination device 14.

[0032] The High Efficiency Video Coding (HEVC) standard is being developed by the Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T Video Coding Experts Group (VCEG) and ISO/IEC Motion Picture Experts Group (MPEG). A recent draft of the HEVC standard, referred to as “HEVC Working Draft 7” is downloadable from http://phenix.it-sudparis.eu/jct/doc_end_user/documents/9_Geneva/wg11/JICTVC-11003-v3, as of Jun. 7, 2012. The full citation for the HEVC Working Draft 7 is document JCTVC-I1003, Bross et al., “High Efficiency Video Coding (HEVC) Text Specification Draft 7,” Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3 and ISO/IEC JTC1/SC29/WG11, 9th Meeting: Geneva, Switzerland, Apr. 27, 2012 to May 7, 2012. An additional recent draft of the HEVC standard, referred to as “HEVC Working Draft (WD) 8,” is available from http://phenix.int-evry.fr/jct/doc_end_user/documents/10_Stockholm/wg11/JCTVC-J1003-v8.zip. The approved HEVC specification can be found at http://www.itu.int/rec/T-REC-H.265-201304-I, as of April 2013. Each of the references set forth in the present paragraph is incorporated by reference in its entirety herein.

[0033] The HEVC standard describes that a video frame or picture may be divided into a sequence of largest coding units (LCU), which may also be referred to as coding tree units (CTU). Each CTU may comprise a plurality of coding tree blocks (e.g., one luma CTB and two chroma CTBs). Each CTU may also be split into coding units (CUs) according to a quadtree. In general, a quadtree data structure includes one node per CU, with a root node corresponding to the CTU. For example, a CTU's quadtree may be split into four child nodes (which may be associated with sub-CUs), and one or more of these child nodes may additionally be a parent node that is split into another four child nodes. Each final, unsplit child node of the quadtree, also referred to as a leaf node, may correspond with a leaf-CU. In general, a CU has a similar purpose as a macroblock of some non-HEVC standards (e.g., the H.264 standard), with a few differences (e.g., CUs may vary in size). Accordingly, the terms “CU” and “macroblock” may be used interchangeably in this document, without any loss of generality.

[0034] Each CU may comprise a coding node as well as one or more prediction units (PUs) and one or more transform units (TUs) associated with the coding node. The CU size may correspond to the size of the coding node and may generally be square in shape. For example, the CU size may range from 8×8 pixels up to the size of the CTU, which may have a maximum size (e.g., 64×64 pixels).

[0035] In this disclosure, “M×M” and “M by M” may be used interchangeably to refer to the pixel dimensions of a video block in terms of horizontal and vertical dimensions, respectively (e.g., 16×16 pixels or 16 by 16 pixels). In general, a 16×16 block will have 16 pixels in a horizontal direction (x=16) and 16 pixels in a vertical direction (y=16). Likewise, an M×M block generally has M pixels in a horizontal direction and M pixels in a vertical direction, where M represents a nonnegative integer value. The pixels in a block may be arranged in rows and columns Common sizes for blocks (e.g., TUs in HEVC) may include 4×4, 8×8, 16×16, and 32×32. However, blocks need not necessarily have the same number of pixels in the horizontal direction as in the vertical direction. For example, blocks may comprise M×P pixels, where P is not necessarily equal to M.

[0036] Syntax data associated with a leaf-CU may describe a partitioning of the leaf-CU into one or more TUs. In some examples, residual samples corresponding to a leaf-CU may be subdivided into TUs using a quadtree structure known as “residual quad tree” (RQT), where the leaf nodes of the RQT are associated with TUs. That is, a leaf-CU may include a quadtree indicating how the leaf-CU is partitioned into TUs. An RQT (e.g., a TU quadtree) may have a root node that generally corresponds to a leaf-CU, while the root node of a CU quadtree may generally correspond to an LCU. Accordingly, one or more TU quadtrees may be nested within a CU quadtree.

[0037] The child nodes of an RQT that are not split may be associated with leaf-TUs. In general, this disclosure uses the terms CU and TU to refer to leaf-CU and leaf-TU, respectively, unless noted otherwise.

[0038] The HEVC standard allows for transformations on a TU-by-TU basis, where the TU size may vary between CUs. A TU may be square or non-square (e.g., rectangular) in shape. Further, the TUs are typically the same size or smaller than the PUs within a given CU during intra-prediction, though TUs may be larger than the corresponding PUs of an inter-predicted CU. Pixel difference values associated with the TUs may be transformed to produce transform coefficients, which may be quantized.

[0039] Syntax data associated with a CU may also describe a partitioning of the CU into one or more PUs, where PUs include data related to prediction. In general, a PU represents a spatial area corresponding to all or a portion of the corresponding CU, and may include data for retrieving a reference sample for the PU. Available PU partitioning modes may vary depending on whether the CU is skip or direct mode encoded, intra-prediction mode encoded, or inter-prediction mode encoded. For example, when a CU is intra-prediction mode encoded, it may correspond with a single PU having the same size as the CU or four square PUs that are each a fourth of the size of the CU. Further, data for the PU (e.g., an intra-prediction mode indicator) may be included in the RQT, such that it may apply to one or more TUs corresponding to the PU. In some scenarios, a PU may be non-square (e.g., rectangular) in shape.

[0040] Generally, for intra-coding (e.g., intra-predictive coding), all the TUs belonging to a CU share the same intra-prediction mode. That is, a CU is may be the same size as the PU for intra-predictive coding, thereby having a common intra-prediction mode to calculate predicted values for all TUs of the CU. In some scenarios, an intra-coded CU may be divided into four PUs, and the CU may also be divided into at least four TUs. Accordingly, during intra-prediction, TUs are smaller than or equal in size to the corresponding PU. For intra-coding, a video encoder may calculate a residual value for each TU using the intra-prediction mode, as a difference between the portion of the CU corresponding to the TU and the original block.

[0041] During inter-prediction, a TU may be larger, smaller, or equal in size to a PU within a common CU. Thus, a TU may correspond to a plurality of PUs, where the TU is larger than each of the corresponding PUs. Alternatively, a PU may correspond to a plurality of TUs, where the PU is larger than each of the corresponding TUs. However, neither a PU nor a TU may be larger than a corresponding CU.

[0042] The video encoder 20 and the video decoder 30 may operate according to a video compression standard, such as the HEVC standard or extensions to the HEVC standard. Alternatively or additionally, the video encoder 20 and the video decoder 30 may operate according to other proprietary or industry standards, such as the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) H.264 standard, alternatively referred to as MPEG-4, Part 10, Advanced Video Coding (AVC), or extensions of such standards. Other examples may include VP9, MPEG-2, and ITU-T H.263. The techniques of this disclosure, however, are not limited to any particular coding standard or technique.

[0043] Although not shown in FIG. 1, in some aspects, the video encoder 20 and the video decoder 30 may each be integrated with an audio encoder and decoder, and may include appropriate multiplexing and demultiplexing modules, or other hardware and/or software, to handle the encoding of both audio and video in a common data stream or separate data streams. If applicable, in some embodiments, the multiplexing and demultiplexing modules may conform to the ITU H.223 multiplexer protocol, or other protocols such as the user datagram protocol (UDP).

[0044] A video coder, as described in this disclosure, may refer to a video encoder, a video decoder, or both. The video encoder 20 and the video decoder 30 may each be implemented as any of a variety of suitable coder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combinations thereof. When the techniques are implemented at least partially in software, a device may store instructions for the software in a suitable, non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, either of which may be integrated as part of a combined encoder/decoder (CODEC) in a respective device.

[0045] One or both of the video encoder 20 and the video decoder 30 may implement any or all of the techniques described in this disclosure to improve the intra-prediction process.

[0046] FIG. 2 shows a block diagram illustrating an example of a video encoder 20 that may be implemented in accordance with the disclosed principles. The video encoder 20 may be configured to perform any or all of the disclosed techniques for improved intra-prediction. The video encoder 20 will be described in the context of HEVC for purposes of illustration, but without limitation of this disclosure as to other coding standards or methods that may benefit from an improved intra-prediction process.

[0047] The video encoder 20 may perform intra- and inter-coding of coding units (CUs) within video frames. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame. Inter-coding relies on temporal prediction to reduce or remove temporal redundancy between a current frame and previously coded frames of a video sequence. The previously coded frames may be before and/or after the current frame in a video sequence (e.g., within a group of pictures (GOP)). Intra-mode (I-mode) may refer to any of several spatial-based video compression modes. Inter-modes such as uni-directional prediction mode (P-mode) or bi-directional prediction mode (B-mode) may refer to any of several temporal-based video compression modes.

[0048] As shown in FIG. 2, the video encoder 20 receives a current video block within a video frame to be encoded. In the example of FIG. 2, the video encoder 20 includes a motion compensation module 44, a motion estimation module 42, an intra-prediction module 46, a reference frame buffer 64, a summer 50, a transform module 52, a quantization module 54, and an entropy encoding module 56. The transform module 52 applies a transform (e.g., discrete cosine transform (DCT)) to a block of residual data containing one or more transform units (TU), where each TU may comprise a plurality of transform blocks (e.g., one luma TB and two chroma TB). For video block reconstruction, the video encoder 20 also includes an inverse quantization module 58, an inverse transform module 60, and a summer 62. A deblocking filter (not shown) may also be included to filter block boundaries to remove blockiness artifacts from reconstructed video. If desired, the deblocking filter would typically filter the output of the summer 62. Additional filters (in-loop or post-loop) may also be used in addition to the deblocking filter. While the additional filters are not shown for brevity, they may be included at various locations within the video encoder 20, such as at the output of the summer 50 for in-loop filtering.

[0049] Although shown as separate functional modules for ease of illustration, the structure and functionality of any of the components within the video encoder 20 may be highly integrated with one another. For example, in some embodiments, the motion estimation module 42 may be highly integrated with the motion compensation module 44.

[0050] During the encoding process, the video encoder 20 receives a video frame or slice to be coded. The frame or slice may be divided into multiple video blocks, e.g., largest coding units (LCUs). The motion estimation module 42 and the motion compensation module 44 perform inter-prediction coding of the received video block relative to one or more blocks in one or more reference frames stored in the reference frame buffer 64 to provide temporal compression. The intra-prediction module 46 may perform intra-predictive coding of the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be coded to provide spatial compression. The intra-prediction module 46 may be improved in accordance with the disclosed principles, as discussed later in this document.

[0051] The mode select module 40 may select one of the coding modes (e.g., intra- or inter-), based on rate distortion analysis for each mode. Based on this selection, the resulting intra- or inter-coded block may be provided to the summer 50 to generate residual data representing a residual block and to the summer 62 to reconstruct the encoded block for use in a reference frame. The mode select module 40 may also provide syntax elements, such as intra-prediction mode indicators, motion vectors, partition information, and other such syntax information, to the entropy encoding module 56. Some video frames may be designated as I-frames, where all blocks in an I-frame are encoded in an intra-prediction mode. In some cases, the intra-prediction module 46 may also perform intra-prediction encoding of a block in a P- or B-frame, for example, when a motion search performed by the motion estimation module 42 does not result in a satisfactory prediction of the block.

[0052] As mentioned above, the motion estimation module 42 and the motion compensation module 44 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation is the process of generating motion vectors, which estimate motion for video blocks. A motion vector, for example, may indicate the displacement of a prediction unit (PU) in a current frame relative to a reference sample in a reference frame, where a PU may comprise a plurality of prediction blocks (e.g., one luma PB and two chroma PBs). A reference sample may be a block that is found to closely match the PU being coded in terms of a pixel difference metric. The pixel difference metric may involve a sum of absolute difference (SAD), sum of square difference (SSD), or other difference metrics.

[0053] The motion estimation module 42 calculates a motion vector for a PU of an inter-coded frame by comparing the PU to reference samples of a reference frame stored in the reference frame buffer 64. In some examples, the video encoder 20 may calculate values for sub-integer pixel positions of reference frames stored in the reference frame buffer 64. For example, the video encoder 20 may calculate values of one-quarter pixel positions, one-eighth pixel positions, or other fractional pixel positions of the reference frame. Therefore, the motion estimation module 42 may perform a motion search relative to both the full and fractional pixel positions and output a motion vector with fractional pixel precision. In some embodiments, the motion search may use luma PBs, which may be sufficiently representative of motion. The motion estimation module 42 may send the calculated motion vector to the entropy encoding module 56 and the motion compensation module 44.

[0054] Motion compensation, performed by the motion compensation module 44, may involve fetching or generating values for the PU based on the motion vector, which may be determined by the motion estimation module 42. Upon receiving the motion vector for the PU of the current video block, the motion compensation module 44 may locate a corresponding block to which the motion vector points in one of the reference frames. The summer 50 forms residual data representing a residual block by subtracting pixel values of the current video block being coded, forming pixel difference values, as discussed below. In general, the motion estimation module 42 performs motion estimation relative to luma components to calculate motion vectors, and the motion compensation module 44 uses the motion vectors for both chroma components and luma components.

[0055] The intra-prediction module 46 may intra-predictively encode the received block, as an alternative to inter-prediction performed by the motion estimation module 42 and the motion compensation module 44. The intra-prediction module 46 may encode the received block (e.g., transform block) relative to neighboring, previously coded blocks, e.g., blocks above, above and to the right, above and to the left, to the left, or below and to the left of the current block assuming a left-to-right, top-to-bottom encoding order for blocks. The intra-prediction module 46 may be configured with a variety of different intra-prediction modes. For example, the intra-prediction module 46 may be configured with a certain number of directional prediction modes, e.g., thirty-three directional prediction modes, based on the size of the coding unit being encoded. The intra-prediction module 46 may also be configured with other types of prediction modes, e.g., a planar prediction mode and a DC prediction mode.

[0056] The intra-prediction module 46 may select an intra-prediction mode by, for example, calculating error values for various intra-prediction modes and selecting a mode that yields the lowest error value or the highest score by another metric. Mode selections may be made at a PU level. Directional prediction modes may include functions for combining values of spatially neighboring pixels (also referred to as “neighbor pixels”) and applying the combined values to one or more pixel positions in an associated PU. Once values for all pixel positions in a PU have been calculated, the intra-prediction module 46 may calculate an error value for the prediction mode based on pixel differences between the associated PU and the received block to be encoded. In some embodiments, the intra-prediction module may test every intra-prediction mode to determine which mode yields the lowest error value or the highest rating by another metric. In some embodiments, the intra-prediction module 46 may continue testing intra-prediction modes until an intra-prediction mode that yields an acceptable error value or a sufficient rating by another metric is determined Testing a subset of the available intra-prediction modes may decrease the processing latency and processing bandwidth required.

[0057] The intra-prediction module 46 may alternatively or additionally use rate-distortion analysis when testing various intra-prediction modes, and may select the intra-prediction mode having the best rate-distortion characteristics (among the tested modes) for each PU. Rate-distortion analysis generally determines an amount of distortion (e.g., error) between an encoded block and a corresponding original, unencoded block, as well as the number of bits (and the resulting bitrate) needed to produce the encoded block. The intra-prediction module 46 may calculate ratios from the distortions and the bitrates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block. After selecting an intra-prediction mode, the intra-prediction module 46 may send the PU associated with the selected intra-prediction mode to the summer 50. An indicator of the selected intra-prediction mode may be sent to the entropy encoding module 56 to be included in the encoded video data output by the video encoder 20.

[0058] During the mode selection phase, the intra-prediction module 46 may intra-predict values of residual blocks using information stored in the reference frame buffer 64. The intra-prediction calculation may occur at a block level (e.g., one transform unit (TU) or transform block (TB) at a time), where some TUs may be smaller than the corresponding coding units (CU) and prediction units (PU) for which the intra-prediction mode is computed. Furthermore, the intra-prediction of one TU within a CU may depend on the intra-prediction and reconstruction of neighboring TUs that are also within the CU. In accordance with the disclosed principles, an intra-prediction engine may allow for improved pipelining of consecutive blocks (e.g., TUs) during intra-prediction. These principles are discussed later in this document.

[0059] The video encoder 20 determines residual data representing a residual block by subtracting the prediction data calculated by the motion compensation module 44 or the intra-prediction module 46 from the original video block being coded. The summer 50 represents the component or components that perform this subtraction operation. The residual block may correspond to a two-dimensional matrix of pixel difference values. The values in the residual block may correspond to the differences (e.g., error) between values of co-located pixels within an intra-predicted or inter-predicted block and within the original block to be coded. The differences may be chroma or luma differences depending on the type of block that is coded.

[0060] During intra-prediction, block-sized (e.g., TU-sized) portions of a partially intra-predicted CU or PU may propagate through the summer 50, the transform module 52, the quantization module 54, the inverse quantization module 58, the inverse transform module 60, the summer 62, and the reference frame buffer 64. Accordingly, intra-prediction of a second block (e.g., TU) may utilize reference values associated with a first block that has already been encoded, even if the first and second blocks are associated with a common CU or PU.

[0061] The transform module 52 may form TUs from the residual data, where each TU may comprise a plurality of transform blocks (e.g., one luma TB and two chroma TBs). The transform module 52 applies an image transform (e.g., two-dimensional transform), such as a discrete cosine transform (DCT), a discrete sine transform (DST), a discrete Fourier transform (DFT), a Hadamard transform, a wavelet transform, an integer transform, a sub-band transform, or another type of transform, to the TU, producing a video block comprising transform coefficients. The plurality of TBs associated with a TU may be computed simultaneously or consecutively.

[0062] The transform module 52 may send the resulting transform coefficients to the quantization module 54, which may then quantize the transform coefficients. Quantization generally refers to a process in which transform coefficients are converted to a reduced set of discrete values, which results in increased efficiency (e.g., higher compression) at the cost of precision. The level of quantization may be adaptively determined, such that higher energy coefficients are quantized less than lower energy coefficients. Further, some of the lower energy coefficients may be set to a value of “0” after being quantized by the quantization module 54. This may allow for improved compression from entropy encoding, as described below.

[0063] Following quantization, the video encoder 20 may scan the transform coefficients, producing a one-dimensional array from the two-dimensional matrix having the quantized transform coefficients. The scan may be designed to place lower frequency coefficients, which often have relatively higher energy, nearer to the front of the array and to place higher frequency coefficients nearer to the back of the array. In some examples, the video encoder 20 may use a predefined scan order (e.g., a zig-zag scan pattern) to scan the transform coefficients. In other examples, the video encoder 20 may perform an adaptive scan, and the selected scan order(s) may be encoded into the data stream. The scanning may be performed, e.g., by the entropy encoding module 56.

[0064] Once the transform coefficients are scanned, the entropy encoding module 56 may apply entropy coding such as context-adaptive variable-length coding (CAVLC), probability interval partitioning entropy (PIPE) coding, or context-based adaptive binary arithmetic coding (CABAC) to the coefficients. In addition, the entropy encoding module 56 may encode motion vector (MV) information and any of a variety of syntax elements useful in decoding the video data at a video decoder. These syntax elements may be used by a video decoder when reconstructing the encoded video data. Following the entropy coding by the entropy encoding module 56, the resulting encoded video may be transmitted to another device, such as a video decoder, or archived for later transmission or retrieval.

[0065] The inverse quantization module 58 and the inverse transform module 60 apply inverse quantization and inverse transformation, respectively, to reconstruct the residual block in the pixel domain, e.g., for creating a reference block for a reference frame that is stored in the reference frame buffer 64. The modules 58, 60 may be used to simulate portions of a video decoder, and accordingly may be substantially similar to corresponding modules present in a video decoder. The motion compensation module 44, in conjunction with the summer 62, may calculate a reference block by adding the residual block to a predictive block of one of the frames of the reference frame buffer 64. The motion compensation module 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values for use in motion estimation. The summer 62 may add the reconstructed residual block to the motion compensated prediction block produced by the motion compensation module 44 to produce a reconstructed video block for storage in the reference frame buffer 64. The reference frame buffer 64 may hold substantially the same video information that is received in a decoder, which may allow the video encoder 20 to accurately encode intra-prediction modes and motion vectors based on this common “state” information seen by the decoder during the time when the intra-prediction modes and motion vectors are applied.

[0066] The reconstructed video block stored in the reference frame buffer 64 may be used by the motion estimation module 42 and the motion compensation module 44 as a reference block to inter-code a block in another video frame. Additionally, or alternatively, the reconstructed video block may be used by the intra-prediction module 46 to intra-code a block in the same video frame. In some scenarios, a particular reference block may not be used for either intra-prediction or inter-prediction, but may merely be a part of a reference frame having other, useful reference blocks.

[0067] FIG. 3 shows a block diagram illustrating an example of a video decoder 30 that may be implemented in accordance with the disclosed principles. The video decoder 30 may be configured to perform any or all of the techniques of this disclosure.

[0068] In the example of FIG. 3, the video decoder 30 includes an entropy decoding module 70, a motion compensation module 72, an intra-prediction module 74, an inverse quantization module 76, an inverse transformation module 78, a reference frame buffer 82, and a summer 80. The video decoder 30 may, in some embodiments, perform a decoding pass generally reciprocal to the encoding pass described with respect to the video encoder 20 as described in FIG. 2.

[0069] Although shown as separate functional modules for ease of illustration, the structure and functionality of any of the components within the video decoder 30 may be highly integrated with one another. For example, in some embodiments, the entropy decoding module 70 may be highly integrated with the inverse quantization module 76.

[0070] During the decoding process, the video decoder 30 receives an encoded video bitstream that represents video blocks of an encoded video slice and associated syntax elements, as generated by a video encoder. The bitstream may be received from a variety of sources, including from a file server, from a storage medium, and from a video encoder via a modem, as shown in FIG. 1.

[0071] The entropy decoding module 70 may scan the received values using a scan mirroring the scan order used by the entropy encoding module of the video encoder that generated the bitstream (e.g., an inverse zig-zag scan pattern). Additionally, or alternatively, the scan order may be inferred by the video decoder 30 based on characteristics of the coded video such as prediction mode, block size, transform, or other characteristics. No matter how the scan order is determined, the entropy decoding module 70 may use the inverse of the scan order to scan the received one-dimensional array into a two-dimensional matrix. The entropy decoding module 70 may then entropy decode the encoded video, which may involve a process that is the inverse of that used by the entropy encoding module 56 of FIG. 2.

[0072] The video decoder 30 may receive the associated syntax elements at a video slice level and/or a video block level. The entropy decoding module 70 may further analyze the bitstream to extract quantized coefficients, motion vectors, intra-prediction mode indicators, and other syntax elements. The motion vectors may be received by the motion compensation module 72 and the intra-prediction mode indicators may be received by the intra-prediction module 74.

[0073] When the slice is coded as an intra-coded (I) slice, the intra-prediction module 74 may generate prediction data for a video block of the current video slice based on an intra-prediction mode indicator and data from previously decoded blocks of the current frame or picture. This process is covered in more detail in FIGS. 4-5 and the accompanying descriptions.

[0074] When the video frame is coded as an inter-coded (e.g., B, P, or GPB) slice, motion compensation module 72 may produce prediction units (PU) for a video block of the current video slice based on a motion vector and other syntax information. The PUs may be produced from one of the reference pictures within one of the reference picture lists. The video decoder 30 may construct the reference frame lists (e.g., List 0 and List 1, as defined by the HEVC standard), using default construction techniques based on reference pictures stored in the reference frame buffer 82. The motion compensation module 72 may determine prediction information for a video block by parsing the motion vectors and other syntax information. This prediction information may be used to produce the PUs for the current video block being decoded. For example, motion compensation module 72 may determine a prediction mode (e.g., intra- or inter-), an inter-prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more reference picture lists for the slice, motion vectors for each inter-encoded video block of the slice, inter-prediction status for each inter-coded video block of the slice, and other information to decode the video blocks in the current video slice.

[0075] The motion compensation module 72 may also perform interpolation based on interpolation filters. The motion compensation module 72 may use interpolation filters similar to those used by the video encoder that encoded the video blocks to calculate interpolated values for sub-integer pixels of reference blocks. In some embodiments, the motion compensation module 72 may determine the interpolation filters used by the video encoder that generated the bitstream from the received syntax elements and use the interpolation filters to produce predictive blocks.

[0076] The inverse quantization module 76 inverse quantizes (e.g., de-quantizes) the quantized transform coefficients provided in the bitstream and decoded by entropy decoding module 70. The inverse quantization process may include use of a quantization parameter, QP_Y, calculated by the video decoder 30 for each video block in the video slice to determine a degree of quantization and, likewise, a degree of inverse quantization that may be applied to determine the original transform coefficients, though some quantization error may be present.

[0077] The inverse transform module 78 may apply an inverse transform (e.g., an inverse DCT) to form residual data in a pixel or pixel difference domain. The inverse transform may be the inverse of that used by the transform module 52 of FIG. 2. The residual data may be computed in residual blocks having the same size as the transform units (TU), and thus transform blocks (TB), being decoded.

[0078] After the motion compensation module 72 and/or the intra-prediction module 74 generates a block-sized (e.g., TU-sized) portion of a coding unit (CU), the video decoder 30 forms a decoded video block by summing the residual block from the inverse transform module 78 with the corresponding block-sized predictive data generated by motion compensation module 72 and/or the intra-prediction module 74. The summer 80 represents the component or components that perform this summation operation and store the results in the reference frame buffer 82.

[0079] During intra-prediction, the summer 80 may iteratively sum each inverse transformed block with a corresponding block from the reference frame buffer 82, where the corresponding block may be the same size as the inverse transformed block and selected based, at least in part, on an intra-prediction mode indicator. Accordingly, the decoded blocks that are written to the reference frame buffer may correspond with and have the same size as TUs. By iteratively reconstructing and storing TU-sized decoded blocks into the reference frame buffer 82, a full, decoded CU may be determined at the reference frame buffer 82.

[0080] One objective of the present disclosure is to optimize scenarios where the TU and CU sizes differ, such that intra-prediction processing is further parallelized.

[0081] If desired, a deblocking filter may be applied to filter the decoded blocks in order to remove blockiness artifacts. Other loop filters (e.g., in the coding loop or after the coding loop) may also be used to smooth pixel transitions, or otherwise improve the video quality. The decoded video blocks in a given frame or picture are then stored in the reference frame buffer 82. The reference frame buffer 82 may store this and other reference frames or pictures that may be used for subsequent motion compensation and/or intra-prediction. The reference frame buffer 82 may also store decoded video for presentation on a display device.

[0082] FIG. 4 shows a schematic diagram illustrating various modes for intra-prediction. Specifically, FIG. 4 illustrates thirty-five intra-prediction modes that may be used for HEVC coders. FIG. 4 is in the context of HEVC for purposes of illustration, but without limitation of this disclosure as to other coding standards or methods that may benefit from an improved intra-prediction process.

[0083] Images and video frames generally include spatially redundant data. Spatial redundancy implies that a given pixel may generally be correlated to at least one of the surrounding pixels. Intra-prediction allows for pixels within a block to be interpolated based on neighbor pixels. The selected neighbor pixels may be located in other blocks that have already been calculated, so as to remove dependencies within the block, thereby promoting parallel intra-prediction of multiple pixels within the block. For example, when using a directional mode, the neighbor pixels may be selected as the nearest pixels in a mode-defined direction that are outside of the block being calculated. This concept of using pixels from other blocks is further exemplified in FIG. 5 below.

[0084] As shown in FIG. 4, an intra-prediction value may be calculated for a pixel 401 based on an intra-prediction mode. The intra-prediction mode may be computed on a per prediction unit basis at a video encoder. When a prediction unit is the same size as a corresponding coding unit and its associated coding blocks, each transform block within the coding blocks may have a common intra-prediction mode. Intra-prediction mode indicators may encoded into a bitstream along with the other video data, so that the mode indicators may be used during a decoding process by a decoder. Some coding units (e.g., those that are inter-coded) may not have corresponding intra-prediction mode indicators.

[0085] Mode 0 may be used for a planar coding technique, where the pixel 401 is calculated according to a planar function defined by the HEVC standard. Mode 1 may be used for a DC coding technique; where the pixel 401 is calculated as the average of the neighbor pixels. Modes 2-34 may be used for directional coding techniques. Here, the arrows associated with Modes 2-34 indicate a search direction from which to find neighbor pixels for determining the value of the pixel 401. In situations where the arrow points between two neighbor pixels, an weighted interpolation function may be used to determine an intermediate value that may be a weighted average of the two neighbor pixels.

[0086] In HEVC, the intra-prediction scheme may be assumed to involve calculating blocks from left-to-right and then top-to-bottom (e.g., as one would read an English-language book), or from top-to-bottom and then left-to-right (e.g., as one would read some traditional East Asian scripts). As would be understood by one of ordinary skill in the art, it would be possible to rotate or otherwise alter the block calculation order (e.g., so that blocks are intra-predicted from right-to-left, then bottom-to-top). In those scenarios, the directional intra-prediction modes may be changed (e.g., rotated) to better promote usage of blocks that have already been calculated.

[0087] Depending on the selected mode, intra-prediction for pixels in a block may depend on the neighbor pixels in one or more other blocks. For example, when intra-prediction mode 26 is selected, intra-prediction of a block comprising pixel 401 may depend on a pixel in the bottom row from a neighboring block above the block. In another example, when intra-prediction mode 10 is selected, intra-prediction of a block comprising pixel 401 may depend on a pixel in the rightmost column of a neighboring block to the right of the block. As a result of these dependencies, intra-prediction may not be arbitrarily applied in parallel on neighboring blocks. As shown by the arrows, the other blocks providing neighbor pixels may be above, above and to the right, above and to the left, to the left, and below and to the left of the current block upon which intra-prediction is performed.

[0088] FIG. 5 shows schematic diagram illustrating the dependencies of a block 511 on neighboring blocks. Each block may, for example, represent a transform unit that is within a common prediction unit to be intra-predicted. The blocks are separated by thick borders in the figure. Each block may have a size of 4×4 pixels. For example, the block 511 may comprise 16 pixels labeled “a” through “p.”

[0089] Blocks 500, 501, 502, 510 and 520 are not completely shown. The block 501 comprises pixels “A,” “B,” “C,” and “D” in its bottom row; the block 502 comprises pixels “E,” “F,” “G,” and “H” in its bottom row; the block 510 comprises pixels “I,” “J,” “K,” and “L” in its rightmost column; the block 520 comprises pixels “M,” “N,” “O,” and “P” in its rightmost column; and the block 500 comprises pixel “R” in its bottom-right corner.

[0090] As described above, some video coding standards such as HEVC dictate that the intra-prediction of pixels within the block 511 depend on at least some of the reconstructed values of neighbor pixels within at least one of the neighboring blocks 500, 501, 502, 510 and 520. The nature of the dependency is characterized by an intra-prediction mode. For example, if a vertical intra-prediction mode (e.g., Mode 26 in HEVC) is selected, intra-prediction of the pixels “a” through “p” in the block 511 may rely upon the reconstructed values of the pixels “A” through “D” in the block 501. The reconstructed pixel data may become available after intra-prediction of the block 501 is complete, or, at minimum, after intra-prediction of the pixels “A” through “D” is complete. Accordingly, intra-prediction of the block 511 may be restricted to begin after intra-prediction of the block 501 is at least partially completed, for certain intra-prediction modes. In another example, if a horizontal intra-prediction mode (e.g., Mode 10 in HEVC) is selected, intra-prediction of the block 511 may rely upon the reconstructed values of the pixels “I” through “L” in the block 510. As another example, if a diagonal intra-prediction mode such as Mode 2 in HEVC is selected, intra-prediction of block 511 may involve calculations based on the pixels “I” through “P,” which are located in the blocks 510 and 520. Some intra-prediction modes such as Mode 0 (DC mode) in HEVC may even utilize values from each of the blocks 500, 501, 502, 510 and 520 (if available).

[0091] In a pipelined architecture, all available intra-prediction modes may need to be supported, and so the at least partial reconstruction of the blocks 500, 501, 502, 510 and 520 may be beneficial before intra-prediction of 511 begins. As will be described below, a coding block may be selectively rotated or transposed based on an intra-prediction mode so as to remove the dependency upon one of the neighboring blocks (e.g., the block 520).

[0092] FIG. 6 shows a block diagram illustrating an intra-prediction subsystem 600. The subsystem 600 includes an intra-prediction module 610, an inverse transform module 620, a reference frame buffer 630, and a summer 640. The subsystem 600 may be incorporated into video encoders and video decoders. For example, if the subsystem 600 is incorporated into the video encoder 20 of FIG. 2, the intra-prediction module 610 may represent the intra-prediction module 46 of FIG. 2, the inverse transform module 620 may represent the inverse transform module 60 of FIG. 2, the reference frame buffer 630 may represent the reference frame buffer 64 of FIG. 2, and the summer 640 may represent the summer 62 of FIG. 2.

[0093] Similarly, if the subsystem 600 is incorporated into the video decoder 30 of FIG. 3, the intra-prediction module 610 may represent the intra-prediction module 74 of FIG. 3, the inverse transform module 620 may represent the inverse transform module 78 of FIG. 3, the reference frame buffer 630 may represent the reference frame buffer 82 of FIG. 3, and the summer 640 may represent the summer 80 of FIG. 3. Certain components of the subsystem 600 (e.g., the reference frame buffer 630) may be additionally used in other subsystems.

[0094] The subsystem 600 may be designed to optimize the intra-prediction of consecutive blocks, even if intra-prediction of a given block may depend, at least in part, on the block immediately preceding it in the intra-prediction order. For example, intra-prediction of each block may depend on reconstructed pixels for either a critical row, a critical column, or neither in the preceding block, as will be shown in further detail in FIG. 7.

[0095] Returning to FIG. 6, the intra-prediction module 610 may comprise a neighbor buffer 612, a neighbor processing unit 614, and an intra-prediction processing unit 616. The neighbor buffer 612 may store and retrieve neighbor pixels using logic and/or circuitry to interface with a neighbor RAM 613. The neighbor pixels may comprise pixels that may be used when intra-predicting the blocks. In the context of HEVC, a block may represent a transform unit which may be associated with a coding unit (and a prediction unit) having an equal or larger size. The pixels may comprise chroma and/or luma values, depending on the type of block that is being intra-predicted. Here, the block size may be M×M, where M is a nonnegative integer value. Exemplary values for M include 4, 8, 16, and 32.

[0096] The pixels stored in the neighbor buffer 612 may need to be filtered, padded, and potentially conditioned in other ways before the pixels are received by the intra-prediction processing unit 616. These conditioning actions may be performed, at least in part, by a neighbor processing unit 614 that receives the pixels from the neighbor buffer 612 in preparation for processing by the intra-prediction processing unit 616. For example, the neighbor processing unit 614 may prepare 4M+1 neighbor pixels for the intra-prediction processing unit 616 to process a M×M block. These neighbor pixels may, for example, represent the neighbor pixels “A” through “R,” as shown in FIG. 5.

[0097] Returning to FIG. 6, the intra-prediction processing unit 616 may comprise processing engines 617-1 to 617-N (also labeled as PE 1 to PE N), which may be implemented as hardware in a video encoding and/or video decoding device. The N processing engines 617 may each simultaneously operate on a common set of inputs received from the neighbor processing unit 614 to intra-predict different pixels within a block being processed. The processing engines 617 may each be capable of computing one pixel per cycle (e.g., clock cycle), and each processing engine 617 may calculate a different pixel or set of pixels within the block. Accordingly, the intra-prediction engine may be capable of computing N pixels per cycle, which may be received by the summer 640.

[0098] The summer 640 may add each set of N intra-predicted pixels that is output by the intra-prediction processing unit 616 to N corresponding residual values calculated by the inverse transform module 620, thereby reconstructing N pixels. The reconstructed pixels may be stored in the reference frame buffer 630. Some reconstructed pixels may be used for intra-predicting a subsequent block, and these reconstructed pixels may additionally be stored in the neighbor buffer 612. In some embodiments, the neighbor buffer 612 reads these reconstructed pixels from the reference frame buffer 630, and, in some embodiments, the summer 640 outputs these reconstructed pixels to both the reference frame buffer 630 and the neighbor buffer 612.

[0099] The number of processing engines, N, may scale with the memory bandwidth of the neighbor buffer 612. For example, in some embodiments, the read bandwidth of the neighbor buffer may be 2N pixels per cycle. However, other ratios of buffer read bandwidth to the number of processing engines may be used. Further, the shape of the neighbor RAM 613 may relate to the read bandwidth and/or the number of processing engines. For example, the neighbor RAM 613 may be configured to have a shape and size of 192/N×16N, where the RAM depth may be 192/N and the RAM width may be 16N. While the neighbor buffer 612 is shown to include RAM 613, other suitable types of memory may additionally or alternatively be used.

[0100] When intra-predicting pixels in a block, the intra-prediction processing unit 616 may prioritize pixels that will be used for intra-prediction of subsequent blocks (e.g., the block immediately following the current block). After these pixels are intra-predicted, they may be summed with corresponding residual values at the summer 640 to generate reconstructed pixels that may be stored back in the neighbor buffer 612 (as well as in the reference frame buffer 630). The reconstructed pixels may then be filtered, padded, and potentially conditioned in other ways for the intra-prediction processing unit 616 by the neighbor processing unit 614. With a suitable prioritization of pixels, the neighbor processing unit 614 may have the reconstructed residual values for the next block ready before the intra-prediction processing unit 616 finishes intra-predicting the current block. Accordingly, the intra-prediction processing unit 616 may begin processing the next block immediately after the current block is finished, even when the next block has dependencies on the current block, and/or when the current block and the next block are within a common coding unit. This is a significant improvement over traditional methodologies.

[0101] FIG. 7 shows a block diagram illustrating a coding block 700 that is selected for intra-prediction. The coding block 700, may have a plurality of transform blocks 710-0, 710-1, 710-2, and 710-3, which may generally be processed in that order. In non-HEVC contexts, the coding block 700 may refer to a macroblock 700 and the transform blocks 710-0, 710-1, 710-2, and 710-3 may more generally refer to blocks 710-0, 710-1, 710-2, and 710-3. During intra-prediction, the coding block 700 may have a single intra-prediction mode that is shared by each of the corresponding transform blocks 710. The blocks 710 may be the unit by which a reference frame buffer is updated during reconstruction.

[0102] In some scenarios, a transform block may be the same size as a corresponding coding block, while in other scenarios (e.g., as shown in FIG. 7), the transform blocks may be smaller than the associated coding block (which itself may be associated with a single prediction block). Accordingly, a group of neighboring transform blocks may have the same intra-prediction mode. As a result, the transform blocks within a coding block may have a regular pattern of dependency with previous transform blocks during intra-prediction.

[0103] As an example of block dependency, if a horizontal intra-prediction mode or certain diagonal intra-prediction modes are selected, intra-prediction of the block 710-1 (also labeled as Block 1) may be dependent on reconstruction of a critical column 720 (e.g., a rightmost column) of the block 710-0 (also labeled as Block 0). Similarly, if certain diagonal intra-prediction modes are selected, intra-prediction of the block 710-2 (also labeled as Block 2) may depend on reconstruction of a critical row 722 (e.g., a bottom row) of the block 710-1.

[0104] The critical column 720 may be the only part of the block 710-0 that the block 710-1 is dependent on during intra-prediction. The other neighbor pixels (e.g., those above the block 710-1) may already be reconstructed and stored in the neighbor buffer as a result of the completed reconstruction of a coding block above the coding block 700. Accordingly, the critical column 720 may be prioritized during the intra-prediction of the block 710-0. When reconstructed pixels for the critical column 720 are determined, preparation for the intra-prediction of block 710-1, (e.g., filtering and padding) may begin. This preparation may occur and be completed while the intra-prediction of the block 710-0 is still in progress, as will be shown below in FIG. 8. Accordingly, the block 710-1 may be intra-predicted immediately after the intra-prediction processing unit finishes processing the block 710-0.

[0105] Similarly, the critical row 722 may be the only part of the block 710-1 that the block 710-2 is dependent on during intra-prediction. The other neighbor pixels (e.g., those to the left of the block 710-2) may already be reconstructed and stored in the neighbor buffer as a result of the completed reconstruction of a coding block to the left of the coding block 700. Accordingly, the critical row 722 may be prioritized during the intra-prediction of the block 710-1, and the block 710-2 may be intra-predicted immediately after the intra-prediction processing unit finishes processing the block 710-1.

[0106] When a critical row or column is prioritized, the N processing engines (as shown in FIG. 6) may each select a different pixel or set of pixels to process. As a single row or column in an M×M block may have M pixels, M/N cycles may be required to intra-predict a single critical row or column. This may be substantially less than the M*M/N cycles that may be required to process an entire block, which may provide sufficient time for the neighbor processing unit to prepare the neighbor pixels for the next block.

[0107] For example, if there are two processing engines working on the block 710-0, which is shown to be a 4×4 block, the first processing engine may calculate an intra-predicted value for the pixel at the intersection of column C3 and row R0 during a first cycle. During this same cycle, the second processing engine may calculate an intra-predicted value for the pixel at the intersection of column C3 and row R1. In the following cycle, the first and second processing engines may interpolate values for the pixel at the intersection of column C3 and row R2 and for the pixel at the intersection of column C3 and row R3, respectively. Accordingly, in two cycles, the intra-prediction of the critical column 720 may be completed. The critical column 720 may then be reconstructed, stored, and prepared for intra-prediction of the next block (e.g., the block 710-1) during a time when the processing engines are interpolating the remaining columns. In other words, the neighbor processing performed by a neighbor processing unit is pipelined with the intra-prediction performed by an intra-prediction processing unit.

[0108] In general, the intra-prediction processing unit may selectively and flexibly select pixels for prediction. For example, as evidenced above, the intra-prediction processing unit may operate on columns or rows, depending, at least in part, on the relative position of the next block (e.g., transform block) to be intra-predicted. If the next block to be intra-predicted is to the right of the current block, a rightmost column of the current block may be prioritized. After the rightmost column is processed, the remainder of the block may be processed in a column-wise pattern. Similarly, if the next block to be intra-predicted is below or below and to the left of the current block, a bottom row of the current block may be prioritized. The remainder of the current block may be processed in a row-wise pattern.

[0109] The prioritization of critical columns and rows may be adjusted for different intra-prediction orders of blocks. Furthermore, in some embodiments, a plurality of columns or rows may be prioritized instead of a single column or row. For example, this may occur when intra-prediction of a block uses both the nearest and the second nearest row or column.

[0110] It may be noted that the block 710-1 is not shown to have any dependency on the block 710-2, even though it was shown in FIGS. 4-5 that intra-prediction may utilize pixels from blocks that are below and to the left of the block being intra-predicted. However, to prevent this dependency from unnecessarily halting or delaying intra-prediction of the block 710-1, the coding block 700 may be rotated or transposed when this type of dependency occurs. This effectively shifts the intra-prediction vector towards completed (or at least partially completed) blocks, thereby promoting optimized processing of consecutive transform blocks within a coding block.

[0111] While FIG. 7 shows a coding block having four transform blocks, the number of transform blocks within a coding block may vary. For example, a coding block may be associated with 1, 4, 16, 64, or 256 transform blocks. Furthermore, while a transform block is shown to have a size of 4×4, other block sizes may be selected. For example, in HEVC, a transform block may have a size of 4×4, 8×8, 16×16, or 32×32.

[0112] FIG. 8 shows a timing diagram illustrating an interleaved intra-prediction technique. Specifically, the timing diagram shows a first timeline 801 for intra-prediction of a first block (Block 0), and a second timeline 802 for intra-prediction of a second block (Block 1). In the context of HEVC, these blocks may represent transform blocks. The timing diagram is associated with an embodiment of the intra-prediction subsystem shown in FIG. 6, and the block size is assumed to be M×M. All timings are exemplary in nature and they may vary depending on processing delays and other factors.

[0113] At a time 810, a neighbor processing unit may fetch or otherwise receive the neighbor pixels that may be used for intra-prediction of the first block. As described above, 4M+1 neighbor pixels may be received, for example, from a neighbor buffer. Here, the “1” represents a corner (e.g., top-left corner) pixel which may be stored in a separate register (and may be discounted for the timing calculation). However, the corner pixel may additionally or alternatively be stored in the neighbor buffer. If the read bandwidth from the buffer is 2N pixels per cycle, then it may take approximately 2M/N cycles for the neighbor processing unit to read all of the neighbor pixels. This action may be completed at a time 820.

[0114] After the neighbor pixels are fetched, they may be padded, filtered, and/or conditioned in other ways in preparation for the intra-prediction processing unit. This may take a fixed amount of cycles (e.g., two cycles) and may be completed at a time 830.

[0115] At the time 830, the intra-prediction processing unit may begin intra-predicting the first block, starting with a critical row or column. The critical row or column in an M×M block may have M pixels, and the intra-prediction processing unit may be capable of processing N pixels per cycle. Accordingly, after approximately M/N cycles, the critical row or column may be intra-predicted. The intra-predicted values may be summed with the corresponding residual values to create reconstructed pixels that are stored in the neighbor buffer, so that they are ready to be received by the neighbor processing unit at a time 840.

[0116] At the time 840, the neighbor processing unit may fetch or otherwise receive the reconstructed neighbor pixels that may be used for intra-prediction of the second block. As mentioned above, some of the received pixels may correspond with the critical row or column of the first block, thereby causing a delay between the time 830 when the neighbor processing unit has prepared and sent the neighbor pixels for the first block to the intra-prediction processing unit and the time 840 when the neighbor processing unit begins receiving neighbor pixels for the second block.

[0117] Over approximately 2M/N cycles, the neighbor processing unit may receive each of the neighbor pixels for intra-predicting the second block, which may be completed at a time 850. After a fixed amount of cycles (e.g., two cycles) for padding, filtering, and/or otherwise conditioning the neighbor pixels, the neighbor processing unit may finish preparing the pixels for the intra-prediction processing unit.

[0118] At a time 860, the intra-prediction processing unit may complete processing the first block and may begin processing the second block. In other words, the second block may be processed immediately after the first block, because the neighbor processing unit may be ready with the neighbor pixels for the second block at the time 860. In some embodiments, the neighbor processing unit may complete preparation (e.g. padding and filtering) of the neighbor pixels for the second block prior to the time 860.

[0119] From the perspective of the intra-prediction processing unit, the first block is received from the neighbor processing unit at the time 830 and processed in approximately M*M/N cycles ending at the time 860. As soon as the first block is completely processed, the intra-prediction processing unit may begin processing the second block, using the corresponding 4M+1 neighbor pixels prepared by the neighbor processing unit. After another time period of approximately M*M/N cycles, the intra-prediction processing unit completes processing the second block (at a time 870) and may immediately begin processing a third block using the corresponding 4M+1 neighbor pixels that may already be prepared by the neighbor processing unit.

[0120] It may be noted that the neighbor processing unit may process (e.g., read from a buffer, pad, and filter) neighbor pixels for the second block during a time when the intra-prediction processing unit is still processing (e.g. intra-predicting) the first block. In this way, the neighbor processing unit may be pipelined (and interleaved) with the intra-prediction processing unit, which can increase efficiency and throughput of the intra-prediction module and subsystem.

[0121] Often, in pipelined architectures, data should remain in buffers for minimal periods of time, because stationary data may lead to (or be indicative of) inefficiencies and lower throughput. It may be noted that the processing cycle for each block is approximately M*M/N cycles, and accordingly, this may also be the average time that each set of 4M (or 4M+1) neighbor pixels may be stored in the neighbor buffer. This shortened period of storage (compared to conventional designs) is indicative of the increased throughput and efficiency that is achievable for intra-prediction modules and subsystems designed in accordance with the disclosed principles.

[0122] Again, the exact number of cycles for any of the actions or periods shown in FIG. 8 are merely exemplary, and they may vary depending on delays due to processing and other factors.

[0123] The number of processing engines (N) and the neighbor buffer read bandwidth (e.g., 2N pixels per cycle) may both scale according to different performance requirements. Table 1 below shows various configurations in accordance with the disclosed principles.

	TABLE 1
	Configuration 1	Configuration 2	Configuration 3	Configuration 4
Cycle Count Budget for 32 × 32	400	800	1600	3200
Coding Unit
Neighbor Buffer Size	2 × 24 × 64	48 × 64	96 × 32	192 × 16
Number of PEs in intra-	8	4	2	1
prediction processing unit
Neighbor Buffer Read	16	8	4	2
Bandwidth
Area (um2)	50000	36000	26550	20500

[0124] The values shown in Table 1 were measured or otherwise determined with respect to hardware designs running at 400 MHz using a 28 nm technology node. The technology node and operating frequency are purely exemplary and other technology nodes and/or operating frequencies may be used.

[0125] The table shows that the number of processing engines (PEs) as well as the neighbor buffer read bandwidth (and, e.g., neighbor RAM width) may increase to accommodate higher performance requirements, where the performance requirements are given as a cycle count budget. For example, Configuration 1 may be selected on high-end coding devices designed to encode and/or decode video, e.g., having Ultra High Definition (UHD) resolution (3,840 by 2,160 pixels or greater). Configurations 2 and 3 may be oriented towards mid-level devices (e.g., 1080p), and Configuration 4 may be oriented towards low-end devices (e.g., 720p).

[0126] While various embodiments in accordance with the disclosed principles have been described above, it should be understood that they have been presented by way of example only, and are not limiting. Thus, the breadth and scope of the disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the claims and their equivalents issuing from this disclosure. Furthermore, the above advantages and features are provided in described embodiments, but shall not limit the application of such issued claims to processes and structures accomplishing any or all of the above advantages.

[0127] Various terms used in the present disclosure have special meanings within the present technical field. Whether a particular term should be construed as such a “term of art” depends on the context in which that term is used. “Connected to,” “in communication with,” “associated with,” or other similar terms should generally be construed broadly to include situations both where communications and connections are direct between referenced elements or through one or more intermediaries between the referenced elements. These and other terms are to be construed in light of the context in which they are used in the present disclosure and as one of ordinary skill in the art would understand those terms in the disclosed context. The above definitions are not exclusive of other meanings that might be imparted to those terms based on the disclosed context.

[0128] Words of comparison, measurement, and timing such as “at the time,” “immediately,” “equivalent,” “during,” “complete,” “identical,” and the like should be understood to mean “substantially at the time,” “substantially immediately,” “substantially equivalent,” “substantially during,” “substantially complete,” “substantially identical,” etc., where “substantially” means that such comparisons, measurements, and timings are practicable to accomplish the implicitly or expressly stated desired result.

[0129] Where the term “block” is used, this may refer to any of a largest coding unit (LCU), a coding tree block (CTB), a coding unit (CU), a coding block (CB), a prediction unit (PU), a prediction block (PB), a transform unit (TU), a transform block (TB), a sub-block as defined by H.264/AVC, and/or similar data structures depending on the context in which it is used.

[0130] While various subject matter is described with respect to HEVC, this subject matter may also be applicable to other standards, including those currently existing, under development, and yet to be developed. For example, where the terms “CU” and “CB” are used, they may also refer to a macroblock as defined by the H.264/AVC standard and/or similar data structures of other encoding standards, depending on the context in which they are used. Similarly, where the terms “TU” and “TB” are used, they may also refer to a sub-block as defined by the H.264/AVC standard and/or similar data structures of other encoding standards, depending on the context in which they are used.

[0131] Further, where the term “pixel” is used, this may refer to a luma value, chroma value, or a combination of chroma and luma values within a video frame or image.

[0132] The disclosed principles are generally compatible with numerous other techniques for achieving parallelism, included wavefront parallel processing (WPP), video slices, and video tiles.

[0133] Additionally, the section headings herein are provided for consistency with the suggestions under 37 C.F.R. 1.77 or otherwise to provide organizational cues. These headings shall not limit or characterize the subject matter set forth in any claims that may issue from this disclosure. Specifically and by way of example, although the headings refer to a “Technical Field,” such claims should not be limited by the language chosen under this heading to describe the so-called technical field. Further, a description of a technology in the “Background” is not to be construed as an admission that technology is prior art to any subject matter in this disclosure. Neither is the “Summary” to be considered as a characterization of the subject matter set forth in issued claims. Furthermore, any reference in this disclosure to “invention” in the singular should not be used to argue that there is only a single point of novelty in this disclosure. Multiple inventions may be set forth according to the limitations of the multiple claims issuing from this disclosure, and such claims accordingly define the invention(s), and their equivalents, that are protected thereby. In all instances, the scope of such claims shall be considered on their own merits in light of this disclosure, but should not be constrained by the headings set forth herein.