Just in time transcoder system, method and architecture for audio and video applications

[0001] This patent application claims the priority and benefits of U.S. Provisional Application No. 63/629,540, filed on Nov. 6, 2023, the disclosure of which is incorporated by reference herein in its entirety.

[0002] The present principles relate to video and/or audio processing, and more specifically, to Just-In-Time-Transcoding (JITT) video and/or audio systems and methods.

[0003] The surge in video streaming services and the increasing reliance on video in all web-related content necessitates the need for affordable, high-quality streaming technology. The varying usage scenarios and deployment situations greatly influence the streaming system costs. For instance, cloud DVR systems can place a substantial financial strain on storage, but the expenses associated with the Content Distribution Network (CDN)/edge distribution architecture should not be overlooked either. Factors such as the application (e.g., DVR, VOD, Live), the quantity of channels or number of media assets stored, the user count, and the CDN or distribution network's architecture/bandwidth all contribute to shaping the system's design and cost centers.

[0004] While advancements in various technologies have boosted streaming performance and cost-efficiency, the deployment of streaming systems remains a considerable investment. Content-aware transcoding and just-in-time packaging (JITP) are two technologies that have positively impacted streaming deployments.

[0016] In this disclosure, we will introduce and discuss “Just-in-Time Transcoding/Transcoder” (JITT). We will explain how a JITT product sets a new standard for affordable video streaming deployment, eases generational codec transitions and enhances video quality and performance for the end-users in the process. An example of such a JITT product is a Blazar Just-in-Time Transcoder being designed by igolgi Inc. of Princeton, NJ.

[0017] In addition, another benefit of the Blazar JITT is that it reduces energy consumption per stream, helping to reduce carbon emissions.

ABR Streaming Background

[0018] A video streaming system employs a technique known as adaptive bitrate (ABR) encoding. The audio and video streams are first chopped up into small time chunks known as segments. These segments may last anywhere from 2 to 10 seconds (this is only a guideline, not a requirement). The file segments are then encoded into a variety of resolutions and bitrates, referred to as a “bitrate ladder”. The audio signal is similarly divided into several audio channels and bitrates. The goal of ABR is to deliver the appropriate version of video and audio segments based on the capabilities of the receiving device and the available bandwidth. The entire streaming operation is thus converted to a series of file transfers (that can be supported by any Internet server) from the headend to the client devices. Rate selection is done by each individual client based on a manifest of rates provided by the ABR encoding head end.

[0019] FIG. 1 shows a high-level view of an ABR streaming system 100 with a cloud DVR. The video/audio content is transformed into various profiles and kept in storage. When a device asks for this content, the Origin Server initially sends a manifest to the requesting device detailing the available profiles. The device, in turn, selects the profile that best fits its characteristics (such as 4K TV, HDTV, smartphone, laptop, and so on) and the bandwidth it has at its disposal, then requests that specific profile. If there are fluctuations in the available bandwidth, the device has the flexibility to request different profiles as needed.

[0020] As alluded to earlier, another key feature of this system 100 is the division of content into brief time segments. The advantage of this is that the device receiving the content can easily switch between profiles by just requesting the next small segment from a different profile. Common packaging formats in use are HTTP Live Streaming (HLS) and Dynamic Adaptive Streaming over HTTP (DASH). The process of packaging these segments can happen at various stages in the network: before the storage, after storage in the Origin Server, or at the edge of the network. Each of these packaging architecture options has tradeoffs with storage size, core network bandwidth, and edge cache complexity and edge storage.

[0021] Just-In-Time Packaging (JITP) is an enhanced version of packaging that's particularly efficient when various digital rights management (DRM) or packaging formats are required for different types of devices. In this setup, the original video and audio signals are stored just once, but they are packaged in real time to satisfy the specific needs of the receiving device. This method significantly cuts down on storage and bandwidth costs.

Bitrate Ladders

[0022] Streaming systems typically have many different resolutions and bitrate profiles available so that each client can stream the content efficiently. In FIG. 2, we show two example bitrate ladders in Table 200. In the first Static Ladder, 8 different profiles are provided in typical resolution and bitrate combinations. Static Ladders are often used in live systems. With a large number of profiles, the endpoint has more choices to choose a profile that dynamically matches its bandwidth and device type requirements. The downside of having this many profiles in the ladder is that the DVR storage requirement is high. This also may reduce the efficiency of the edge caches since more profiles may be stored at the edge.

[0023] In the Content Aware Ladder system, a smaller number of profiles are generated using “content-aware” encoding, which creates unique profiles based on the specific requirements of the content. For instance, a high-action movie may need different combinations of resolutions and bitrates than a news program featuring people simply talking. Video-on-Demand (VOD) systems are good candidates to employ a content-aware ladder. While Digital Video Recorder (DVR) systems might also use content-aware encoding, they may require re-encoding from the live signals.

[0024] The advantage of an Automated (content-aware) ladder is that it uses fewer profiles. This means a DVR system would use less storage, and the bandwidth needed by the Content Delivery Network (CDN) to deliver a certain video quality to a specific client could also be marginally reduced compared to the static ladder. In the provided example shown in Table 200, the content-aware ladder results in a 60% reduction in storage requirements.

[0025] In both the content aware ladder or static ladder, neither provides an optimal profile based on the end-client screen size or dynamically available bandwidth. Since both systems require transcoding to the ladder profiles and storing ahead of time in the DVR, the end client can only choose from a limited set of operating points. The static ladder provides more operating points so gives more choices for bandwidth options but requires more storage. The content aware ladder uses less storage but does not give as many options for bandwidth which creates less efficient streaming for certain end point devices.

[0026] For example, if an end client had available 1400 Kbps, the static ladder could provide the 540p profile at 1200 Kbps, while the content aware could only provide 480p at 811 Kbps, so in this case the content aware system actually produced worse quality than the static system.

[0027] The result is that neither of these solutions are ideal.

Blazar JITT

[0028] Just-in-Time Transcoding (JITT) is a system and a method where compressed video and audio signals are converted from one format to another at the precise moment when a request for the signal is made by a video or audio player (client). This necessitates that the transcoding system functions at a speed much faster than real time. Blazar, a JITT transcoder, possesses several unique characteristics, primarily its ability to transcode at a speed up to 100 times faster than real time. Blazar JITT is also dynamically scalable which allows for an array of new applications and use cases.

[0029] Blazar JITT fully leverages the latest developments in CPU and GPU technologies and makes optimal utilization of the various resources in both computational platforms. While JITT has been contemplated in the past, neither the computational power nor the SW architecture could achieve the types of consistent gains in speed for all the hybrid operations that Blazar is able to accomplish.

Blazar JITT Ladder

[0030] The most ideal bitrate ladder would be an “infinite” ladder, offering every possible profile. This would deliver the exact resolution and bitrate that aligns with the client device, the available bandwidth from the Content Delivery Network (CDN), and the type of content, thereby ensuring the highest video quality. This is precisely what the Blazar solution delivers.

[0031] In the Blazar JTTT system, only the top most profile—the one with the highest resolution and bitrate—needs to be stored on the DVR storage system. When a client device requests the content, Blazar's Just-In-Time Transcoding (JITT) is capable of immediately converting the content to the best bitrate/resolution profile that suits the client's device type, available last-mile bandwidth, CDN bandwidth, and content type. The system architecture is shown in FIG. 3.

[0032] When the end device has sufficient bandwidth for the top profile, then the Origin can serve that profile directly from the DVR storage. When the endpoint needs to use less bandwidth, then the Origin re-directs the request to Blazar JITT which retrieves the top profile from storage and instantly transcodes it to the lower profile needed.

[0033] In practice an infinite ladder would lead to an infinitely large manifest file that the client would need to parse. Therefore, a practical approximation of an infinite ladder is to construct a ladder with 20-50 profiles, yet only the top profile media is preserved in the DVR storage. The current client can then select one from these 20 or more profiles and request it from the Origin server. Blazar is able to create the specific profile faster than real time. To the end device it appears as if the file was served from the storage server in the same amount of latency. In the future, the concept of static multiple bit rates with different profiles can be replaced by an intelligent client that can make segment by segment requests

[0034] FIG. 4 shows an exemplary Table 400 where the new Blazar infinite ladder saves 84% storage compared to a static ladder, and 59% compared to a content aware ladder. It should be noted that independent of the ladder type, Blazar JITT can save the same amount of DVR storage since only the top profile is in storage.

[0035] To put this into perspective for some use cases. A 500-channel cloud HD DVR storing all the channels for 1 year would require nearly 30 PetaBytes with the static ladder, 11.3 PetaBytes with the Content Aware Ladder, and 7.7 PetaBytes with the Blazar system that only stores the top profile.

[0036] FIG. 5 illustrates an example 500 of a 41 profile ladder with 50 Kbps increments between profiles. Using the example given earlier, where an endpoint device has 1400 Kbps available bandwidth, the Blazar ladder provides a near match at 1385 Kbps with 540p resolution. Whereas the content aware ladder has nothing near 1400 Kbps and must provide a lower resolution 480p at 811 Kbps. The static ladder has 1200 Kbps available at 540p. Worse quality than Blazar.

[0037] This example demonstrates both advantages of Blazar JITT:

• • [0038] 1) Lower Storage and bandwidth cost: Blazar JITT only stores the top profile
  • [0039] 2) Better video quality and optimum for the system available bandwidth.
    Blazar JITT Codecs

[0040] Another benefit Blazar JITT provides is an adaptable solution for the ever-evolving landscape of audio and video compression standards. During the codec transition phase, which typically lasts several years, both codec ladders need to be supported by the DVR for streaming to legacy and newer devices. In the past, this has always slowed the introduction of newer codecs in the marketplace. With Blazar JITT, the newer codec can be supported by Blazar JITT. When the number of client devices supporting the newer standard (which is usually more bandwidth efficient) reaches a critical mass, then the entire DVR library can be turned over to the newer codec and legacy devices are now supported with the same Blazar JITT. For instance, today H.264 is the most widely adopted video compression standard in client devices and set-top boxes, with newer devices increasingly integrating H.265 and AV1. The transcode conversion by Blazar would initially be from H.264 to HEVC/AV1. When the client population is more mature with HEVC or AV1, the library could be converted to that technology and all other devices would be supported by the Blazar JITT. This way, there is no requirement to pick technology winners early in the game as well.

[0041] H.265 and AV1 demonstrate bitrate efficiency surpassing H.264 by 50% or more. Consequently, employing these more efficient codecs when the end device supports them can result in significant savings in network bandwidth and edge cache storage.

[0042] Blazar broadens the concept of a bitrate ladder to encompass the video or audio compression format, effectively multiplying the effective ladder size by the number of codecs supported. A full Blazar ladder for every codec that is needed can be supported with still just a single codec and one top profile stored on the DVR storage.

[0043] This is not feasible with static or content aware ladders unless all the codec versions and associated ladders are stored which is an expensive proposition.

[0044] Blazar's dynamic ability to switch video codec formats offers several substantial benefits in terms of architecture and cost:

• • [0045] When end devices support more efficient codecs, Blazar can perform Just-In-Time Transcoding (JITT) to these lower bitrate codecs. This can lead to bandwidth savings or an enhancement in video quality for the end device.
  • [0046] The DVR storage can now maintain the highest profile in the most bitrate-efficient codec, with Blazar only transcoding to less efficient legacy codecs when an end device cannot support the newer ones. For instance, H.265 could be used for storing the highest profile in the cloud DVR. If an end device requires H.264, Blazar's JITT can transcode to H.264 for those specific devices, while transcoding to H.265 for devices that support it.
    Blazar JITT Architecture and Implementation:

[0047] FIG. 6 shows an exemplary innovative system and architecture 600 that generalizes and illustrates all the variants that have been discussed above. The Blazar JITT can accept an “arbitrary” profile request. The parameters that can be requested on the fly include any supported video or audio codec, bit rate, and resolution of the requested profile. This can be done on a segment-by-segment basis. An innovation is the computational architecture supported to enable the JITT with a predictable latency.

[0048] In FIG. 6, when the client device makes a content request, it is delivered a simple manifest that just shows the currently available profile in the store. Based on its bandwidth or computational availability, the client may request an arbitrarily lower profile or even a different codec. For example, if the profile stored is HD H.264 1280×720, 60 fps at 8 Mbps, and the available bandwidth is only 4 Mbps, the client device may have the intelligence to request a HD H.264 stream at 4 Mbps at the same resolution. In another example, if the currently available bandwidth is only 2 Mbps, the client may request an SD H.264 stream at 720×480 resolution. In another variation, when the currently available bandwidth is only 2 Mbps, the client may request an HD HEVC 1280×720 resolution stream at 2 Mbps.

[0049] In another example, the client may be constrained by computational power or ability to decode specific codecs. In this case, it can request a stream that is most suitable for its capabilities. For example, if the stored profile is a HD HEVC stream 1280×720 at 60 fps encoded at 4 Mbps, the client may request a HD H.264 segment at 60 fps and encoded at 8 Mbps because it is not capable of decoding HEVC streams. Such client requests may also be determined by the power constraints. For example, if the battery level on a mobile device falls below a threshold, the mobile client may request lower level profiles with codec choices that are more power efficient. The client may even request lower resolution profiles at lower bit rates for the same reason. The power of Blazar JITT architecture is that such decisions made by the client can be done instantaneously on a segment by segment basis based on the each client's constraints. Traditional ABR systems have not supported different codec selection on a segment by segment basis. They also only support a limited set of pre-determined profiles with a single codec. With arbitrary profile requests, the Blazar JITT system guarantees the optimal profile delivery tailored to the bandwidth constraint and the computational power constraints of each client.

On-Demand ABR Profile Generation to Support Legacy Client

[0050] Currently, adaptive bit rate (ABR) system client starts out with making an asset request. It gets returned, from the streaming server, a manifest file that lays out the available profiles that can be requested. The profiles contain the bit rates and other relevant information on the media segments. In current systems, the number of profiles in the manifest is predetermined and determines the granularity with which client requests can be made.

[0051] In our contemplated use of Blazar JITT to support the legacy client, when the asset request is first made, the server returns a manifest with a fixed ABR profile set. The Client calculates the best ABR profile to receive based on the current bandwidth. An extension to this calculation could take into account the available power in the client since a lower profile will generally consume less battery power. When the client sends back a profile request for a segment (from the list of profiles in the manifest), the server determines if the top profile was requested. With the Blazar JITT architecture, this is the only profile that is available (either file or live source). If the top profile is requested, it is returned from the live or file source to the client. If a different profile has been requested, then the JITT is invoked to create the requested profile from the live or file source. The transcoded segment (per the requested profile) is then sent to the client.

[0052] An advantage with this architecture is that the master manifest can include a large set of available fixed profiles since the Blazar JITT will make them available when requested. Since all those profiles are not created/stored unless requested, the master manifest can have many more of them compared to a system without Blazar JITT which would have to create and store all the profiles ahead of time.

Smart Client Architecture & Implementation

[0053] In the new architecture, there is no predetermined set of profiles except a single top profile.

[0054] FIG. 8 illustrates a typical client-server interaction flow 800. The exemplary flow is also described below:

• • [0055] 1) The client requests a particular asset (may be live or file source that is “streamed” using file segments).
  • [0056] 2) The server responds with a manifest containing a single default profile. In most applications, this will be presumed to be the top profile although in the most general case, this need not be the case.
  • [0057] 3) Based on the default available profile, the Client then calculates the optimal profile it needs to receive as a function of available network bandwidth and its own compute power.
  • [0058] 4a) If the conditions (network bandwidth and compute power) can support the default profile, the client requests the default profile from storage or the live source.
  • [0059] 4b) If the conditions (network bandwidth and compute power) and the client device type (characterized by resolution and screen size) support a different profile, the client device requests the profile optimum for the client. In this case the server invokes the JITT transcoder to produce the specific requested profile.
    Smart Server Architecture & Implementation

[0060] An alternative architecture to deliver optimal media based on the constraints of network bandwidth and client compute resources is to move the calculation of the segment characteristics to the server side. FIG. 9 illustrates this architecture.

[0061] In FIG. 9, the event flow 900 is pictured, as described below:

• • [0062] 1) The client requests an asset
  • [0063] 2) Server sends back a manifest with the default profile (as before).
  • [0064] 3) Using the available network bandwidth, the client can determine a bit rate that it needs to request for the next segment or segments. Based on the power constraints at the client, it can determine a video compression codec that it can best decode. Both these pieces of information are sent back to the server.
  • [0065] 4a) The server algorithm calculates the best resolution, frame rate based on the bit rate and codec choice from the client. If the default profile can be supported, then the server sends back the default profile from file or live source.
  • [0066] 4b) If the server algorithm makes a choice other than the default profile, then the Just in Time Transcoder is invoked to create the arbitrary profile on the fly and deliver it to the client.
  • [0067] Note: The server may also know the client screen resolution, screen size (and perhaps other information) to also help determine the profile.
    Architecture to Support Legacy Clients with JITT Server

[0068] In both the smart client and smart server architectures discussed above, there are semantic changes in the client or server workflows that are needed in order to leverage the arbitrary profile creation and optimal use of the system. We now discuss a legacy mode support architecture that accommodates legacy devices to still operate with the JITT and take advantage of near-optimal bandwidth use to deliver the best quality.

[0069] In this architecture, we revert back to creating a content aware architecture where the server provides the list of possible profiles to the client. The client, based on network bandwidth (and possibly power available) make a profile selection from what has been presented to it. This requires none to very minimal changes in the legacy client software, The server will now have to decide if it needs to invoke the JITT (if the top profile has not been requested) and deliver the appropriately requested profile from the client.

Variations on JITT Processing

[0070] Other variations of JITT could include delivery of a profile that is very similar to one requested by a different end client. The server determines that the profile requested by the current client matches very closely (but not exactly) by a different client and may choose to send the transcoded version of the different client. The threshold of difference in profile request can be set at the server. This choice obviates the need for an additional JITT of the segment.

[0071] It must be mentioned that all of the architectures for smart client and smart server extend in the same manner to JIT audio transcoding. Whether it is conversion between mono, stereo and 5.1 or higher channel audio formats or conversions between different audio codecs, the same principles of recreating the requested audio profile on the fly would apply.

Blazar and Edge Caches

[0072] The storage and core network bandwidth savings are even higher when the edge caches are considered. In an Adaptive Bitrate (ABR) video streaming system, a Content Delivery Network (CDN) and edge caches work together to deliver a smooth, high-quality viewing experience.

[0073] The edge caches store the different video profiles (different bitrates and resolutions) that make up the ABR ladder for a particular video. When a client device requests a video, the edge cache can quickly provide the version of the video that's most appropriate for the device's current network conditions, thus minimizing buffering and maximizing quality.

[0074] The device doesn't just request and stick with one profile but dynamically adapts to changing network conditions. When network conditions change, the client device may switch to a different profile (either higher or lower quality), which the edge cache should also have available.

[0075] This is why it is beneficial for edge caches to store all the profiles of the ABR ladder rather than just the one that the client device initially requests. By storing all profiles, the edge cache can quickly respond to changes in the client's network conditions and provide the most appropriate video profile.

[0076] Of course, this might not be the case for all systems. In some systems with severe storage limitations, the edge cache might store only the most requested profiles. But such a setup could lead to more cache misses and poorer performance if the network conditions change, and the client device needs to switch to a profile that the cache doesn't have.

[0077] Blazar JITT at the edge can greatly optimize such a system. FIG. 5 shows an ABR system with Blazar JITT used at the edge caches. In this case, the core network only has to store and forward the top profile to the edge caches for any video content. When an endpoint device requests a lesser profile, the blazar JITT can provide that on the fly, reducing the edge cache requirement to store all the profiles.

[0078] Therefore, the same storage savings as described before can be achieved at each edge point. Depending on the number of edge cache devices this can be a significant multiplying effect of edge cache storage savings. In addition, the network bandwidth from the origin to the edge has the equivalent percent savings since only the top profile needs to be pushed to the edge.

[0079] FIG. 10 shows the ABR system 1000 with a Blazar JITT implemented at the edge cache. When an endpoint device requests the top profile, then the origin provides that top profile from the DVR storage and the edge cache can cache it for future use. When an endpoint client requests a lesser profile then the edge cache redirects the request to Blazar JITT which retrieves the top profile from the Origin and then transcodes it to the desired profile the endpoint device requests.

[0080] The benefits of this JITT edge approach are multifold

• • [0081] Cache Hit Ratio will be twice or better with the same cache size. This is due to the cache only storing the top profile. (Alternatively, the edge cache storage size can be reduced by twice or more with the same cache hit ratio).
  • [0082] Network bandwidth from the edge cache back to the origin will be reduced by the effect of the edge cache hit ratio increasing. This reduces network capacity requirements on multiple components including switches, origin, and storage system.

[0083] As with the Smart Server or Smart Client architectures referred to in the previous discussion, the edge cache system can have a similar Edge Smart Server or a Smart Client to support the on-the-fly decision making.

[0084] In some cases, different clients might request similar but slightly different profiles. In this case, the cache or origin could decide to only generate 1 profile that would be used for both clients. This will reduce the load on the Blazar system and the cache/origin system as well. For example, if 2 clients request 2 bitrates that are within a small threshold different, then only the lower bitrate could be generated by Blazar and used for both.

Blazar JITT Design

[0085] Blazar JITT transcoder is a software solution that runs on standard Linux servers with GPU compute accelerator hardware. Blazar is available as a software component that can run on any cloud provider such as Google Cloud, AWS, Azure, OCI etc., or is available as an on premise solution (sold as a software license or full turnkey server appliance).

[0086] FIG. 11 shows the software stack architecture 1100 that control the underlying GPU hardware cluster. The Segment API is key enabler for managing segment based transcode processing of video streams. The underlying GPU hardware cluster resources are managed by the Resource Handler SW stack. The key innovation is to convert all forms of processing into segment-based chunks for efficient use of the underlying resources. The segments sizes can be of different lengths and can be used to support different codec choices. The resource handler receives, through its Segment Processing API, segment transformation requests. It allocates those requests to specific GPU pipelines that perform the transformation. Each GPU pipeline has an optimal number of slots to perform the transformations in an allotted amount of time. This entire infrastructure's resources are managed by the resource handler.

[0087] The Segment Processing API can also be used natively or invoked from a real-time transcoding SW stack that supports transcoding/processing applications via an external application API.

[0088] The Blazar Software stack is controlled from a simple but powerful API that can be integrated into any workflow The API can be customized for certain use cases or workflows if needed. The hardware can also be selected for optimum cost/performance tradeoffs depending on the system workload.

[0089] In some use cases, it makes sense to have Blazar JITT nodes be dynamically spun up or down depending on workloads (peak viewing times for example). This can most easily be done in cloud deployments, but Blazar nodes can also be used as VOD transcoding resources in an on premises deployment when not needed for Blazar functions.

[0090] For JITT operation, Blazar takes segmented default profile media files as input (usually fMP4 or MPEG-TS segments), and transcodes them fast enough for the end device playback to not be disturbed any more than if that end device was retrieving content from a storage server. A typical use case is 2 second media segments transcoded 10 times faster than real time which achieves this goal.

[0091] In some use cases more than 10× faster than real-time or greater transcode speed are needed and in those cases the Blazar hardware architecture must be scaled to support that. This can be done statically with the hardware deployment up front, or dynamically by combining Blazar JITT nodes together.