Ensemble-Based Multimedia Asset Recommendation System

[0001] This application claims benefit of Provisional Application No. 62/029,713, filed Jul. 28, 2014, the entire contents of which is incorporated by reference as if fully set forth herein under 35 U.S.C. §120.

[0002] The present disclosure generally relates to the automatic selection of multimedia content.

[0003] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0004] Content suggestion systems traditionally select content based on a relationship to a user's preferences or a relationship to parameters of a search query. Recommendation engines use keyword or tags in order to select content that match a particular content selected by a user. These systems have a singular function that is focused on presenting a set of content that is similar in type, e.g., video, to the content being accessed by the user. Further, these systems typically present the suggested content to the user in a list format. Content providers lack a mechanism for getting assets from their mixed-multimedia asset libraries in front of users and to present the mixed-multimedia assets to users in meaningful and engaging ways.

[0020] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

[0021] Embodiments are described herein according to the following outline:

• • [0022] 1.0. General Overview
  • [0023] 2.0. Methodology
  • [0024] 2.1. Ensemble Strength
  • [0025] 2.2. Ensemble Duplication
  • [0026] 2.3. Ensemble Depth
  • [0027] 2.4. Construction of Ensembles
  • [0028] 2.4.1. Similarity Matrix Preparation for Ensemble Construction
  • [0029] 2.4.2. Weak Ensemble Construction
  • [0030] 2.4.3. Strong Ensemble Construction
  • [0031] 2.4.4. Semi-strong Ensemble Construction
  • [0032] 2.4.5. Improved Semi-strong Ensemble Construction
  • [0033] 2.5 Layout Comparison
  • [0034] 3.0 Example Embodiments
  • [0035] 4.0 Implementation Mechanisms—Hardware Overview
  • [0036] 5.0. Extensions and Alternatives

[0037] This overview presents a basic description of some aspects of a possible embodiment of the present invention. It should be noted that this overview is not an extensive or exhaustive summary of aspects of the possible embodiment. Moreover, it should be noted that this overview is not intended to be understood as identifying any particularly significant aspects or elements of the possible embodiment, nor as delineating any scope of the possible embodiment in particular, nor the invention in general. This overview merely presents some concepts that relate to the example possible embodiment in a condensed and simplified format, and should be understood as merely a conceptual prelude to a more detailed description of example possible embodiments that follows below.

[0038] Disclosed herein are systems and methods for pairing assets for online presentation where the plurality of electronic assets are available in a wide range of formats. The processes associated with selection and ordering of these assets to be made available for consumption in an online environment is increasingly becoming more data-driven and requiring that electronically-stored assets of different formats be presented as an ensemble or collection. A non-exhaustive list of asset types include text (TXT, HTML, etc.), images (GIF, JPG, TIFF, BMP, PNG, etc.), videos (MP4, WMV, FLV, etc.), geographic (SHP, KML, etc.), coded (JavaScript, PHP, etc.), and sound (MP3, WMV, QT, etc.) files. The asset itself is a distinct electronic file that may be presented to a user for consumption either by itself or simultaneously with other electronic files.

[0039] For each asset type, both the storage/presentation requirements and resulting consumption of the assets may differ and require cognitive processing by using one or more of the human user's five senses: sound, sight, touch, taste, and smell. An engagement is the timeframe during which a user is consuming the electronic asset using one or more of the five human senses. Presentation of information to the human user may appeal to one or more of these senses and there exists a business purpose for pairing files of the various types to increase the user's satisfaction with their consumption and thereby allowing the asset portfolio manager (the individual responsible for deciding upon the assets to display) to yields some desirable benefit from a satisfactory consumption of the assets. The concept of pairing assets is not restricted to two distinct assets—rather, a collection of assets (containing at least two assets) is selected for presentation during a single engagement by a user. An asset may be categorized as stationary or non-stationary. A stationary asset will not change over time—examples include an image, a non-interactive electronic map, or a sound file with a single tone (“ding”, “beep”, etc.). Non-stationary assets are electronic files that vary over time such as a video, an animated image, a song, or an interactive map. The changes over time that cause an electronic asset to be considered non-stationary can be a) executed automatically when an appropriate application exists that allows the temporal variations to be presented; b) triggered according to the internal clock of the device—where available; or c) initiated interactively by the electronic assets' consumer during an engagement.

[0040] Management of an asset collection may be conducted by the owner of the collection or another party given authorization to generate recommendations and pursue revenue acquisition on behalf of the owner. This entity may also be referred to as the “asset portfolio manager”. Referring to FIG. 1, the asset portfolio manager 101 may be software and/or hardware resident in a server, client computer, distributed cloud computing system, virtual machine, etc. An embodiment performs a methodology for pairing multimedia assets in a single presentation via an ensemble of assets with the intent of enriching and lengthening the engagement experience to allow a higher level of value to be achieved by the content provider Value may be based on an objective function determined by the asset portfolio manager and may include, but is not limited to: a) maximization of revenue b) user satisfaction c) recurrence of visitors. Allowing the user to more deeply explore related assets in a variety of different formats, in perhaps more detail, and/or by appealing to various sensory modes is assumed to lead to a preferred engagement for the user. Over time, the rotation of different number, placement, and types of assets can be used to determine which presentations yield higher revenues. The system therefore not only enriches the user experience through a co-presentation of a collection of multi-sensory and multimedia assets, but provides a mechanism for testing various layouts to yield a value outcome that is perhaps more favorable to the owner/managers of the displayable content.

[0041] During an online engagement where a recommendation system algorithm 102 considers candidate assets stored in the asset library 103 and available for presentation and consumption by a user, there often arises the desire to use the available device presentation capabilities to present more than one asset at the same time in the shared display space. The asset portfolio manager may be interested in simultaneous co-presentation of a set of assets related to the online asset 104 to enhance the viewing experience and provide a means of presenting the content in a way that appeals to the multi-sensory capabilities of human consumers. The asset portfolio manager—which has access to a library of multimedia assets 103 available to be displayed (either remotely over a network, such as an intranet, internet, etc., or locally over a local network of connection)—automatically selects which assets are most appropriate for co-display given their preferences for content delivery 105. The selected assets may be sent by the asset portfolio manager to the receiving device, or a display screen, web page, etc., may be created that includes the selected assets and the assets that are to be displayed online during the online engagement 105. Alternatively, the asset portfolio manager may display a presentation screen template to an administrator and suggest the placement of each of the selected assets within the presentation screen template. The administrator may indicate the placement of each selected asset and instruct the asset portfolio manager to send the completed presentation screen to a receiving device. In an embodiment, the selected assets may be homogeneous in nature, e.g., video only, etc. In another embodiment, the selected assets may be heterogeneous and may comprise any combination of: text, video, hyperlinks, images, audio, etc. To meet this need, the system performs several features:

• • [0042] 1. Definition of an anchor asset—one that may be presented with certainty during the online engagement
  • [0043] 2. Selection of assets for co-presentation that can be simultaneously delivered in the same user engagement session
  • [0044] 3. Consideration of the user's human sense(s) to which the asset portfolio manager wishes to be present in the online environment
  • [0045] 4. Selection of assets for presentation that remain fixed or may vary during the engagement (such as a fixed image or a video) according to the asset manager's choice
  • [0046] 5. Enforcement of a rule set that ensures the probability of the selection of assets to vary based on the preferences of the asset portfolio manager and physical constraints on the display device
  • [0047] 6. An automated system that, given the constraints and preferences set by the asset portfolio manager, selects an ensemble of assets for co-presentation that share a similar theme
  • [0048] 7. A mechanism for down-weighting an asset so that it has a reduced chance of being selected for display
  • [0049] 8. A process by which various presentation alternatives (in terms of the number and type of assets, the similarity among the assets, and the physical layout of the presentation of assets in an online environment) can be explored and measured to determine the presentation format that yields the most value to the user.

[0050] One application of an embodiment can be found in the online news publishing industry. For any given asset (e.g., textual, such as a news article, etc.) that can be presented on a web page, there may arise the desire of the asset portfolio manager 101 to also co-present relevant images, videos, sound clips, and hyperlinks to related articles to provide additional information in a variety of format with the intent of improving the satisfaction associated with the consumer's engagement with the content. Referring to FIG. 2, an example of a presentation of multimedia assets 200 is shown where a news article 201 (a text asset) is shown in the center of a display and accompanied by a variety of assets 202 (audio, image, video, advertisements, links to other articles, etc.) in the ensemble. A successful presentation would deliver appropriate assets that consider the user's various sensory modes to which the asset portfolio manager 101 wishes to appeal (e.g., given predetermined limits on content and the physical constraints on the display device, etc.) while ensuring the ensemble of presented assets is, as a group, relevant to the article that has been selected to be displayed with certainty (the anchor asset).

[0051] An embodiment displays or creates a display of video asset recommendations to users engaging a variety of online websites using mobile and non-mobile electronic data providers. The system:

• • [0052] a. Creates recommendations to a user based on an asset (e.g., video, audio, text, etc.) library
  • [0053] b. Ensures that content in an asset (e.g., video, audio, text, etc.) portfolio is monetized
  • [0054] c. Delivers analytics to learn the most valuable content in a client's video library, for example.
  • [0055] d. By operating as a supply-side platform (SSP) for online video, the system utilizes data to increase engagement and provide solutions to assure that advertisements are delivered and advertisement rates are optimized.

[0056] Inputs: The inputs to the system may be:

[0057] L_t is the physical arrangement of assets in presentation t limited by the display parameters of the presentation device, such as the physical characteristics of the presentation device and the embedded software that enables certain asset types to be consumed (for example, presentation of a video file in the layout may require that there exists a capability for the video to be played, if desired by the user during engagement). FIG. 2, shown earlier, is an example of a possible physical arrangement of assets.

[0058] A is the set of portfolio assets available for selection and subsequent display. Individual asset a_i is the i^th asset among n assets: A={a₁, a₂, . . . , a_n}

[0059] a_i is the anchor asset that can be presented with certainty and that is capable of being presented given the physical characteristics and the other limitations of L_t.

[0060] A⁽⁻ⁱ⁾ is the remainder of the portfolio assets that may be selected for co-presentation along with the anchor asset, a_i: A=A⁽⁻¹⁾∪a_i, A⁽⁻¹⁾∩a_i=ø

[0061] X is a set of k features describing each of the n assets.

X={x₁,x₂, . . . ,x_n} where x_i={x_i,1,x_i,1, . . . ,x_i,k}.

[0062] τ_i,c is a subset of a feature of asset a_i referred to as a collection descriptor which is a label or index indicating a theme under which the asset is grouped (e.g., sports, entertainment, justice, etc.). An asset a_i may belong to c≧1 asset groups so that τ_i={τ_i,1, τ_i,2, . . . , τ_i,c}. Additionally, more refined theme division may exist describing a physical locale and more detailed membership requirements. For simplicity, each distinct collection label, sub-collection criteria and locale can be uniquely identified using a collectionID as shown in Table 1. Note that the entry corresponding to collectionID=3 has a locale of “Worldwide” which is equivalent to describing an asset whose relevance is without geographic limits.

[0063] ssfID_i is a single feature of asset a_i indicating three characteristics of the asset; the human sensory mode to which it appeals, whether the asset is stationary or non-stationary, and a description of the asset type. Table 2 gives an example of some instances of various ssfIDs where the plurality of ssfIDs has a size of P.

TABLE 1
CollectionID Example Encompassing Distinct Locale/Theme
Combinations
collectionID	collectionLabel	collectionDetail	Locale
1	Sports	Football	United States
2	Sports	Soccer	Europe
3	Politics	Congress	United States
4	Entertainment	Celebrities	Worldwide
5	Entertainment	Movie Reviews	United States
6	Justice	Crime	Illinois USA
7	Justice	Legislation	Illinois USA
. . .	. . .	. . .	. . .
C	. . .	. . .	. . .

TABLE 2
ssfID Examples Describing Distinct Sense/Stationarity/Description
Triplets
ssfID	Sense	is Stationary	fileType
1	Slight	Yes	Image
2	Slight	No	Map
3	Slight	No	Animated GIF
4	Slight	No	Video
. . .	Slight	. . .	. . .
	Sound	Yes	Tone
	Sound	No	Music
	Sound	No	Audio Clip
	Sound	. . .	. . .
P	. . .	. . .	. . .

[0064] κ_i,c is a text-based subset of a feature of asset a_i referred to as a keyword list which is set of labels describing asset a_i. Asset ai a_i may have w≧0 keywords so that κ_i={κ_i,1, κ_i,2, . . . , κ_i,w}. Keywords may be in any language and contain special characters and symbols.

[0065] FileTypeID_i is the type of file (based on the file extension associated with asset paired with a ssfID). Examples of various file types that may be associated with each distinct ssfID are shown as a many-to-one relationship in Table 3. Included in this table is a value for each row called additiveValue (always greater than or equal to 0) that can be used to induce a preference for certain file types sharing the same ssfID.

[0066] Larger values associated with additiveValue increase the chances that, among all files selected for co-presentation with the anchor asset a_i, the file having the specified type will be selected.

TABLE 3
File Types Associated with Various ssfIDs
	FileTypeID	ssfID	fileExtension	additiveValue
	1	1	GIF	3
	2	1	TIFF	2
	3	1	JPG	1
	4	1	JPEG	1
	5	2	KML	1
	6	3	GIF	1
	7	4	MP4	1
	. . .	. . .	. . .	. . .
	f

[0067] x_i as described earlier contains k features describing asset a_i; x_i={x_i,1, x_i,1, . . . , x_i,k} representing the asset's;

• • [0068] 1) collectionID list, τ_i
  • [0069] 2) distinct sense/stationarity/description triplet denoted by ssfID_i
  • [0070] 3) keyword list, κ_i
  • [0071] 4) All of the remaining k−3 features describing asset a_i.

[0072] An example of the n×k table use to store the various sets of k features for each of the n assets in A is shown in Table 4. In addition to the k features, the stored table may include an assetID i=1, . . . , n along with a FileTypeID.

TABLE 4
k-dimensional Feature Set for n Assets
	Features
Asset	FileTypeID	collectionID(s)	ssfID	Keywords	field_4	field_5	. . .	field_k
1	4	1.4	1	New England Patricts,
				Tom Brady, AFC,
				Touchdown
2	4	1	1	Jacksonville Jaguars,
				Jordan Todman,
				Kickoffreturn
3	1	4	3	Humphrey Bogart
				Caseblance, Oscar
				Award
. . .	. . .	. . .
n

[0073] s_i,j is a computed similarity between asset a_i and asset a_j based on pairwise difference between the features of the two assets, not including the collectionID(s), ssfID, or keyword(s). The computation of this pairwise similarity may represent a more complex metric that is employed by the underlying recommendation engine at the disposal of the asset portfolio manager. In this example, a simple Minkowski metric (a computation of pairwise differences between asset a_i and itself represented by s_i,i typically yields a value of 1 (after standardization) and the matrix S is symmetric and non-negative for all i,j) is discussed that may be applied after all of the features have been standardized to a [0,1] scale.

$s_{i,j}=1-{\left\{\sum _{m=1}^k{w_m\left(x_{i,m}-x_{j,m}\right)}^{\gamma }\right\}}^{1/\gamma }0\le w_m\le 1,\sum _{m=1}^kw_m,\gamma >0$

The n×n matrix of pairwise similarities may be computed (independent of the ssfID, FileTypeID, and collectionIDs) in order for the co-presentation process to be employed. FIG. 3 illustrates the identification of similarities between pairs of assets.

[0074] The computed pairwise similarities may then be represented as:

$S=\left[\begin{array}{ccccc}1& s_{1,2}& \dots & s_{1,n-1}& s_{1,n}\\ s_{2,1}& 1& \dots & s_{2,n-1}& s_{2,n}\\ \dots & \dots & \dots & \dots & \dots \\ s_{n-1,1}& s_{n-1,2}& \dots & 1& s_{n-1,n}\\ s_{n,1}& s_{n,2}& \dots & s_{n,n-1}& 1\end{array}\right].$

[0075] To illustrate how the similarities are computed, FIGS. 4-9 provide an example of three assets {a_i, a_j, a_k} and the observed k=3 features values for each asset. FIG. 4 shows the values of the assets. FIGS. 5-7 show the absolute values of pairwise differences among the three assets {a_i, a_j, a_k} in the example data for features 1, 2, and 3, respectively. Using an exponent γ=1 and weights of w={w₁, w₂, w₃}={0.40, 0.30, 0.10} for features 1, 2, and 3, respectively, then FIG. 8 shows the composite weighted values of the three features for each pair of assets. From these values, the composite similarities for the example set of assets is shown in FIG. 9.

[0076] α_i,j a multiplier selected by the asset portfolio manager indicating if certain pairings of assets should have a reduced chance of being co-presented where 0≦α_i,j<1 and, for any pair i,j not defined, it is assumed that α_i,j=1. Table 5 provides an example of some pairs of assets for which co-presentation is discouraged and for any value for which α_i,j=0, co-presentation may not be allowed.

TABLE 5
Down-weighting of Asset Pairs
i	j	αi,j
1	2	0.00
1	4	0.10
3	1	0.12
4	2	0.58
. . .	. . .	. . .
q	. . .	. . .

[0077] The pairwise similarity matrix S along with Tables 1-5 represent the schema that can be used to build an ensemble of assets for presentation—given a_i (the anchor asset)—once the asset portfolio manager has designed the physical arrangement of asset for presentation, L_t.

[0078] With all of the inputs in place, the item portfolio manager may design the layout L_t and the underlying computations associated with an embodiment can be executed to automatically populate the presentation. FIG. 10 illustrates a sample layout, L_o, which considers the physical layout of the presentation space available given the physical constraints of the device used to present the selected assets. Note that the layout region, B, can be subdivided in d=1, . . . , D sections, each having an anchor asset, a_i^(d). In each of the D sections, the item portfolio manager may then identify the particular ssfID and position where they would like to place each asset to be co-presented. In each slot r=1, . . . , R^(d)—the position where an asset is to be placed in a section of the presentation—an ssfID is defined by the asset portfolio manager.

[0079] Within a section d, the entire set of co-presented assets in the R_d can be called an ensemble,

E_d={a_i^(d),a_j^(d), . . . ,a_R^(d)}.

[0080] In an embodiment, there may be four levels of ensemble strength:

• • [0081] 1) Weak Ensemble: within a section d, the selection for assets to be included for co-presentation are based on similarities among every candidate asset in A⁽⁻ⁱ⁾ and the anchor asset, a_i. At most, the number of pairwise comparisons of eligible assets that needs to be completed is n−1 and so this a highly computationally efficient approach to building an ensemble but may lead to solutions where the elements selected from A⁽⁻ⁱ⁾ are not very similar to each other.
  • [0082] 2) Strong Ensemble: within a section d, the selection for assets to be included for co-presentation are based not only on similarities among every candidate asset in A⁽⁻ⁱ⁾ and the anchor asset, a_i, but also every other candidate asset in A⁽⁻ⁱ⁾. If there are n_r^(d) eligible candidate assets to fill slot r in section d, then the maximum number of possible solutions to the problem of selecting the r=1, . . . , R_d assets (disallowing duplication of the same asset within a section) is:

$N_d=\frac{\left(n-1\right)!}{\left(n-R_d-1\right)!}.$

• •  Note that, as n grows large, N_d→∞, making the strong ensemble method computationally intractable in many cases.
  • [0083] 3) Semi-strong Ensemble: Since the semi-strong ensemble is computationally costly and the weak ensemble may result in a solution where the assets from A⁽⁻ⁱ⁾ that are chosen for co-presentation with a_i are dissimilar, a compromise solution is offered. Construction of a semi-strong ensemble is done iteratively over each of the r=1, . . . , R_d slots in a section d. In the initial asset selection the first slot (r=1) is filled with an asset, a_j, that is selected based on its similarity with a_i. In the next step, slot r=2 is filled with asset a_k based on its similarity with both a_i and a_j. This process repeats over the R_d slots, each time comparing candidate assets to those that have been selected for inclusion in the section in previous iterations. Essentially, the ensemble is “backward-looking” and adds an asset to a slot based on the similarities with all assets in slots that have already been filled—but does not consider the similarities of assets that may be added in the filling of future slots. For small values of R_d and n, it is possible that the strong, semi-strong, and weak ensembles may be identical, though such an outcome is not expected. However, the upper bound on the total number of required comparisons for a given section d is no greater than

$\frac{{\mathrm{nR}}_d\left(R_d+1\right)}{2},$

• •  a lesser computational burden (for larger n) than required build a strong ensemble.
  • [0084] 4) Improved Semi-strong Ensemble: After a semi-strong ensemble is built, a second round of iterations can be applied that seek to find improvements to the objective function. This is, of course, more computationally costly than the semi-strong ensemble method, but the additional cost is proportional to (n′−2D) additional computations where n′ is the total number of slots to be filled in the display L_t and D is the total number of sections. The option to pursue an improved ensemble using this construction method can be considered by the asset portfolio manager based on a comparison of the tradeoff between a desire to seek a solution that is: a) more consistent with a final layout that may be achieved using the strong ensemble method; and b) does not add an unacceptable computational burden.

[0085] Within an ensemble, a requirement prohibiting or allowing duplicate assets within or across a section can also be defined. Where there always exists the requirement that the candidate assets for inclusion in the presentation of a section-level set of assets not duplicate the anchor asset a_i—that is A=A⁽⁻ⁱ⁾∪a_i, A⁽⁻ⁱ⁾∩a_i=ø—duplication may be allowed among the other non-anchor assets in the presentation of section d.

• • [0086] 1. When local duplication is prohibited, no two assets within a section d may be the same.
  • [0087] 2. When global duplication is prohibited, no two assets within a section d may be the same nor may any two assets across the d=1, . . . , D sections be allowed in the layout, L_t.

[0088] Another consideration when building an ensemble is the degree to which the similarities among the assets in the d=1, . . . , D sections should be. The enforcement of similarity across sections in a layout L_t as the depth—depth can either be local or global. If the ensemble depth is local, then there is no requirement that the similarities across assets presented in different sections be considered in the selection of assets of L_t. If the depth is global, pairwise comparisons of assets are computed both within a section d and across all sections d=1, . . . , D. Notationally, the similarities among assets is s_i,j, but when further transformations are applied to adjust for commonality among keywords, file types, and enforce ssfID and down-weighting requirements, then similarities in section d among two its members, a^(d)_i and a^(d)_j, can be denoted by f^(d)_i,j and the vector of assets selected for presentation in section d is:

a^(d)={a_i^(d),a_j^(d), . . . ,a_Rd^(d)}.

[0089] The application of depth may be accomplished by taking an overall representative of similarities of assets with a section d and the overall similarity of assets across the various sections d=1, . . . , D. Let;

$w^{\left(d\right)}=\{\begin{array}{c}\underset{j}{\min }f_{i,j}^{\left(d\right)}\mathrm{if}\mathrm{weak}\mathrm{ensembles}\mathrm{are}\mathrm{desired}j=1\dots R_d,i\ne j\\ \underset{j,k}{\min }f_{j,k}^{\left(d\right)}\mathrm{otherwise}j,k=1\dots R_d,j\ne k\end{array}b^{\left(d\right)}=\{\begin{array}{c}\underset{j}{\min }\left(f_{i,j}^{\left(d\right)},f_{i,j}^{\left(g\right)}\right)\mathrm{if}\mathrm{weak}\mathrm{ensembles}\mathrm{are}\mathrm{desired}j=1\dots R_d,i\ne j,g\ne d\\ \underset{j,k}{\min }f_{j,k}^{\left(d\right)},f_{j,k}^{\left(g\right)}\mathrm{otherwise}j,k=1\dots R_d,j\ne k,g\ne d\end{array}$

[0090] and let the measure of depth across all d=1, . . . , D sections in L_t be

m(L_t)=w^(d)+λ_tb^(d) where 0≦λ_t≦1.  {Equation 1}

[0091] Selection of assets for presentation within a layout, L_t, is then equivalent to maximizing m(L_t);

${\lambda }_t=\{\begin{array}{cc}0& \mathrm{when}\mathrm{local}\mathrm{depth}\mathrm{is}\mathrm{desired}\\ 1& \mathrm{when}\mathrm{global}\mathrm{depth}\mathrm{is}\mathrm{desired}\end{array}.$

[0092] In this section, the mechanics of the construction of weak, strong, and semi-strong ensembles is presented along with the steps to apply local or global duplication prohibition and local/global depth restrictions. Among all three methods of constructing ensembles, there are common steps to prepare the similarity matrix based on the availability of theme and keyword features possessed by each asset along with the down-weighting of certain asset pairs made to discourage co-presentation in a display.

[0093] Step 1a: each element in the diagonal of matrix S is set equal 0 to yield a matrix of similarities;

$S=\left[\begin{array}{ccccc}0& s_{1,2}& \dots & s_{1,n-1}& s_{1,n}\\ s_{2,1}& 0& \dots & s_{2,n-1}& s_{2,n}\\ \dots & \dots & \dots & \dots & \dots \\ s_{n-1,1}& s_{n-1,2}& \dots & 0& s_{n-1,n}\\ s_{n,1}& s_{n,2}& \dots & s_{n,n-1}& 0\end{array}\right]$

[0094] This is equivalent to defining A⁽⁻ⁱ⁾ since asset a_i has been removed from the set of candidate assets for co-display (avoiding duplication of the anchor asset) since one requirement for selection will be that the similarity value, s_i,j, of a selected item a_j to be paired with the anchor asset, a_j, is that s_i,j>0.

[0095] Step 1b: Identify entries in Table 5 to allow adjustment for the chance of being selected (lower values of s_i,j reduce the chances of selection for presentation). Undefined pairs for any i,j receive a value of α_i,j=1.

[0096] Step 1c: For each pair of assets {a_i, a_j} of A (i,j=1, . . . , n) a multiplier, δ_ij, is computed;

${\delta }_{i,j}=\frac{w_i\bigcap w_j}{\max \left(w_i,w_j\right)}0\le {\delta }_{i,j}\le 1$

[0097] where w_i is the number of keywords associated with a_i, w_j is the number of keywords associated with a_z, and w_i ∩w_j is the number of keywords the two assets have in common.

[0098] Step 1d: For each pair of assets {a_i, a_j} of A (i,j=1, . . . , n) a multiplier, ν_j,k, is computed;

$v_{i,j}=\{\begin{array}{cc}1& \mathrm{if}\mathrm{asset}i\mathrm{shares}a\mathrm{collectionID}\mathrm{with}\mathrm{asset}j\\ {\epsilon }_v& \mathrm{otherwise}\end{array}{\epsilon }_v\ge 0.$

[0099] Step 2: Update the values of S by multiplying against the values computed in Step 1a-1c to adjust the similarities based on the relative concordance of keywords (lower concordance reduces the chances of selection for presentation), consistency of themes, and:

s′_i,j=s_i,j×δ_i,j×ν_i,j×α_i,j

[0100] The similarity matrix prepared for ensemble construction in then an adjusted n×n symmetric matrix, S′, representing all pair of assets {a_i, a_j} of A (i,j=1, . . . , n).

$S^{\prime }=\left[\begin{array}{ccccc}0& s_{1,2}^{\prime }& \dots & s_{1,n-1}^{\prime }& s_{1,n}^{\prime }\\ s_{2,1}^{\prime }& 0& \dots & s_{2,n-1}^{\prime }& s_{2,n}^{\prime }\\ \dots & \dots & \dots & \dots & \dots \\ s_{n-1,1}^{\prime }& s_{n-1,2}^{\prime }& \dots & 0& s_{n-1,n}^{\prime }\\ s_{n,1}^{\prime }& s_{n,2}^{\prime }& \dots & s_{n,n-1}^{\prime }& 0\end{array}\right]$

[0101] FIG. 11 depicts an example of how the adjusted similarity matrix is computed, specifically showing the process for two pairs of example assets {a₁, a₄} and {a₁,a₇}.

[0102] Step 3: For each element section d, slot r, and element s′_i,j of S′, three sub-steps are executed:

[0103] Step 3.a: a multiplier is computed, φ_j^(d,r), as

${\phi }_j^{\left(d,r\right)}=\{\begin{array}{cc}1& \mathrm{if}a_j\mathrm{has}\mathrm{the}\mathrm{ssfID}\mathrm{required}\mathrm{for}\mathrm{slot}r,\mathrm{section}d\\ 0& \mathrm{otherwise}\end{array}$

[0104] This step forces the asset selection process to prefer those assets that are consistent with the sense/stationarity/fileType specified by the asset portfolio manager. Setting a value of ε_φ>0 for non-matched ssfID values is a possible less-strict option that can be employed to allow a selection not exactly specified by the asset portfolio manager.

[0105] Step 3.b: the additive Value, ρ_j, is extracted from Table 3 (the various file types associated with each ssfID) for each asset S′ based on a lookup of the fileTypeID found in Table 4 (the features of each asset including the file type). the purpose of this sub-step is to adjust the similarity values to prefer, among all assets that remain eligible for selection, the asset having the desired file type. FIG. 12 shows an example of how the ssfID and fileTypeIDs required for a slot r and section d are computed—matches are shown for eligible candidates for section 1

[0106] Step 3.c: Compute a final set of similarity pair for each section d each slot r as:

f_i,j^(d,r)=s′_i,j×φ_i,j^(d,r)×(1+ρ_j^(d,r))

[0107] Step 4: Among all similarity matrices, f_j,k^(d,r) where d=1, . . . D and r=1, . . . , R_d, select the values such that f_j,k^(d,r)>0 as initial candidates for that slot r and section d. To be eligible to fill a slot in section d, an asset a_j^(d), can have a non-zero similarity value (s′_i,j>0) when paired with the anchor asset a_i^(d) in that section. Note that it is disallowed for another asset to populate the slot(s) reserved for the anchor asset in each section.

[0108] The resulting candidate list may then be used to create a table F_t containing a list of possible assets that may be used to populate the display L_t for all sections d=1, . . . , D and their respective slots r=1, . . . , R_d.

[0109] An example table F_t from preparation of the similarities is shown in Table 6. In the example, section d=1 has R_d=2 slots; one is reserved for the anchor asset (a₁ and a₂) and eligible anchor assets are based on the example in FIG. 12. while the second slot may be filled with one of two assets {a₄ or a₇}. The implication is that the similarities of the candidates in the second slot of section d=1 have values that can be used to select an appropriate asset to pair with the anchor asset for section d=1 and slot r=2 given that the anchor asset is selected.

[0110] Following through with the examples in FIGS. 9-11 and Table 6, the value of pairwise

TABLE 6
Example Candidate Assets after Preparation for Ensemble Construction
			ssfID
		Eligible Asset	Multiplier	additive
Section (d)	slot(r)	(j)	ρ(d, r)	Value ρ(d, r)
1	1	1	1	0.10
1	1	2	1	0.10
1	2	4	1	0.05
1	2	7	1	0.01
2	1	5	1	0.10
2	1	6	1	0.10
2	2	5	1	0.15
2	2	6	1	0.08
2	3	5	1	0.10
2	3	6	1	0.01
2	4	5	1	0.10
2	4	6	1	0.01

similarities, f_i,j^(d,r)=s′_i,j×φ_i,j^(d,r)×(1+ρ_j^(d,r)), used to fill position r=2 assuming the a₁ is selected as the anchor asset for section d=1 are (refer to FIG. 10 for the adjusted similarities):

f_1,4^(1,2)=s′_1,4×f^(1,2)_1,4×ρ₄^(1,2)=0×1×(1+0.05)=0 and

f_1,7^(1,2)=s′_1,7×f^(1,2)_1,7×ρ₇^(1,2)=s1,7×0.2×1×(1+0.01)=s_1,7×0.202

[0111] In an embodiment, after L_t has been defined, the automated process for selecting assets for co-presentation of a weak ensemble works as follows for each of the D sections:

[0112] Step 1: Given the desired choice of duplication restrictions (local or global), identify the set of all possible and allowable arrangements, a^(d), of assets from A that can be used to populate the display, L_t under the restriction that the anchor assets and their positioning in each section, a^(d)_i, is required.

[0113] Step 2. Select a*, the unique arrangement of a that maximizes

• • [0114] m(L_t)=w^(d) as defined earlier by Equation 1

[0115] which is equivalent to, for each slot, choosing the eligible candidate asset a_j for which the similarity between it and the section's anchor asset a_i^(d) is maximized (Note that in the weak ensemble method, a global depth restriction may not be sensible). The arrangement of assets selected as a* are then used to populate the display and can guarantee that every asset selected for presentation in section d has a non-zero similarity with the anchor asset in the section (local depth). In the example discussed above, earlier, if a₁ is selected as the anchor asset and another asset meeting the ssfID and fileTypeID restrictions are applied to slot r=2 and section d=1, then only assets a₄ or a₇ can be selected to fill the position. Since a₇ has a larger value (0.202×s_1,7 assuming that s_1,7>0) for the final similarity computation f^(1,2)_1,j than asset a₄ when paired with asset a=_1l, then it would be chosen to fill the d=1, r=2 position in the display.

[0116] In an embodiment, after L_t has been defined, the automated process for selecting assets for co-presentation of a strong ensemble works as follows for each of the d=1, . . . , D sections:

[0117] Step 1: Given the desired choice of duplication restrictions (local or global), identify the set of all possible and allowable arrangements, a^(d), of assets from A that can be used to populate the display, L_t under the restriction that the anchor assets and their positioning in each section, a^(d)_i, is required.

[0118] Step 2: Some possible arrangements of a^(d) are removed from eligibility as an acceptable solution for populating layout L_t. Any arrangement containing a pair of assets such that the similarity f^(d)_j,k=0 (j,k=1, . . . , n) within a section d is removed. This is equivalent to enforcing the restriction that an eligible asset need to not only have a non-zero similarity with the anchor asset, but also to every other asset that may be co-displayed in the same section (Since this step may result in an empty set of solutions, non-zero choices for ε_δ, ε_φ and α_i,j can be set to small-but-positive values to ensure that some possible arrangements are available, albeit sub-optimal). For example, if a_i is selected as the anchor asset for a display with r=3 assets, then the other two assets, a_j and a_k, selected to fill the other positions must not only be similar to a_i, but must be similar to each other—that is, consideration must be given to the similarities among {a_i,a_j}, {a_i,a_k} and {a_j,a_k}.

[0119] Step 3: If a global depth restriction is desired, then some possible arrangements of a^(d) are removed from eligibility to be an acceptable solution for populating layout L_t. Any arrangement containing a pair of assets such that the similarity f^(d)_j,k=0 (j,k=1, . . . , n) is removed. This is equivalent to enforcing the restriction that an eligible asset need to not only have a non-zero similarity with the anchor asset, but also to every other asset that will be co-displayed across all sections d=1, . . . D.

[0120] Step 4: For each, select a*, the unique arrangement of a that maximizes m(L_t)=w^(d)+λ_t b^(d) given the desired depth (via appropriate selection of λ).

[0121] The arrangement of assets selected as a* may then be used to populate the display and can guarantee that every pair of assets within a section d has a non-zero similarity with every other asset selected for co-display in the section (local depth). If a global depth restriction is applied, then there will be an additional guarantee that every pair of assets within the display L_t has a non-zero similarity with every other asset selected for co-display in the display.

[0122] In an embodiment, after L_o has been defined, the automated process for selecting assets for co-presentation of a semi-strong ensemble can be executed. The semi-strong approach is performed iteratively to strike a compromise between the weak and strong ensemble approaches in terms of computational efficiency. The key difference in this approach relative to the other ensemble-building steps is that the final arrangement of assets, a*, is not necessarily the optimal one (that would be selected using all possible arrangements of eligible assets using the strong ensemble approach) but rather seeks to iteratively find the best asset for display given all assets chosen in earlier iterations.

[0123] Step 1: Some possible elements of a^(d) are removed from eligibility as an acceptable solution for populating layout L_t. Any element of F_t containing a pair of assets such that the similarity f^(d)_j,k=0 (j,k=1, . . . , n) within a section d is removed. This is equivalent to enforcing the restriction that an eligible asset need to not only have a non-zero similarity with the anchor asset, but also to every other asset that will be co-displayed in the same section (Since this step may result in an empty set of solutions, non-zero choices for ε_δ, ε_φ and α_i,j can be set to small-but-positive values to ensure that some possible arrangements are available, albeit sub-optimal).

[0124] Step 2: If a global depth restriction is desired, then some elements of a^(d) are removed from eligibility to be an acceptable solution for populating layout L_t. Any element of F_t containing a pair of assets such that the similarity f^(d)_j,k=0 (j,k=1, . . . , n) is removed. This is equivalent to enforcing the restriction that an eligible asset need to not only have a non-zero similarity with the anchor asset, but also to every other asset that will be co-displayed across all sections d=1 . . . , D.

[0125] Step 3: Among remaining elements of F_t that remain, select a*, the unique arrangement of a that maximizes the objective function m(L_t) is computed. In sequential order, iterate over each section d=1, . . . , D and each slot within a section r=1, . . . , R_d—skipping over each anchor asset a_i^(d) since its slot choice is predetermined and set by L_t as follows:

[0126] Step 3a: To yield an ordered vector of similarities representing the assets that may be placed in the current (the r^th) position of section d, select the asset a_j^(d,r) eligible to be placed in that position (based on F_t) such that the similarity between that asset and all other assets previously identified to fill slots in the iterative process is non-zero. Additionally, among the assets with non-zero similarities and with local depth, compute

$w^{\left(d\right)}=\underset{j,k}{\min }f_{j,k}^{\left(d\right)}j,k=1\dots \left(r-1\right),j\ne k$

[0127] and, for the a^(d,r) element of a*, choose the value which maximizes w^(d) for which duplicates have been omitted according to the local or global choice made for duplicate values.

[0128] Step 3b: If global depth is required, also compute:

b^(d)=min{f_j,k^(d),f_j,k^(g)} j,k=1 . . . (r−1)∪{a_i⁽¹⁾,a_i⁽²⁾, . . . ,a_i^(D)},j≠k,g≦d

[0129] and select the asset a_j to fill slot r in section d with the asset that maximizes

m(L_t)=w^(d)+λ_tb^(d) where 0≦λ_t≦1

[0130] during the current iteration and taking into account all other assets that have been previously selected to join a* in previous iterations.

[0131] The arrangement of assets selected as a* are then used to populate the display and, though may not be the global optimum, are considered to be optimal at the time during the iterative process when each asset was selected. These steps also ensure that depth and duplication requirements are considered when choosing assets for inclusion in a*.

[0132] In an example of how the semi-strong example would work for a given section d where the number of slots R_d=3 the local depth approach is used so the objective within the section is to maximize the value of m_t(L_t) in the section's ensemble. Initially, the first slot r=1 is populated with the anchor asset, a_i, so a*^(d,1)=a_i. Moving on to the next slot, r+1=2, the asset, a_j, is selected having the largest similarity to the asset in slot 1, a*^(d,1). Once selected, the optimal set now contains a*^(d)={a_i,a_j}. In the next iteration, the selected asset, a_k, is the one taken over all candidates for which the smallest of similarities f_i,k, f_j,k is maximized. That is, the measure of similarity of an asset consideration to join the optimal set, a*^(d) is based on a comparison to all existing members of the set established in prior iterations.

[0133] Knowing that the semi-strong ensemble construction might not be the optimal solution—through constructed optimization considered at each iteration over the d=1, . . . D and r=1, . . . , R_d possible slot—it may be possible to review the results of the resulting semi-strong solution a* and seek improvements. To construct the improved semi-strong ensembles for the layout L_t, it is of course required that semi-strong iterations first be conducted and a* established. The improvement process is fairly simple and based on a second set of iterations over each section d=1, . . . , D and each slot within a section r=1, . . . , R_d—skipping over each anchor asset a_i^(d) since their slot choice is predetermined and set by L_t—as follows:

[0134] Step 1: Define a^(−j) as the optimal set, excluding the asset aj filling the (d,r)th position of a* “(corresponding to the current iteration);

a⁽⁻ⁱ⁾∩a*=ø,a*=a*^(−j)∪a* .

[0135] Step 2: Substitute another candidate asset a_k≠a_j that is eligible to fill the a^(d,r) position of the optimal ensemble given duplicate and depth restrictions. Recompute:

m(L_t)=w^(d)+λ_tb(d) where 0≦λ_t≦1

• • [0136] to obtain the value that would result from the substitution.

[0137] Step 3: If the objective function m(L_t) is improved based on the substitution, replace the appropriate value of a* with the substitution.

[0138] The net result is that the final layout reflects a positioning and ensemble a* that yields a value of the objective function m(L_t) that is the equivalent or more acceptable than the semi-strong ensemble that existed prior to the improvement process. Additionally, if substitutions do exist, the improvement process forces the final ensemble a* to be more similar to the one that would have existed if the computationally-costly strong ensemble method had been employed.

[0139] Although a single layout, L_t has been discussed, embodiments can be used to evaluate the performance of various layouts and choices of depth, duplicate asset restrictions, and ensemble construction. In an online environment, the various inputs and asset portfolio manager choices may be varied to determine if an alternative to a single layout might be more effective. If the asset portfolio manager has a known and measurable performance metric in mind (for example, maximize revenue or engagement time), he/she may then use an embodiment to compare the performance of various presentation layouts/requirements in a multivariate or A/B testing environment to identify how various layouts might improve performance. If A is the asset library available to the portfolio manager, L={L_o, L₁, . . . , L_m} and R is a set of rules encompassing the choice of ensemble strength, duplicates, and down-weighted asset pairs, and X is the depth parameter, then the item portfolio may evaluate m(L_t|R_t,A, λ_t) for various combinations of R_t, λ_t to determine the set of inputs that yields a maximum value of the performance metric.

[0140] It should be noted that the execution of this series of steps may be computed by a single actor (the asset portfolio manager) based at single physical location or multiple actors in single or multiple physical locations (FIG. 13).

[0141] An embodiment can use a version discussed above to maximize the objective function m(L_t|R_t,A, λ_t) for any set of values. The embodiment can perform:

[0142] 1) video recommendations in an ordered playlist (a single-section layout) having a fixed number of slots (as determined by the client). The system uses a single sense/stationarity combination (sight/non-stationary) and allows any video type that is playable by the client software. The system may prohibit global duplication, uses a global depth requirement, down-weights asset pairs probability of co-presentation using client-provided business rules, and employs the semi-strong ensemble-building method.

[0143] 2) article-to-video layouts that take a text-based asset (a news article) as the anchor asset in each section and provides video recommendations in an ordered playlist (a single-section layout) having a fixed number of slots (as determined by the client) to accompany the text. The system uses a single sense/stationarity combination (sight/non-stationary) and allows any video type that is playable by the client software and selects video pairings that have the highest similarity to the anchor asset and other assets in the ensemble using the semi-strong ensemble approach.

[0144] 3) Multivariate and A/B testing can be performed to determine if the weak ensemble and semi-strong ensemble methods yield similar results and, if so, using the weak ensemble method to minimize computational resources needed to support the presentation.

[0145] 4) Recommendation engine—a recommendation engine can be implemented that determines how similarities are computed and feeds the ensemble-building.

[0146] 5) Allows a continuous-play feature that seamlessly builds a sequential set of videos that may be viewed without interruption. The feature allows an entire video playlist to be constructed and consumed and, should user interaction suggest that a particular video asset consumption has terminated, present a revised list of recommended assets presented as an ensemble.

[0147] In an embodiment, an apparatus comprises a processor and is configured to perform any of the foregoing methods.

[0148] In an embodiment, a non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of any of the foregoing methods.

[0149] Note that, although separate embodiments are discussed herein, any combination of embodiments and/or partial embodiments discussed herein may be combined to form further embodiments.

[0150] Examples of some embodiments are provided, without limitation, in the following paragraphs.

[0151] According to some embodiments, a method or non-transitory computer readable medium comprises: receiving a specification of an asset to be displayed during an online engagement; selecting a variety of assets related to the asset to be displayed during the online engagement, the variety of assets including any combination of: audio, video, text, or image, wherein the variety of assets are selected based on a relationship with the asset to be displayed during the online engagement; arranging the selected variety of assets into a single presentation screen; sending the presentation screen to a receiving device for display.

[0152] According to some embodiments, an apparatus comprises: a specification receiver that receives a specification of an asset to be displayed during an online engagement; an asset selector that selects a variety of assets related to the asset to be displayed during the online engagement, the variety of assets including any combination of: audio, video, text, or image, wherein the variety of assets are selected based on a relationship with the asset to be displayed during the online engagement; a presentation screen creator that arranges the selected variety of assets into a single presentation screen; a presentation screen sender that sends the presentation screen to a receiving device for display.

[0153] In some of the above embodiments, the single presentation screen is arranged using display parameters of the receiving device.

[0154] In some of the above embodiments, the variety of assets are selected based on a relationship between each asset among the variety of assets and the asset to be displayed during the online engagement.

[0155] In some of the above embodiments, each asset among the variety of assets is selected based on a relationship between the asset and the asset to be displayed during the online engagement, and wherein the asset is selected based on a relationship between the asset and all other assets in the variety of assets.

[0156] In some of the above embodiments, each asset among the variety of assets is selected based on a relationship between the asset and each asset among the variety of assets that was selected previous to the asset.

[0157] In some of the above embodiments, the asset to be displayed during the online engagement is textual.

[0158] In some of the above embodiments, the variety of assets are selected based on a common theme of the variety of assets.

[0159] In some of the above embodiments, each asset among the variety of assets is selected based on a variable probability.

[0160] In some of the above embodiments, at least a portion of the variety of assets are selected and arranged in the presentation screen based on human senses.

[0161] In some of the embodiments, the single presentation screen is arranged using display parameters of the receiving device.

[0162] In some of the above embodiments, the variety of assets are selected based on a relationship between each asset among the variety of assets and the asset to be displayed during the online engagement.

[0163] Other examples of these and other embodiments are found throughout this disclosure.

[0164] According to an embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.

[0165] For example, FIG. 14 is a block diagram that illustrates a computer system 1400 upon which an embodiment may be implemented. Computer system 1400 includes a bus 1402 or other communication mechanism for communicating information, and a hardware processor 1404 coupled with bus 1402 for processing information. Hardware processor 1404 may be, for example, a general purpose microprocessor.

[0166] Computer system 1400 also includes a main memory 1406, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 1402 for storing information and instructions to be executed by processor 1404. Main memory 1406 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 1404. Such instructions, when stored in non-transitory storage media accessible to processor 1404, render computer system 1400 into a special-purpose machine that is customized to perform the operations specified in the instructions.

[0167] Computer system 1400 further includes a read only memory (ROM) 1408 or other static storage device coupled to bus 1402 for storing static information and instructions for processor 1404. A storage device 1410, such as a magnetic disk, optical disk, or solid-state drive is provided and coupled to bus 1402 for storing information and instructions.

[0168] Computer system 1400 may be coupled via bus 1402 to a display 1412, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 1414, including alphanumeric and other keys, is coupled to bus 1402 for communicating information and command selections to processor 1404. Another type of user input device is cursor control 1416, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 1404 and for controlling cursor movement on display 1412. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

[0169] Computer system 1400 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 1400 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 1400 in response to processor 1404 executing one or more sequences of one or more instructions contained in main memory 1406. Such instructions may be read into main memory 1406 from another storage medium, such as storage device 1410. Execution of the sequences of instructions contained in main memory 1406 causes processor 1404 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0170] The term “storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operate in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, or solid-state drives, such as storage device 1410. Volatile media includes dynamic memory, such as main memory 1406. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0171] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 1402. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

[0172] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 1404 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 1400 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 1402. Bus 1402 carries the data to main memory 1406, from which processor 1404 retrieves and executes the instructions. The instructions received by main memory 1406 may optionally be stored on storage device 1410 either before or after execution by processor 1404.

[0173] Computer system 1400 also includes a communication interface 1418 coupled to bus 1402. Communication interface 1418 provides a two-way data communication coupling to a network link 1420 that is connected to a local network 1422. For example, communication interface 1418 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 1418 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 1418 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0174] Network link 1420 typically provides data communication through one or more networks to other data devices. For example, network link 1420 may provide a connection through local network 1422 to a host computer 1424 or to data equipment operated by an Internet Service Provider (ISP) 1426. ISP 1426 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 1428. Local network 1422 and Internet 1428 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 1420 and through communication interface 1418, which carry the digital data to and from computer system 1400, are example forms of transmission media.

[0175] Computer system 1400 can send messages and receive data, including program code, through the network(s), network link 1420 and communication interface 1418. In the Internet example, a server 1430 might transmit a requested code for an application program through Internet 1428, ISP 1426, local network 1422 and communication interface 1418.

[0176] The received code may be executed by processor 1404 as it is received, and/or stored in storage device 1410, or other non-volatile storage for later execution.

[0177] In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the embodiments, and what is intended by the applicants to be the scope of the embodiments, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.