Synchronized multimedia integration language extensions

[0001] 1. Field of the Invention

[0002] The present invention relates to a system for developing and executing a multimedia presentation. More particularly, the present invention concerns an extensible markup language (XML)-based system which allows development and execution of a synchronized multimedia presentation, wherein a sequence of the presentation and/or the behavior of a multimedia object therein is alterable in response to a user event, a system event, a timed event, and/or a detected state of another multimedia object.

[0003] 2. Description of the Related Art

[0004] The hypertext markup language (HTML) is an XML-based language commonly used to create pages to be displayed by a World Wide Web browser application. More particularly, a developer creates a web page by developing an HTML source file which includes data specifying particular items to be included in the page and spatial relationships among the items. Upon receiving an HTML source file, a web browser parses the file to obtain instructions for formatting and displaying the items of the web page to a user.

[0005] Like other high-level programming languages, HTML utilizes fairly intuitive syntax and text-based commands. In addition, a previously-created HTML source file can easily be used as a model for a new source file and resulting web page, or can be edited so as to reformat its resulting web page using simple text-based editing. HTML has proved popular due to its reusability, editability, the simplicity of its required syntax and the functionality provided by its defined commands.

[0006] However, traditional HTML can be used only to define spatial, rather than temporal, relationships of objects within a web page. One proposal for providing control over temporal relationships between objects is video streaming, in which frames of a video presentation are sequentially-delivered by an Internet server directly to a web browser. However, such a video presentation can be edited only by using sophisticated video-editing equipment, and is not easily-editable using a text-based markup language. Moreover, the video approach requires a very high bandwidth to produce a presentation of reasonable quality.

[0007] Another proposed approach requires animated GIF-formatted images to be merged into a single file, which is then downloaded to a web browser for rapid sequential display. Although such animation can be edited by adding, deleting, or substituting images of the animation, editing is significantly more difficult and less flexible than web page editing using a traditional markup language, particularly in the typical case in which a user has access only to the merged file. This approach also requires constant refresh of the web browser display, and therefore unacceptably consumes transmission bandwidth.

[0008] In addition to the above shortcomings, the content of a video presentation or an animated GIF presentation cannot be modified during execution of the presentation based on an occurrence of an event. It is possible to display, through a web browser, multimedia presentations which are responsive to events and/or user input by executing a suitable Java-language program using the browser's Java Virtual Machine. Development of these programs, however, requires a significant computer programming background. Therefore, this Java-based approach fails to provide the simplicity and editability of languages such as HTML.

[0009] In response to the foregoing, the World

[0010] Wide Web Consortium (W3C) has circulated a proposed standard, entitled Synchronized Multimedia Integration Language (SMIL) 1.0 Specification, the contents of which are incorporated herein by reference. SMIL is text-based, XML-based, editable and reusable. In this regard, the W3C standard Extensible Markup Language (XML) 1.0 is also incorporated herein by reference.

[0011] SMIL is intended to provide control over temporal relationships between web page objects. Specifically, SMIL allows a user to define whether multimedia objects of a page are executed in parallel or sequentially, and further provides nested control of parallel/sequential object execution.

[0012] For example, the SMIL specification defines the <par> element, which is used to specify elements to be executed in parallel. FIG. 1 shows an illustrative example of a SMIL source file portion utilizing the <par> element. The source file portion shown in FIG. 1 begins with <par> element 100 and ends with corresponding </par> notation 101. It should be noted that the <element> </element> beginning/end syntax is an XML grammar requirement.

[0013] According to the SMIL specification, all child elements nested one level below elements 100 and 101 are to be executed in parallel. Therefore, since two child elements <seq> 110 and <seq>120 exist at a first level below <par> element 100, objects corresponding to elements 110 and 120 are each executed in parallel.

[0014] In the case of <seq> 110, two media object elements exist between <seq> 110 and end notation 111. In this regard, and also according to the SMIL specification, elements existing as children to a <seq> element are executed sequentially. Therefore, video element 130 is processed, followed by video element 140. It should be noted that statements 130 and 140 each utilize the XML shorthand syntax which allows end notation / to be located within an element declaration.

[0015] As described above, child elements to <seq> element 120 are executed in parallel with the child elements of <seq> element 110 by virtue of <par> element 100. Therefore, all elements between <seq> element 120 and notation 121 are processed in parallel with elements 130 and 140. In this regard, nested within element 120 and notation 121 are <par> element 150, corresponding notation 151, <seq> element 160 and notation 161. According to <seq> element 120, the video sources indicated by video elements 170 and 180 are first played in parallel, followed by the video sources of video elements 190 and 200, which are played sequentially.

[0016] FIG. 2a and FIG. 2b illustrate a portion of a multimedia presentation governed by the SMIL source file shown in FIG. 1. As shown in FIG. 2a, video V₁ begins executing at a same time that video V₃₁ and video V₃₂ begin executing in parallel. In this example, while either one of video V₃₁ or video V₃₂ continues to play, video V₁ finishes and, by virtue of <seq> element 110, video V₂ begins to play. At time t₁, the one of video V₃, and video V₃₂ having a longer duration than the other terminates.

[0017] Next, as shown in FIG. 2b, video V₂ continues to play and video V₄ begins to play in parallel by virtue of <par> element 100. It should be noted that video V₄ begins to play upon termination of the longer of video V₃₁ and V₃₂ due to <seq> element 120. After termination of video V₄, video V₅ is played.

[0018] FIG. 2c shows a timeline describing the presentation illustrated in FIG. 2a and FIG. 2b. It also should be noted that FIG. 2c represents only one possible timeline resulting from the FIG. 1 SMIL source file, and that the timeline of FIG. 2c depends heavily upon relative durations of the video objects used therein. As described with respect to FIG. 1, FIG. 2a and FIG. 2b, the timeline of FIG. 2c shows that video V₁ and video V₂ are played sequentially while video V₃₁ and video V₃₂ are played in parallel. After termination of V₃₂, and while video V₂ is playing in parallel, video V₄ and video V₅ are sequentially played.

[0019] The SMIL specification provides for presentation control mechanisms in addition to those shown in FIG. 1. For example, attributes can be added to the <par> and <seq> statements to alter their above-described functionality. The SMIL specification also describes several media object elements which can be used in addition to the <video> element of FIG. 1. These elements, which are described in detail in the SMIL specification, include <ref>, <animation>, <audio>, <img>, <video>, <text> and <textstream>. Each of the listed media object elements can also be used with specified attributes which influence their respective functioning.

[0020] Notably, the <par>, <seq>, and media object elements are each controllable based on system attributes, such as bit rate and screen size. As described in the SMIL specification, when an attribute specified for an element evaluates to false, the element carrying this attribute is ignored. For example, if statement 170 of FIG. 1 were replaced by

[0021] <video src v₃₁.mpg system-bitrate56000/>

[0022] then, upon encountering line 170, a SMIL-enabled browser would evaluate the approximate bandwidth, in bits-per-second, available to the currently-executing system. In a case that the bandwidth is smaller than 56000, the replaced line 170 would be ignored. As a result, video V₃, would be absent from the display shown in FIG. 2a.

[0023] Other system attributes defined in detail in the SMIL specification include system-captions, system-language, system-overdub-or-caption, system-required and system-screen-depth. Each of these attributes can be included in a <par>, <seq>, or media object statement as described above to change the functioning thereof.

[0024] SMIL 1.0 also provides a <switch> element which can be helpful in controlling a multimedia presentation based on the above-described system attributes. The <switch> element allows an author to specify a set of alternative elements from which only one acceptable element should be chosen.

[0025] In this regard, upon encountering a <switch> element in a SMIL document, a browser evaluates the children of the <switch> element in the order in which they are listed. The first accepted child is selected to the exclusion of all other children of the <switch> element.

[0026] For example, in a case that lines 160, 190, 200 and 161 of FIG. 1 were replaced with

	<switch>
	<video srcv4.mpg system-bitrate56000/>
	<video srcv5.mpg system-bitrate28800/>
	</switch>,

[0027] the presentation sequence governed by FIG. 1 would depend upon the approximated bandwidth. Specifically, in a case that the bandwidth is greater than or equal to 56000, only video V₄ would be played in FIG. 2b upon termination of parallely-executing video V₃, and video V₃₂. On the contrary, if the system bitrate is less than 56000 but greater than 28800, video v₅ would be played exclusively in FIG. 2b.

[0028] One shortcoming of the above-described elements is that the testable attributes defined by SMIL 1.0 are all system-based attributes, in that they concern characteristics of the computing system upon which the browser is executing, or the system to which the browser is connected. Thus, although the <par> and <seq> synchronization elements and the testable attributes described above are useful in creating a multimedia presentation, these features do not provide control of multimedia presentations based on anything non-system based, such as a user interaction, a timed event, or a state change of an object.

[0029] In addition, SMIL 1.0 offers no ability to change an attribute of an object during execution of a multimedia presentation. As such, the functionality and flexibility of SMIL 1.0 is somewhat limited.

[0030] Accordingly, what is needed is a system for producing and displaying pages to be transmitted over the World Wide Web which provides control of multimedia objects on a web page based on user events, system events, timed events, or object state, and control over object attributes. Such a system also preferably allows simple and efficient creation and editing of such a web page and the control mechanisms therein.

[0031] The present invention addresses the foregoing by providing XML-based markers representing functions to detect events and to control attributes of media objects in accordance with the detected events.

[0032] In one aspect, the present invention is an XML-based event marker which includes an event parameter indicating an event, the marker representing a function to detect the indicated event. According to the further aspects, the event can be a user event such as a mouse click or a system event such as a memory low alert.

[0033] The present invention also includes an XML-based action marker which contains an object parameter indicating a media object, an attribute parameter indicating an attribute of the media object, and a value parameter indicating a value of the attribute. Advantageously, the action marker represents a function to assign the value to the attribute of the media object.

[0034] In another aspect, an XML-based interpolate marker includes an object parameter indicating a media object, an attribute parameter indicating an attribute of the media object, a begin parameter indicating a first time, an end parameter indicating a second time, and an end value parameter indicating a value of the attribute. The interpolate marker according to the invention represents a function to gradually change a value of the attribute to the end value over a period beginning at the first time and ending at the second time.

[0035] In addition, the present invention contemplates an XML-based condition marker containing an Id parameter indicating an element, an attribute parameter indicating an attribute of the element, and a value parameter indicating a value of the attribute, wherein the condition marker represents a function to detect whether or not the attribute of the element possesses the value.

[0036] According to the invention, the above-described markers can be utilized together in order to provide sophisticated functionality. For example, in one embodiment, the invention is a set of XML-based markers including an event marker indicating an event, and an action marker indicating a media object, an attribute of the media object, and a value of the attribute, wherein the set of markers represents a function to assign the value to the attribute if the event is detected.

[0037] In another embodiment, the present invention is a set of XML-based markers including an event marker indicating an event, a condition marker indicating a state of a first media object, and an action marker indicating a second media object, an attribute of the second media object, and a value of the attribute. This set of markers represents a function to assign the value to the attribute of the second media object if the event is detected and if the first media object possesses the indicated state.

[0038] In an additional embodiment, the invention is a set of XML-based markers including an event marker indicating an event, and an interpolate marker indicating a media object, an attribute of the media object, a first time, a second time, and a value of the attribute, wherein the markers represent a function to gradually change a value of the attribute to the end value over a period beginning at the first time and ending at the second time if the event is detected.

[0039] In yet another embodiment, the invention is a set of XML-based markers which include an event marker indicating an event, a condition marker indicating a state of a first media object, and an interpolate marker indicating a second media object, an attribute of the second media object, a first time, a second time, and a value of the attribute. According to this embodiment, the markers represent a function to gradually change a value of the attribute to the end value over a period beginning at the first time and ending at the second time if the event is detected and if the first media object possesses the indicated state.

[0040] In another aspect, the present invention relates to a set of XML-based markers representing media object elements, each marker including a test-element attribute for indicating a particular media object element, a test-attribute attribute for indicating an attribute of the particular element, and a test-value attribute for indicating a test value to be compared with a value of the specified attribute.

[0041] Because of the extension of test attributes from mere system-based attributes according to the prior art, the invention is able to provide selection between alternative different multimedia presentations based on non-system-based characteristics, such as user interaction or the characteristics and content of other multimedia objects.

[0042] This brief summary has been provided so that the nature of the invention may be understood quickly. A more complete understanding of the invention can be obtained by reference to the following detailed description of the preferred embodiment thereof in connection with the attached drawings.

[0043] FIG. 1 shows a portion of document source code according to the SMIL 1.0 specification.

[0044] FIG. 2a shows a display of a web browser executing a first part of the FIG. 1 source code.

[0045] FIG. 2b shows a display of a web browser executing a second part of the FIG. 1 source code.

[0046] FIG. 2c is a timeline showing temporal relationships of multimedia objects executing according to the FIG. 1 source code.

[0047] FIG. 3 is an outward view of a computing environment utilizing the present invention.

[0048] FIG. 4 is a block diagram illustrating the internal architecture of a computer utilizing the present invention.

[0049] FIG. 5 is a block diagram illustrating the internal architecture of a web server utilizing the present invention.

[0050] FIGS. 6a and 6b illustrate sequences of a multimedia presentation according to the present invention.

[0051] FIG. 7a is a graph illustrating a volume change of an audio object according to the present invention.

[0052] FIG. 7b is a graph of illustrating a volume change of an audio object according to the present invention.

[0053] FIG. 8a and FIG. 8b are views illustrating event detection and object state change during a multimedia presentation according to the present invention.

[0054] FIG. 3 is an outward view of a representative computing system utilizing the present invention.

[0055] Computing equipment 1 is preferably an Intel Pentium-based computer executing a windowing operating system such as Microsoft Windows98. Computing equipment 1 includes-display 2 for displaying images to a user and fixed disk 3 which stores computer-executable process steps of the windowing operating system and of other applications executed by computing equipment 1, such as a World Wide Web browser application. Fixed disk 3 also stores data files and device drivers for use by computing equipment 1. Also provided with computing equipment 1 are keyboard 4 for entering text and commands into appropriate fields displayed on display 2, and pointing device 5, such as a mouse, for pointing to, selecting and manipulating objects displayed on display 2.

[0056] Floppy disk drive 6 provides an interface to computing equipment 1 for reading data from and writing data to a floppy disk inserted therein. Using floppy disk drive 6, the above-described computer-executable process steps and/or data files may be input to computing equipment 1 and stored on fixed disk 3. Computer-executable process steps and data files may also be retrieved over a network via network connection 8 or via telephone line 9 from World Wide Web 10. In addition, image data files can be retrieved from scanner 12 and stored on fixed disk 3.

[0057] Multimedia speakers 14 provide sound output based on audio data files executed by computing equipment 1. Such an audio file may be in a monaural or stereo format, or in any other type of audio format, so long as computing equipment 1 is provided with a corresponding audio decoder and player application.

[0058] Computer-executable process steps and data files obtained by computing equipment 1 over World Wide Web 10 are transferred thereto by servers such as server 15. In response to a request for data, server 15 collects the required data, properly formats the data, and sends the data to computing equipment 1 over World Wide Web 10.

[0059] FIG. 4 is a block diagram of the internal architecture of computing equipment 1. Shown in FIG. 4 is CPU 20, which, as described above, is preferably a Pentium processor. CPU 20 interfaces to computer bus 21, as does scanner interface 22 for interfacing to scanner 12, speaker interface 24 for interfacing to speakers 14, network interface 25 for interfacing to network connection 8, modem interface 26 for interfacing to telephone line 9 and display interface for interfacing to display 2. Mouse interface 29, which interfaces to mouse 5, and keyboard interface 30, which interfaces to keyboard 4, are also connected to bus 21. In this regard, interfaces 22 to 30 allow computing equipment 1 to access the functionality of their corresponding components. Also shown in FIG. 4 is disk 3, having stored thereon the aforementioned windowing operating system, a web browser, XML-based source files according to the present invention, which, for convenience sake, are hereinafter referred to as Synchronized Multimedia Integration Language Extended (SMILE) source files, a SMILE file editor application, other applications, other data files and device drivers.

[0060] The web browser stored on fixed disk 3 is preferably capable of interpreting elements and attributes of a SMILE source file and executing a corresponding multimedia presentation in accordance with functionality dictated by the elements and attributes. For example, Netscape Navigator and Internet Explorer are common HTML-enabled browsers, and a SMIL enabled browser is currently available from Realnetworks.

[0061] Read only memory (ROM) 31 stores invariant computer-executable process steps for basic system functions such as basic I/O, start-up, or reception of key strokes from keyboard 4.

[0062] Main random access memory (RAM) 32 provides CPU 20 with memory storage which can be accessed quickly. In this regard, computer-executable process steps of a web browser or other application are transferred from disk 3 over computer bus 21 to RAM 32 and executed therefrom by CPU 20.

[0063] FIG. 5 is a block diagram of several relevant components internal to server 15. As shown, server 15 is connected to World Wide Web 10 via World Wide Web connection 40, which may be a telephone line, a T1 line, a local area network connection, or the like. In a case that World Wide Web connection 40 connects directly to a local area network, the local area network is preferably connected to a router which, in turn, is connected to World Wide Web 10. In such a configuration, the router includes firewall software for prevention of unauthorized access to the local area network.

[0064] Data packets received over World Wide Web 10 (IP packets) travel over connection 40 to TCP/IP layer 41. TCP/IP layer 41 re-orders the IP packets and parses data therefrom. The parsed data is delivered to HTTP (hypertext transfer protocol) server 43. Based on the parsed data, HTTP server 43 retrieves appropriate files from file storage 44 and transfers the files to TCP/IP layer 41. The files are then formatted into IP packets and sent over World Wide Web connection 40 to computing equipment 1.

[0065] According to the present invention, file storage 44 stores at least source files in a SMILE format according to the present invention, as well as text, video and audio objects which are referenced by stored SMILE source files. File storage 44 may also store Java applets which can be executed by a Java virtual machine of a web browser executing in computing equipment 1.

[0066] It should be noted that other servers and protocol layers may be used by server 15. In this regard, although HTTP server 43 and TCP/IP layer 41 are useful for transmitting text and fixed images over a firewall, a specialized streaming server utilizing the TCP or UDP protocol may be preferred for sending streaming audio or video data over World Wide Web 10.

[0067] I. Test Attributes

[0068] As mentioned in the Background of the Invention, the SMIL specification allows conditional execution of elements based only on system attributes, such as bit rate or screen size. According to one aspect of the present invention, test attributes, which can be any attributes of an object or element, can be defined for use in any of the Spar>, <seq>, or media object elements. As a result, presentation flow can be controlled based on non-system-based attributes.

[0069] More specifically, the present invention defines three new attributes for the <par>, <seq> and media object elements, test-element, test-attribute, and test-value. The test-element attribute can be assigned an Id of a particular element to indicate that an attribute of the particular element is to be tested. In this regard, the test-attribute attribute specifies the attribute of the particular element to be tested. Finally, the test-value attribute indicates the value with which the value of the specified attribute is to be compared.

[0070] For example, the source code:

	<audio idAudio srcAudio.wav test-
	elementVideo test-attributesound test-
	valueon/>
	<textstream idText srctext.txt test-
	elementVideo test-attributesound test-
	valueoff/>

[0071] causes a sound file Audio.wav to be played in a case that a sound attribute of a video object having the Id Video has the value on, and causes a text file text.txt to be displayed in a case that the sound attribute of the video object has the value off.

[0072] For media object elements, the present invention also defines a status attribute, a sync attribute, and a start-asap attribute. The status attribute indicates whether an element is activated or deactivated and can be assigned the values activate (the default value) or deactivate.

[0073] The sync attribute is used to indicate a group of elements with which execution of an object should be strictly synchronized. In addition, the start-asap attribute indicates that object execution should begin as soon as possible.

[0074] II. The <Action> element

[0075] A SMILE source file according to the present invention may include an <action> element for changing a value of an attribute of another element. The <action> element can therefore be used for inter-object communication or for changing the behavior of objects in a presentation in real-time. According to the present invention, the <action> element can include the id and skip-content attributes defined in the SMIL specification. Moreover, the <action> element can include the attributes dest, order-number, attribute, value, and begin.

[0076] The dest attribute refers to a particular element upon which the <action> element is intended to operate. In this case, an Id for the particular element can be defined using the SMIL id attribute and the Id can then be referred to using the dest attribute of the <action> element. For example, based on the source code

[0077] <video IdmyVideo srcvideo.mpg/>,

[0078] an <action> element statement for operating on the video element would contain the attribute definition destmyVideo.

[0079] The order number attribute assigns an order number to an <action> element so as to indicate a time at which the particular <action> element is to be executed relative to other <action> elements. Such an attribute is useful if, as described above with respect to FIG. 1, an object-execution sequence of a source file is dependent upon relative durations of the objects referenced therein.

[0080] The attribute element identifies an attribute of the particular element indicated by the dest attribute. In this regard, the value attribute indicates a value to which the identified attribute is to be altered. Accordingly, any defined attribute of any defined element can be altered simply by referencing the element and the attribute using the dest and attribute attributes, respectively. Finally, the begin attribute indicates a time delay from a time at which the <action> element is encountered by the web browser to actual execution of a specified value change.

[0081] The following is a representative example of code utilizing the above-described <action> element.

	<par>
	<audio srcAudio1.wav volume10 Idaudio1/>
	<seq>
	<video srcVideo1.mpg/>
	<action destAudio1 attributevolume
	value20/>
	<seq>
	</par>

[0082] As explained above, all child elements at a first level below the <par> element are to be executed in parallel. Therefore, in the present case, the above <audio> element and the elements encompassed by the <seq> element are performed in parallel.

[0083] As shown, the <audio> element according to the present invention can include a volume attribute, the value of which corresponds to a volume at which the referenced audio object is played through speakers 14. By virtue of the above source code, the audio file Audio₁.wav is played at a volume 10.

[0084] FIG. 6a shows a display of a browser executing the above source code. At time 0, both audio file Audio₁.wav and video file Video₁.mpg are executed. It should be noted that Video₁.mpg is executed since it is a first element in a sequence which is to be performed in parallel with the Audio₁.wav file.

[0085] For the purpose of the present example, it is assumed that Video₁.mpg has a duration of 5 seconds. Accordingly, after 5 seconds has elapsed, the above-listed <action> statement is executed. In this regard, the <action> statement includes a dest attribute and an attribute attribute, which identify, respectively, the listed <audio> element and the volume attribute of the <audio> element. The <action> element also includes a value attribute, which, as described above, refers to a value to which the attribute referenced by the <action> element is to be changed.

[0086] In the present example, the volume attribute of the <audio> element should be changed to 20. Accordingly, FIG. 6b shows a representation of a multimedia presentation dictated by the above source code after 5 seconds have elapsed. As shown, Video₁.mpg has ceased to play and file Audio₁.wav is played at a volume corresponding to a volume attribute value of 20.

[0087] The functionality provided by the above-described <action> element is not provided by the SMIL specification. The <action> element advantageously provides control over object attributes before, during and after object execution, thereby greatly increasing the presentation options available to a presentation developer. A further advantage of the <action> element is that the <action> element is easy to understand, use and edit, thereby preserving the user friendliness of XML-based languages.

[0088] III. The <Interpolate> Element

[0089] According to the present invention, an <interpolate> element is provided for changing a value of an element attribute. In this respect, it will be noted that the <interpolate> element is similar to the <action> element. However, as described in detail below, the <interpolate> element provides greater control than the <action> element over the manner in which an attribute value is changed over time.

[0090] The <interpolate> element includes a id attribute and a skip-content attribute as defined by the SMIL specification. The <interpolate> element also includes a dest attribute which identifies an element, an attribute of which will be changed according to the <interpolate> element. In this regard, the <interpolate> element also includes an attribute attribute, which specifies the attribute of the referenced element which is to be changed.

[0091] The <interpolate> element further includes a begin attribute which specifies an elapsed time, from a time when the <interpolate> element is encountered by a web browser, after which the specified attribute should begin changing.

[0092] Relatedly, an end attribute is also provided which specifies an elapsed time, from a time when the <interpolate> element is encountered, at which the attribute should cease changing. In addition, the <interpolate> element includes an end-value attribute, which indicates a value to which the referenced attribute should be changed.

[0093] FIG. 7a is a graph of volume versus time for the presentation shown in FIG. 6a and FIG. 6b. As shown, and as described above, the volume of audio file Audio₁.wav is set to 10 for the first 5 seconds of the presentation, after which time the volume attribute is set to 20 by virtue of the described <action> element.

[0094] FIG. 7b shows a corresponding graph in a case that the <action> element statement is substituted with the statement:

	<interpolate destAudio1 attributevolume
	end-value20 begin2s end5s/>.

[0095] As described above, the attributes of the foregoing <interpolate> element indicate that the volume attribute of the element having the Id Audio₁ should begin changing 2 seconds after encountering the <interpolate> element. Moreover, the attributes indicate that the changing should continue for 3 seconds (5 seconds-2 seconds) and the volume should be 20 at the end of the changing.

[0096] Continuing with the assumption that the Video₁.mpg has a duration of 0.5 seconds, FIG. 7b shows that Audio₁.wav is played at a volume of 10 for the first 5 seconds of the presentation, after which point the <interpolate> element is encountered. Based on the begin attribute, the volume attribute begins changing 2 seconds after the <interpolate> element is encountered, or 7 seconds into the presentation. The changing continues until 5 seconds after the <interpolate> element was encountered, or 10 seconds into the presentation. As shown by FIG. 7b, the result of the <interpolate> element is a gradual transition from the initial attribute value to the final attribute value.

[0097] Accordingly, the <interpolate> element offers new functionality similar to that offered by the <action> element, and also allows for gradual change of element attribute values.

[0098] Conventionally, controls of an object utilized in a HTML or a SMIL presentation are not exposed in corresponding HTML or SMIL source code. Rather, according to those languages, an object is merely downloaded from a web location indicated in the source code. As such, object controls-cannot be manipulated through editing of the source file. Therefore, the <action> and <interpolate> elements of the present invention advantageously offer simple control over presentation objects not offered by the above conventional systems.

[0099] IV. The <Event> Element

[0100] According to another aspect, the present invention includes a <event> element which captures user events or system events. Upon capture of a specified event, child elements to the <event> element are processed.

[0101] The <event> element includes the id and skip-content attributes defined in the SMIL specification. Also included are a source-event attribute, which defines the type of event to be captured, and a coord attribute for specifying a coordinate position on a browser display. The source-event attribute can have the following values: leftmouseclick, rightmouseclick, mouseover, mousein, mouseout, mousedown, and timerevent. If no value for the source-event attribute is specified in an <event> element, the element defaults to a TRUE value.

[0102] The source-event attribute value leftmouseclick refers to a press-and-release of the left button of mouse 5. Several distinct types of situations can be detected by setting the source-event attribute to leftmouseclick. First, in a case that the <event> element is not a child element to a media object and the source-event attribute is assigned a value of leftmouseclick, any press and release of the left mouse button is detected upon encountering the <event> element. On the other hand, if particular coordinates are specified using the coord attribute, only those press-and-releases of the left mouse button which occur while a displayed mouse pointer is within the specified coordinates are captured.

[0103] Moreover, in a case that an <event> element is a child element to an element such as, for example, a video element specifying a video object, only those press-and-releases of the left mouse button which occur while the pointer is on the executing video object are captured. However, if coordinates are also specified using the coord attribute, then only those press-and-releases occurring while the pointer is within the specified coordinates are captured.

[0104] The rightmouseclick attribute value functions as described above with respect to the leftmouseclick value, but instead with respect to the right button of mouse 5. In addition, the mousedown value of the source-event attribute operates similarly to the leftmouseclick and rightmouseclick values, however, the mousedown value describes and is used to detect a situation in which a mouse button is merely pressed but not necessarily released in an appropriate area.

[0105] The mouseover value operates similarly to the leftmouseclick and rightmouseclick values described above, however, a mouseover source event does not require any movement of a mouse button. Rather, if a mouseover attribute value is specified with an <event> element, then an event is captured in a case that the pointer is in a region of the browser display specified by a parent element of the <event> element or by values of a coordinate attribute of the <event> element. Using the previous example, in a case that such an <event> element is a child element to a <video> element, the event is captured in a case that a pointer is moved to any area of the executing video. In a case that the <event> element is not a child of a media object and does not include a coordinate attribute, the region of detection is the entire browser display.

[0106] A mousein or mouseout source event is captured in a case that a mouse pointer is moved into a region of a parent media object element or out of the region of the parent media object element, respectively. If used in conjunction with a coordinate attribute, a mousein or mouseout source event is captured in a case that a mouse pointer is moved into a region defined by a specified value of the coordinate attribute. Similarly to the mouseover value, the region of interest is the entire browser display for an <event> element specifying a mousein or mouseout source event value which is not a child of a media object and does not include a coordinate attribute.

[0107] The timerevent value of the source-event attribute is preferably not used with a specified coord attribute value. This attribute value is used in <event> elements which are child elements to a media object element. The timerevent value is used to detect an interrupt included with the parent media object.

[0108] The foregoing is an example of a portion of a source code file according to the present invention which utilizes the above-described <event> and <action> elements:

	<par>
	<video srcVideo2.avi IdVideo2
	statusdeactivate/>
	<Img srcImage1.gif>
	<event source-eventleftmouseclick/>
	<action destVideo2
	attributestatus
	valueactivate/>
	</event>
	</img>
	</par>

[0109] The <video> element of the above source code is assigned an Id Video₂ and a source Video₂.avi. Moreover, the <video> element is assigned a deactivate value to its status attribute. Accordingly, upon encountering the <video> element, a web browser executing the source file does not play the referenced video file Video₂.avi . The next statement in the source code is an <img> element having a source file Image₁.gif. Accordingly, as shown in FIG. 8a, the referenced image Image₁.gif is displayed by the web browser.

[0110] An <event> element is defined as a child element to the above-described <img> element. Accordingly, the <event> element is evaluated during display of the referenced image. As shown above, the assigned value of the source-event attribute is leftmouseclick. Based on the above discussion, a leftmouseclick event is deemed to be captured upon detection of a press and release of the left mouse button of mouse 5 while a mouse pointer is within the display area of Image₁ as shown in FIG. 8a. As also described above, in a case that a coord attribute is also defined in the <image> element, the pointer must be located within the specified coordinates for the event to be deemed captured.

[0111] Upon capture of the specified event, the listed <action> element is processed. As shown, the <action> element contains attributes which indicate that the status attribute of the <video> element indicated by the Id Video₂ is to be changed to activate. Accordingly, and as shown in FIG. 8b, Video₂.avi begins to play.

[0112] V. The <Condition> Element

[0113] The <condition> element can be used to test for the existence of a particular situation so as to allow selective performance of an action based on whether or not the situation exists.

[0114] The <condition> element includes an id and a skip-content attribute as defined by the SMIL specification. The <condition> element also includes a dest attribute, which refers to an Id of a particular element to be tested, an attribute attribute, which refers to a particular attribute of the particular element to be test, and a value attribute, which refers to a value to be compared with the value of the particular attribute to be tested. In operation, if the value indicated by the value attribute of the <condition> element is equal to the value of the referenced attribute, the condition is deemed true and a next line of source code is executed.

[0115] The following portion of source code is identical to that described above with respect to FIG. 8a and FIG. 8b, with the addition of a <condition> statement:

	<par>
	<video srcVideo2.avi IdVideo2
	statusdeactivate/>
	<img srcimage1.gif>
	<event source-eventleftmouseclick/>
	<condition destVideo2.avi
	attributestatus valueactivate/>
	<action destVideo2
	attributestatus
	valueactivate/>
	</event>
	</img>
	</par>

[0116] In the above example, the <condition> statement is evaluated after a leftmouseclick is detected within the display area of image₁. In this regard, the <condition> statement evaluates whether the status attribute of the element indicated by the Video₂ Id has the value activate. If not, the subsequent <action> statement is not processed, even though the <action> statement is not nested within the <condition> statement. Conversely, if the <condition> statement is evaluated as true, the <action> statement is processed.

[0117] According to the invention, the <event>, <condition> and <action> elements can also be used together so as to change an attribute of an object based on a state change of an object. For example, in a case that the <event . . . . > element of the previous example was replaced with <event> , a browser encountering the replaced line would immediately evaluate the following <condition> element and execute or not execute the following <action> element based thereon.

[0118] VI. The <and-Condition> and <or-Condition> Elements

[0119] The <and-condition> element allows several conditions to be logically ANDd together. Specifically, the <and-condition> element can be used as follows to logically AND two conditions A and B:

	<and-condition>
	<condition idA .../>
	<condition idB .../>
	</and-condition>

[0120] The <and-condition> element returns a TRUE value in a case that all <condition> elements nested therein evaluate to TRUE. The <or-condition> element follows a similar syntax as that described above with respect to the <and-condition> element, however, the <or-condition> element logically ORs any <condition> elements nested therein.

[0121] The <and-condition> and <or-condition> elements can be nested. In this regard, the following portion of source code performs the function ((C or D) and (E and (F or G))):

	<and-condition>
	<or-condition>
	<condition idC .../>
	<condition idD .../>
	</or-condition>
	<and-condition>
	<condition idE .../>
	<or-condition>
	<condition idF .../>
	<condition idG .../>
	</or-condition>
	</and-condition>
	</and-condition>

[0122] Both the <and-condition> and the <or-condition> elements contain the attributes id and skip-content as defined by the SMIL specification. Also, as described above, both elements can contain <and-condition>, <or-condition> and <condition> child elements.

[0123] In a case that either an <and-condition> or an <or-condition> element are evaluated as true, a subsequent <action> statement is processed, as described above with respect to <condition> element.

[0124] VII. The <Switch> Element

[0125] As described above, the <switch> element as presently defined by the SMIL 0.1.0 specification does not provide for switching between child elements based on attributes other than system-based attributes. According to the present invention, however, the <switch> element includes test attributes which allow testing of attributes of other elements defined in a source code file.

[0126] The <switch> element according to the present invention includes a id and a title attribute, as defined by the SMIL 1.0 specification. The <switch> element also includes a selection attribute, which can be changed during source file execution using the elements described herein. The following source code is an example of such a use.

	<image srcselection.gif>
	<event source-eventleftmouseclick coord0%,
	0%, 50%, 100%>
	<action destmySwitch
	attributeselection value 1/>
	</event>
	<event source-eventleftmouseclick coord0%,
	50%, 100%, 100%>
	<action destmySwitch
	attributeselection value2/>
	</event>
	</image>
	<switch idmySwitch>
	<video srcmyVideo1.mpg test-
	elementmySwitch test-attributeselection
	test-value1/>
	<video srcmyVideo2.mpg test-
	elementmySwitch test-attributeselection
	test-value2/>
	</switch>

[0127] According to the above code, a detection of a left mouse click at one portion of a browser display causes a value of the selection attribute of a <switch> element having the Id myswitch to change to (or remain at) 1. If the event is detected at another part of the display, the value changes to (or remains at) 2.

[0128] Next, a first TRUE conditional statement nested in the <switch> element is processed, and the other is ignored. As shown, both statements nested in the <switch> statement are conditional media object statements by virtue of the attributes test-element, test-attribute and test-value explained above. Accordingly, myVideo1.mpg is played in a case that the selection attribute of the <switch> element has a value of 1, while myVideo2.mpg is played if the attribute has a value of 2.

[0129] The foregoing example illustrates several advantages of the present invention. Among others, one advantage lies in the ability provided by the invention to use <par>, <seq> and media object element statements which are conditionally executed based on an evaluation of non-system attribute values.

[0130] While the present invention is described above with respect to what is currently considered its preferred embodiments, it is to be understood that the invention is not limited to that described above. To the contrary, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.