INFORMATION PROCESSING APPARATUS AND INFORMATION PROCESSING METHOD

[0001] 1. Field of the Invention

[0002] The present invention relates to a technique for presenting Mixed Reality (MR).

[0003] 2. Description of the Related Art

[0004] Recently, studies of MR aiming at seamless combination of a physical space and virtual space have been made actively. An image display device which presents mixed reality is, for example, a device which displays an image obtained by superimposing and rendering, on an image of a physical space that is sensed by an image sensing device such as a video camera, an image in a virtual space (for example, a virtual object or character information rendered by computer graphics) that is generated in accordance with the position and orientation of the image sensing device. As such a device, for example, a Head Mounted Display (HMD) is usable, e.g., see Japanese Patent Application No. 2006-320872, Japanese Patent Application No. 2010-189458, and Japanese Patent Laid-Open No. 2007-172596 . The image display device is also implemented by an optical see-through method of displaying, on an optical see-through display mounted on the head of a user, an image in a virtual space that is generated in accordance with the position and orientation of the viewpoint of the user. There is disclosed a method using markers for calculating the position and orientation of an image sensing device from an image in a physical space that is acquired from the image sensing device (e.g., see Japanese Patent Application No. 2006-320872, and Japanese Patent Laid-Open No. 2007-172596).

[0005] When building a system which obtains the position and orientation of the viewpoint by using an image in the physical space including markers and enables a user to experience MR by using the obtained position and orientation of the viewpoint, a user having little knowledge of MR does not know an arrangement pattern of markers that stabilizes calculation of the position and orientation of the viewpoint.

[0006] The present invention has been made in consideration of the above-described problem, and provides a technique for creating information for notifying a user of an arrangement pattern of indices that stabilizes calculation of the position and orientation of the viewpoint.

[0007] According to an aspect of an exemplary embodiment, an information processing apparatus includes: an acquisition unit configured to acquire information of a position at which a user can experience mixed reality in a physical space; a generation unit configured to generate an arrangement pattern of indices in the physical space used to calculate a position and orientation of a viewpoint of the user based on the information of the position; and an output unit configured to output the arrangement pattern generated by the generation unit.

[0008] According to another aspect of an exemplary embodiment, an information processing method includes: acquiring information of a position at which a user can experience mixed reality in a physical space; generating an arrangement pattern of indices in the physical space used to calculate a position and orientation of a viewpoint of the user based on the information of the position; and outputting the generated arrangement pattern.

[0009] Further features will become apparent from the following description of exemplary embodiments with reference to the attached drawings.

[0025] Exemplary Embodiments are described below with reference to the accompanying drawings.

[0026] The first embodiment will explain one aspect of an information processing apparatus. This information processing apparatus generates (first generation) a plurality of types of arrangement pattern candidates. Each arrangement pattern candidate is an arrangement pattern candidate of a plurality of indices in a physical space used to calculate the position and orientation of the viewpoint. The arrangement pattern candidate enables observation of at least a predetermined number of indices from a position in an area where mixed reality can be experienced in the physical space. Then, the information processing apparatus generates (second generation) and outputs information representing the arrangement pattern of a plurality of indices in the physical space by using the generated types of arrangement pattern candidates.

[0027] First, an example of the functional arrangement of the information processing apparatus according to the first embodiment will be described with reference to the block diagram of FIG. 1. As shown in FIG. 1, the information processing apparatus according to the embodiment includes a main body unit 1000, display unit 1080, and operation unit 1010.

[0028] The display unit 1080 will be explained first. The display unit 1080 is constituted by a CRT, liquid crystal screen, or the like, and displays an image, character, and the like based on data output from the main body unit 1000.

[0029] The operation unit 1010 will be explained next. The operation unit 1010 is constituted by a keyboard, mouse, or the like. The user can operate the operation unit 1010 to input various instructions to the main body unit 1000.

[0030] The main body unit 1000 will be explained next. As shown in FIG. 1, the main body unit 1000 includes a data storage unit 1020, arrangement information generation unit 1030, arrangement information output unit 1040, and image output unit 1070.

[0031] The data storage unit 1020 stores MR experience information input by operating the operation unit 1010 by the user. The MR experience information is information used to decide a plurality of types of arrangement pattern candidates that enable observation of at least a predetermined number of indices from a position in an area where mixed reality can be experienced in a physical space. Details of the MR experience information will be described later.

[0032] By using the MR experience information stored in the data storage unit 1020, the arrangement information generation unit 1030 decides a plurality of types of arrangement pattern candidates that enable observation of a predetermined number or more of indices from a position in an area where mixed reality can be experienced in a physical space. The arrangement information generation unit 1030 stores the respective decided arrangement pattern candidates as pieces of index arrangement information in the data storage unit 1020.

[0033] By using the pieces of index arrangement information stored in the data storage unit 1020, the arrangement information output unit 1040 generates information representing the arrangement pattern of a plurality of indices in a physical space. The arrangement information output unit 1040 stores the generated information in the data storage unit 1020.

[0034] Based on the information stored in the data storage unit 1020 by the arrangement information output unit 1040, the image output unit 1070 generates information representing the arrangement pattern of a plurality of indices in the physical space. The image output unit 1070 outputs the information to the display unit 1080.

[0035] FIG. 2 shows a state in which the user experiences mixed reality. In FIG. 2, a cube 2000 represents a physical space in an area observable by the user who experiences mixed reality. A plurality of indices are arranged on a wall 2010 in the physical space.

[0036] To experience mixed reality, the user mounts a head mounted display on his head. As is well known, the head mounted display includes a display unit, and a camera which senses a physical space. In FIG. 2, reference numeral 2090 denotes a camera.

[0037] A cube 2050 represents an area where mixed reality can be experienced in the physical space represented by the cube 2000. As long as the head mounted display is located within the cube 2050, a user who mounts the head mounted display can experience mixed reality. In FIG. 2, the camera senses indices in a region 2060 falling within a field 2040 of view of the camera 2090 on a wall 2010 and outputs the sensed image to a mixed reality presentation apparatus (this apparatus may be the information processing apparatus according to the embodiment or another apparatus). The mixed reality presentation apparatus obtains the position and orientation of the camera 2090 by using the sensed image. A technique for obtaining the position and orientation of the camera 2090 by using a sensed image of indices is a well-known technique as disclosed in, for example, Japanese Patent Application No. 2006-320872, and Japanese Patent Laid-Open No. 2007-172596, described above, and a description of this technique will not be included herein. Then, the mixed reality presentation apparatus generates an image of a virtual object 2030 corresponding to the obtained position and orientation, and sends, to the display unit of the head mounted display, a composite image of the generated image of the virtual object 2030 and the image sensed by the camera 2090. Since this composite image is displayed on the display unit of the head mounted display, the user who mounts the head mounted display on his head can observe the composite image before his eyes, and experience mixed reality.

[0038] As described above, the camera 2090 is movable in the area represented by the cube 2050. In accordance with the position of the camera 2090, the appearance of indices arranged on the wall 2010 (in particular, the number of indices falling within the field 2040 of view) changes. If a sufficient number of indices do not fall within the field 2040 of view, the number of indices appearing on a sensed image becomes insufficient, reducing the position/orientation calculation accuracy.

[0039] In the embodiment, therefore, the information processing apparatus in FIG. 1 performs processing for deciding an index arrangement pattern in which a sufficient number of indices are sensed in a sensed image regardless of movement of the camera 2090 in the area represented by the cube 2050. The processing to be performed by the information processing apparatus according to the embodiment in order to decide an index arrangement pattern in which a sufficient number of indices are sensed in a sensed image will be explained with reference to FIG. 15 showing the flowchart of this processing.

[0040] In step S201, the user inputs MR experience information by operating the operation unit 1010. The control unit (not shown) of the information processing apparatus stores the MR experience information in the data storage unit 1020. As described above, the MR experience information is information used to decide a plurality of types of arrangement pattern candidates that enable observation of at least a predetermined number of indices from a position in an area where mixed reality can be experienced in a physical space. Thus, any information is available as long as the same purpose can be achieved by using the information. For example, assume that the MR experience information contains the following kinds of information:

[0041] information (physical space information) which defines a physical space (in FIG. 2, the space represented by the cube 2000) in an area observable by a user who experiences mixed reality

[0042] information (area information) which defines an area (in FIG. 2, the area represented by the cube 2050) where mixed reality can be experienced in a physical space in an area observable by a user who experiences mixed reality

[0043] information (for example, information representing the angle of view) representing the field of view of the camera of the head mounted display

[0044] The physical space information is, for example, information which defines a physical space in an area observable by a user who experiences mixed reality, in a world coordinate system set in the physical space (a coordinate system in which one point in the physical space is set as the origin and three axes perpendicular to each other at the origin are defined as the x-, y- and z-axes). In FIG. 2, the physical space information may be information which defines the space represented by the cube 2000 in the world coordinate system, or information which defines a portion at which indices are arranged in the world coordinate system. The physical space information may also be information which defines, in the world coordinate system, a portion of a virtual object that is designated by the user by operating the operation unit 1010 when the display unit 1080 displays the virtual object simulating a physical space.

[0045] The area information is, for example, information representing an area where mixed reality can be experienced in the world coordinate system.

[0046] In addition to these pieces of information, the MR experience information may contain pieces of information such as the camera resolution, lens distortion information, focal length information, and information which defines a region where a virtual object is to be arranged.

[0047] In step S202, the arrangement information generation unit 1030 reads out the MR experience information from the data storage unit 1020. The positional relationship between a “portion (in FIG. 2, the wall 2010) at which indices are arranged”, and an “area where mixed reality can be experienced” in the world coordinate system can be specified by using the physical space information and area information contained in the MR experience information. The arrangement information generation unit 1030 obtains a distance d1 from the portion (in FIG. 2, the wall 2010) at which indices are arranged, to a portion farthest from this portion (in FIG. 2, the wall 2010) in the area where mixed reality can be experienced, and a distance d2 from the portion (in FIG. 2, the wall 2010) at which indices are arranged, to a portion closest to this portion (in FIG. 2, the wall 2010) in the area where mixed reality can be experienced.

[0048] The processing in step S202 will be explained with reference to an example shown in FIG. 5. FIG. 5 shows a state in which the cube 2000 shown in FIG. 2 is looked down from just above it, as indicated by an arrow 2070.

[0049] In FIG. 5, the positional relationship between the wall 2010 serving as a portion at which indices are arranged, and the area (the area represented by the cube 2050) where mixed reality can be experienced can be specified by using the physical space information and area information contained in the MR experience information. In this case, a portion farthest from the wall 2010 in the area where mixed reality can be experienced is a portion 5100, so a distance 5000 between the wall 2010 and the portion 5100 is obtained as d1. A portion closest to the wall 2010 in the area where mixed reality can be experienced is a portion 5300, and thus a distance 5020 between the wall 2010 and the portion 5300 is obtained as d2.

[0050] Note that the distances d1 and d2 are not limited to those obtained by the above-mentioned method. For example, maximum and minimum distances input by the user by operating the operation unit 1010 may be set as d1 and d2, respectively.

[0051] Referring back to FIG. 15, in step S203, the arrangement information generation unit 1030 first sets d1 as the initial value of a variable D. Then, the arrangement information generation unit 1030 obtains a size S of a rectangle representing an index by calculating S=M×D. M is an appropriate constant, and may be set in advance or set by the user by operating the operation unit 1010. If the rectangle is a square, S may be the length and width. If the rectangle is an oblong, proper constants may be set as M1 and M2 to obtain a length S1=M1×D and a width S2=M2×D. In this manner, the rectangle size can be decided by an arbitrary method as long as the rectangle size is increased as D increases, and is decreased as D decreases.

[0052] By using the physical space information contained in the MR experience information, the arrangement information generation unit 1030 specifies a portion at which indices are arranged. The arrangement information generation unit 1030 arranges rectangles each having the size S side by side at this portion, thereby arranging a rectangle array (for example, 10000 in FIG. 8) at this portion. The arrangement is not limited to a specific one, and any arrangement can be employed as long as rectangles can be arranged side by side without projecting from the portion at which indices are arranged. “Arranging a rectangle array” is not processing of actually arranging and rendering rectangles, but processing which is performed schematically in calculation.

[0053] For example, FIG. 6B shows a portion of the rectangle array in a region 6060 falling within a field 6040 of view of a viewpoint 6020 set at a position spaced apart by the distance d1 (5000) from the wall 2010, as shown in FIG. 6A. In FIG. 6B, nine rectangles 6001 to 6009 are contained in the region 6060 and arranged at equal intervals. Further, in step S203, the arrangement information generation unit 1030 registers the set rectangle array as index arrangement information in the data storage unit 1020. The index arrangement information represents the three-dimensional coordinate position (for example, a coordinate position in the world coordinate system) of each rectangle constituting the array. In the example of FIG. 2, the rectangle array is arranged on the wall 2010, and the position and orientation of the wall 2010 in the world coordinate system are defined by physical space information. Hence, the three-dimensional coordinate position of each rectangle on the wall 2010 can be calculated from the physical space information.

[0054] In step S204, the arrangement information generation unit 1030 subtracts a predetermined value from the value of the variable D, thereby decreasing the value of the variable D by the predetermined value.

[0055] In step S205, the arrangement information generation unit 1030 determines whether the value of the variable D is equal to or smaller than d2. If the arrangement information generation unit 1030 determines that the value of the variable D is equal to or smaller than d2, the process advances to step S209. If the arrangement information generation unit 1030 determines that the value of the variable D is larger than d2, the process advances to step S207.

[0056] In step S207, the arrangement information generation unit 1030 determines whether a predetermined number (for example, four) or more of rectangles fall within the field of view of the viewpoint in the currently set rectangle array when the viewpoint is set at a position spaced apart by the distance D from the portion at which indices are arranged. If the arrangement information generation unit 1030 determines that the predetermined number or more of rectangles fall within the field of view of the viewpoint, the process advances to step S204. If the arrangement information generation unit 1030 determines that the predetermined number or more of rectangles do not fall within the field of view of the viewpoint, as shown in FIG. 7 (reference numeral 7001 denotes the inside of the field of view of the viewpoint), the process advances to step S208.

[0057] In step S208, the arrangement information generation unit 1030 obtains the size S of the rectangle by calculating S=M×D, as in step S203 described above. By using the physical space information contained in the MR experience information, the arrangement information generation unit 1030 specifies a portion at which indices are arranged. The arrangement information generation unit 1030 arranges rectangles each having the size S side by side at this portion, instead of the previously arranged array, thereby arranging a rectangle array (for example, 11000 in FIG. 9) at this portion. The arrangement information generation unit 1030 registers the set rectangle array as index arrangement information in the data storage unit 1020.

[0058] In step S209, the arrangement information output unit 1040 reads out pieces of index arrangement information stored in the data storage unit 1020. By using the readout pieces of index arrangement information, the arrangement information output unit 1040 generates information representing the arrangement pattern of a plurality of indices in the physical space. As processing to be performed in this step, various processes are conceivable. An example of the processing will be explained below.

[0059] For descriptive convenience, a case in which information representing the arrangement pattern of a plurality of indices in the physical space is generated by using two types of index arrangement information will be explained. Even when three or more pieces of index arrangement information are used, the following processing is applied in the same way.

[0060] First, a composite arrangement pattern is generated by superimposing an arrangement pattern candidate represented by one index arrangement information, and an arrangement pattern candidate represented by the other index arrangement information. FIG. 10 shows an example of a composite arrangement pattern 12000 of the arrangement pattern candidate 10000 in FIG. 8 and the arrangement pattern candidate 11000 in FIG. 9. FIG. 11 shows an enlarged partial region 12100 in the composite arrangement pattern 12000.

[0061] When the composite arrangement pattern 12000 in FIG. 10 of the arrangement pattern candidate 10000 in FIG. 8 and the arrangement pattern candidate 11000 in FIG. 9 is created, a rectangle 13002 in the arrangement pattern candidate 10000 of FIG. 8 and a rectangle 13001 in the arrangement pattern candidate 11000 of FIG. 9 overlap each other as shown in in FIG. 11. Overlapping of rectangles (indices) can be determined by using pieces of index arrangement information corresponding to arrays to which the respective rectangles belong.

[0062] As processing to be performed when it is determined that rectangles overlap each other, various processes are conceivable. For example, when A% or more of the area of one rectangle (the rectangle 13001) is contained in the other rectangle (the rectangle 13002), as shown in FIG. 11, this rectangle may be deleted. When this processing is performed for respective rectangles on the composite arrangement pattern 12000 (respective rectangles of the arrangement pattern candidate 10000 in FIG. 8 and the arrangement pattern candidate 11000 in FIG. 9), the composite arrangement pattern 12000 is edited into a composite arrangement pattern 13100 as shown in FIG. 12.

[0063] When it is determined that rectangles overlap each other, for example, when A% or more of the area of one rectangle (a rectangle 14001) is contained in the other rectangle (a rectangle 14003), as shown in FIG. 13, this rectangle may be moved in a direction indicated by an arrow to a position 14002 so as to cancel overlapping of the rectangles.

[0064] When the wall 2010 has an outlet, a region where no index can be physically arranged, or a region 17000 where no index is to be arranged, as shown in FIG. 3, information representing this region may be contained in physical space information in advance. If a rectangle 17010 overlaps the position of the region 17000, the rectangle 17010 may be moved in a direction indicated by an arrow to a position 18000 so as to cancel overlapping of the rectangle.

[0065] It is also possible to contain, in advance in physical space information or MR experience information, a position where a virtual object is displayed, and move a rectangle so as to arrange it at the region position. In this fashion, when it is determined that rectangles overlap each other, the composite arrangement pattern may be edited appropriately. Depending on the application, the composite arrangement pattern may not be edited. Alternatively, the composite arrangement pattern may be displayed on the display unit 1080 to prompt the user to edit it. In this case, the user edits the composite arrangement pattern by operating the operation unit 1010 to move or delete one or more rectangles.

[0066] In step S210, the arrangement information output unit 1040 stores the edited/unedited composite arrangement pattern in the data storage unit 1020. The composite arrangement pattern may be stored in an arbitrary format in the data storage unit 1020. For example, the composite arrangement pattern may be stored in a digital document format such as pdf or xps, or an image format such as jpeg or bmp. Note that the composite arrangement pattern contains data which defines each rectangle contained in the composite arrangement pattern, such as data of the size or three-dimensional position of each rectangle contained in the composite arrangement pattern.

[0067] By executing the above-described processing according to the flowchart of FIG. 15, a composite arrangement pattern serving as information representing the arrangement pattern of a plurality of indices in a physical space can be registered in the data storage unit 1020.

[0068] The image output unit 1070 reads out the composite arrangement pattern registered in the data storage unit 1020, and displays it as an image or digital document on the display unit 1080. At this time, the image output unit 1070 may display, on the display unit 1080, even the three-dimensional coordinate position of each rectangle contained in the composite arrangement pattern. The contents displayed on the display unit 1080 are not limited to them, and an interface for experiencing mixed reality may be further displayed.

[0069] Note that the portion at which indices are arranged is not limited to the wall, and various portions in a physical space are conceivable. In the embodiment, the user mounts a head mounted display on his head in order to experience mixed reality. However, mixed reality can be experienced even by using an HHD (Hand Held Display), tablet, smartphone, or the like.

[0070] In the first embodiment, the display unit 1080 displays an image or digital document based on a composite arrangement pattern. However, the output destination of an image or digital document based on a composite arrangement pattern is not limited to the display unit 1080. The second embodiment will describe an information processing apparatus capable of causing a printing device to print an image or digital document based on a composite arrangement pattern.

[0071] An example of the functional arrangement of the information processing apparatus according to the second embodiment will be described with reference to the block diagram of FIG. 4. In FIG. 4, the same reference numerals as those in FIG. 1 denote the same functional units, and a description of these functional units is the same as the above one and will not be repeated. The arrangement shown in FIG. 4 is different from the arrangement in FIG. 1 in that a main body unit 1900 is configured by adding an arrangement information printing unit 18010 to a main body unit 1000 in FIG. 1, and a printing device 18020 is connected to the main body unit 1900. A difference from the first embodiment will be explained below, and the arrangement is the same as that in the first embodiment, unless otherwise specified below.

[0072] The printing device 18020 is connected to the main body unit 1900 directly or indirectly so as to be able to perform data communication with the main body unit 1900. The connection may be wired or wireless.

[0073] The arrangement information printing unit 18010 reads out a composite arrangement pattern registered in the data storage unit 1020, generates print data of the composite arrangement pattern, and sends the generated print data to the printing device 18020. The printing device 18020 prints an image, character, or the like on a printing medium such as paper in accordance with the print data.

[0074] In the first embodiment, the user inputs physical space information by operating the operation unit 1010. However, a three-dimensional measurement device or the like may be used to measure three-dimensional information of a physical space in an area observable by a user who experiences mixed reality, and the measurement result may be acquired as physical space information.

[0075] In the first embodiment, the user inputs physical space information by operating the operation unit 1010. After that, the user may appropriately edit the physical space information by operating the operation unit 1010. Depending on the application, the input physical space information may be converted into, for example, physical space information representing a more simplified physical space. That is, input physical space information may be directly used, or may be modified and used in subsequent processing.

[0076] Although the respective functional units constituting a main body unit 1000 shown in FIG. 1 or a main body unit 1900 shown in FIG. 4 may be constructed by hardware, the respective functional units except for a data storage unit 1020 may be constructed by software (computer programs). In the latter case, a computer which includes a memory functioning as the data storage unit 1020 and includes a processor capable of executing the computer programs can be applied to the main body unit 1000 shown in FIG. 1 or the main body unit 1900 shown in FIG. 4.

[0077] An example of the hardware arrangement of the computer applicable to the main body unit 1000 shown in FIG. 1 or the main body unit 1900 shown in FIG. 4 will be explained with reference to the block diagram of FIG. 14. Note that the arrangement shown in FIG. 14 is an example of the arrangement of this computer, and the present invention is not limited to this arrangement.

[0078] A CPU 15001 controls the operation of the overall computer by executing processing using a computer program and data stored in a RAM 15002 or ROM 15003. In addition, the CPU 15001 executes each processing described to be performed by the main body unit 1000 shown in FIG. 1 or the main body unit 1900 shown in FIG. 4.

[0079] The RAM 15002 has an area for storing computer programs and data loaded from an external storage device 15007 or storage medium drive 15008, and computer programs and data externally received via an I/F (interface) 15009. Further, the RAM 15002 has a work area used when the CPU 15001 executes various processes. That is, the RAM 15002 can properly provide various areas.

[0080] The ROM 15003 stores setting data of the computer, a boot program, and the like.

[0081] The external storage device 15007 is a mass information storage device typified by a hard disk drive. The external storage device 15007 saves an OS (Operating System), and a computer program and data for causing the CPU 15001 to execute each processing described to be performed by the main body unit 1000 shown in FIG. 1 or the main body unit 1900 shown in FIG. 4. The external storage device 15007 also saves information described as known information. The computer programs and data saved in the external storage device 15007 are appropriately loaded into the RAM 15002 under the control of the CPU 15001, and processed by the CPU 15001. Note that the RAM 15002 and external storage device 15007 function as the data storage unit 1020 in FIG. 1.

[0082] The storage medium drive 15008 reads out computer programs and data recorded on a storage medium such as a CD-ROM or DVD-ROM, and outputs them to the RAM 15002 or external storage device 15007. Some of the computer programs and data described to be saved in the external storage device 15007 may be recorded on this storage medium.

[0083] The I/F 15009 is constituted by one or more interfaces for connecting an external device to the computer. An operation unit 1010, display unit 1080, and printing device 18020 described above are connected to the I/F 15009. For example, the I/F 15009 is constituted by a digital input/output port such as a USB for connecting the printing device 18020, and a DVI or HDMI port for connecting the display unit 1080. The above-mentioned head mounted display may be connected to the I/F 15009. The respective units described above are connected to a bus 15010.

[0084] Embodiments can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions recorded on a storage medium (e.g., non-transitory computer-readable storage medium) to perform the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more of a central processing unit (CPU), micro processing unit (MPU), or other circuitry, and may include a network of separate computers or separate computer processors. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)TM), a flash memory device, a memory card, and the like.

[0085] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.