Extending program languages with source-program attribute tags

[0002] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the software and data as described below and in the drawing hereto: Copyright © 1998, Microsoft Corporation, All Rights Reserved.

[0003] The present invention involves electronic data processing, and more specifically concerns extensions to programming languages for providing dynamic open-ended modification of a language compiler under the control of source code during compilation of a program.

[0004] C++ and other general-purpose programming languages are commonly employed in writing low- to medium-level code for systems such as applications, operating systems, and utility programs. Many programmers rely heavily on libraries of classes specific to the domain of the particular program they are writing. A number of sources make these libraries available to developers. For example, Microsoft® Foundation Classes (MFC) is a library of C++ classes useful in developing applications for Microsoft Windows® Active Template Library (ATL) is a library of classes and templates useful in coding components according to the Component Object Model (COM).

[0005] Class libraries and similar tools help programmers in several ways. They provide a uniform model or framework for constructing a program. They hide a certain amount of complexity and detail behind simple, easy to use application program interfaces (APIs). For example, hosting an ActiveX control is a significant task for C++ programmers. Using the appropriate available classes reduces the total effort by allowing programmers to concentrate upon coding the actual functionality of the control.

[0006] Incorporating these tools into a program, however, requires additional code for fastening together the different classes or other functionalities and for integrating them into the overall program. This additional plumbing or glue code conforms to a general pattern that tends to be repeated many times in a single program. Because it resides in the source code of the program rather than in an external library, this verbose additional code adds complexity and reduces readability. Conventional approaches to hiding detail—macros, include statements, compiler directives, and so forth—have neither the power nor the flexibility to alleviate this problem to any significant degree.

[0007] A system of intentional programming, conceived by Charles Simonyi, provides an abstraction mechanism for hiding detail and for providing language features as independent components. This system, however, requires extensive modifications to presently available compilers, and is not dynamically reconfigurable for different applications. Another concept, aspect-oriented programming (AOP), uses self-contained subprograms to modify global properties of an application program. Producing programs in this paradigm requires an entirely new kind of compiler, called a “weaver.”

[0008] Therefore, a need remains for hiding the complexity that is introduced by the very tools whose purpose is to reduce complexity in writing programs for particular contexts or domains, and to do so without extensive modifications to conventional compilers.

[0009] The present invention introduces the concept of attributes that can be attached to programming constructs in source-code for hiding the glue code or any other operations required to employ certain constructs within programs for a specific type of applications. The glue-code or operations may relate to code added to the program at any point, either before or after the attribute is encountered, to modifications of code written by the programmer, to the manner in which certain constructs operate, and to other aspects of the program.

[0010] The invention can also be thought of as an instrumentality for extending a programming language in a way that is both powerful and flexible. Extensions for particular purposes become add-in or plug-in modules, easily inserted and easily removed. Different add-ins can be used in sets to provide customized sublanguages for particular problem domains.

[0011] When a program is compiled, a simple internal interface added to an otherwise conventional compiler calls an external attribute provider or handler for each of the attribute tags embedded in the source code. The attribute provider can then operate upon any of the code in the program in the compiler to provide the required operations.

[0012] Operations upon the code include adding additional code (glue code) to the program, modifying code in the program, deleting code from the program, and changing the manner in which certain constructs function. Operations can also include adding new base classes and switching existing base classes, adding new class members, adding new methods to a class, adding methods or code to existing methods, and adding new entries to a COM map, a MSG map, or a COLUMN map. Many other operations are also possible. The effects of these new attributes can occur at points within the program removed from the point at which the attribute tags occur. For example, an attribute can inject additional code lines at any point in a program source file, whereas a conventional macro can only expand a line of code at the point where the macro statement occurs.

[0013] Beyond providing a simple mechanism for hiding details with minimal compiler modifications, the invention offers easy reconfigurability and modifiability for compilers. Because the attribute providers are external to the compiler, they can be changed and upgraded separately, without requiring a new release, or even a new build, of the entire compiler. Vendors can supply different groups of attribute compilers for different purposes, such as for writing Microsoft Active® controls. Third parties can employ the interface to supply specialized or improved attribute providers for compilers sold by other manufacturers.

[0014] FIG. 1 is a block diagram of a typical environment for hosting the present invention.

[0015] FIGS. 2A and 2B show respectively a short section of code written in a conventional manner and employing attributes according to the invention.

[0016] FIG. 3, comprising FIGS. 3A, 3B, 3C, 3D, 3E, 3F, and 3G, is a longer example of code using attributes of the invention.

[0017] FIG. 4 is a block diagram showing the structure of a system for implementing the invention.

[0018] FIG. 5 is a flowchart showing a method for carrying out the invention.

[0019] The following detailed description of preferred embodiments refers to the accompanying drawings that form a part hereof, and shows by way of illustration specific embodiments of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. Structural, logical, and procedural modifications within the spirit and scope of the invention will occur to those in the art. The following description is therefore not to be taken in a limiting sense, and the scope of the inventions is defined only by the appended claims.

[0020] FIG. 1 provides a brief, general description of a suitable computing environment in which the invention may be implemented. The invention will hereinafter be described in the general context of computer-executable instructions such as program modules, executed by a personal computer (PC); however, other environments are possible. Program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Those skilled in the art will appreciate that the invention may be practiced with other computer-system configurations, including hand-held devices, multiprocessor systems, microprocessor-based programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0021] FIG. 1 shows an exemplary system for implementing the invention. It employs a general-purpose computing device in the form of a conventional personal computer 20, which includes processing unit 21, system memory 22, and system bus 23 that couples the system memory and other system components to processing unit 21. System bus 23 may be any of several types, including a memory bus or memory controller, a peripheral bus, and a local bus, and may use any of a variety of bus structures. System memory 22 includes read-only memory (ROM) 24 and random-access memory (RAM) 25. A basic input/output system (BIOS) 26, stored in ROM 24, contains the basic routines that transfer information between components of personal computer 20. BIOS 24 also contains start-up routines for the system. Personal computer 20 further includes hard disk drive 27 for reading from and writing to a hard disk (not shown), magnetic disk drive 28 for reading from and writing to a removable magnetic disk 29, and optical disk drive 30 for reading from and writing to a removable optical disk 31 such as a CD-ROM or other optical medium. Hard disk drive 27, magnetic disk drive 28, and optical disk drive 30 are connected to system bus 23 by a hard-disk drive interface 32, a magnetic-disk drive interface 33, and an optical-drive interface 34, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules and other data for personal computer 20. Although the exemplary environment described herein employs a hard disk, a removable magnetic disk 29 and a removable optical disk 31, those skilled in the art will appreciate that other types of computer-readable media which can store data accessible by a computer may also be used in the exemplary operating environment. Such media may include magnetic cassettes, flash-memory cards, digital versatile disks, Bernoulli cartridges, RAMs, ROMs, and the like.

[0022] Program modules may be stored on the hard disk, magnetic disk 29, optical disk 31, ROM 24 and RAM 25. Program modules may include operating system 35, one or more application programs 36, other program modules 37, and program data 38. A user may enter commands and information into personal computer 20 through input devices such as a keyboard 40 and a pointing device 42. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 21 through a serial-port interface 46 coupled to system bus 23; but they may be connected through other interfaces not shown in FIG. 1, such as a parallel port, a game port, or a universal serial bus (USB). A monitor 47 or other display device also connects to system bus 23 via an interface such as a video adapter 48. In addition to the monitor, personal computers typically include other peripheral output devices (not shown) such as speakers and printers.

[0023] Personal computer 20 may operate in a networked environment using logical connections to one or more remote computers such as remote computer 49. Remote computer 49 may be another personal computer, a server, a router, a network PC, a peer device, or other common network node. It typically includes many or all of the components described above in connection with personal computer 20; however, only a storage device 50 is illustrated in FIG. 1. The logical connections depicted in FIG. 1 include local-area network (LAN) 51 and a wide-area network (WAN) 52. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

[0024] When placed in a LAN networking environment, PC 20 connects to local network 51 through a network interface or adapter 53. When used in a WAN networking environment such as the Internet, PC 20 typically includes modem 54 or other means for establishing communications over network 52. Modem 54 may be internal or external to PC 20, and connects to system bus 23 via serial-port interface 46. In a networked environment, program modules depicted as residing within 20 or portions thereof may be stored in remote storage device 50. Of course, the network connections shown are illustrative, and other means of establishing a communications link between the computers may be substituted.

[0025] FIG. 2A is a short segment 210 Of a source-code program written in C++. The overall purpose of segment 210 is to define a function OnClick 201 that determines what action to perform when a user presses a mouse button in the program. Function 201 is defined to operate in the context of the Microsoft® Windows® operating system. Statement 211 declares a class in the proper context, and statement 212 supplies detailed glue code that has to be manually added by the programmer in conventional practice. Lines 213 define a block containing further glue code for handling a Windows message for the function. The detailed code required to patch the desired function code 201 into the proper setting takes up a large fraction of the total segment.

[0026] FIG. 2B shows a corresponding source-code segment 220 employing attributes according to the invention. First an attribute tag 221 specifies that the following block takes place in the Windows context, so that line 222 can refer to the required class without any additional particularization. That is, the code in the segment can be written entirely as though it were being written within the restricted environment of a normal Windows OS program, without having to add or modify any code in the block because the actual environment differs. Another attribute tag 223 applies to function 201, so that the function code can be written in an environment that knows about the relevant messages for a mouse click, without having to supply those details in segment 220. Thus, apart from the attribute tags, the entire segment relates directly to implementing the functionality of the defined function, without any detailed glue code for particularizing the code to its context or environment.

[0027] In the short example of FIG. 2, the glue code of segment 210 only needed to be written once. In a longer program, conventional glue code would be included repeatedly, sometimes at a great distance from the functional code that it influenced; macros would not ameliorate this situation significantly, because a programmer would still have to find individually all the source-program locations requiring the extra code, and then include a separate macro statement at each location.

[0028] An attribute tag can be attached to any programming construct, including an entire block or program. The following table lists some representative forms.

[0029] An example of attribute-tag syntax is shown below.

[0030] [

[0031] attr1,

[0032] attr2=val1,

[0033] attr3(val2),

[0034] attr4(prop1=val3, prop2=val4),

[0035] attr5(val5,val6,prop3=val7,prop4={val8, val9})={val10,val11}

[0036] ]

[0037] Within the paired brackets denoting an attribute tag, a first attribute has only a name. A second attribute is set equal to a specific value. A third attribute has a value passed to it as a parameter; this is equivalent to the second form. Another attribute contains a list of properties set to specific values. The last attribute illustrates a parameter list containing values, properties set to individual values, and a property set to a list of values. Finally, a parameterized attribute itself is set to a list of values. Commas separate multiple attributes in a single tag.

[0038] FIG. 3 is a longer example 300 of C++ code written with attributes. This listing shows the original source code and bracketed attribute declarations in boldface. The lines in lighter type are “injected code” resulting from the attributes. The injected code is normally located remotely from the point at which the attribute declaration appears in the source program. Code can be injected at multiple locations as well. Where the injected code is long or distant from the location of the attribute declarations in FIG. 3, it is surrounded by comment lines that identify the attribute for which it was injected. The scope of an attribute is not bound to the scope of its associated construct (variable, class, etc.); it can extend from the point of its use to the end of a compilation unit. As a practical matter, however, most attribute-provider writers and application-program programmers only deal with an attribute in the context of its associated variable or declaration. In FIG. 3, the module attribute in line 3001 operates over the entire program, the datafield(1) attribute in line 3045 is coextensive with the scope of variable m_nID, and command_handler(IDC_DIALOG1) in line 3083 operates over the block in lines 3079-3105.

[0039] FIG. 4 is a block diagram of a system 400 for processing a source-code program, such as 220, having a sequence of statements with embedded attribute tags. Compiler 410 has an overall structure of a conventional type in which a front end 411 reads source program 220 written in a particular source language such as C++. Conventional block 412 converts the source program into a parse tree 414 in an intermediate or tokenized language. Basically, a compiler reads and translates a sequence of elements in the statements or commands of a source program. In program 220, for example, the compiler recognizes [window] as an attribute-tag element to be processed as a unit. The outer brackets {.} denote a block element. Within that element is a function-declaration language-construct element void OnClick( ), and so forth. A symbol table 415 lists all the symbol names used in the program, and their characteristics. Conventional states 413 keep track of where converter 412 is in the conversion of the program and the parse tree. For example, different states indicate that the compiler has seen an attribute, is at the start of a class definition or a function, has just declared a class member, or has completed an expression. As the compiler progresses, it continually updates its internal state 413. For the present purposes, a state can be considered to be any point at which the compiler is willing to accept external inputs; individual compilers might differ as to which states they are willing to expose to an outside program such as the present attribute providers.

[0040] Compiler back end 416 translates the intermediate program in 414 and 415 into an object-code program 420 by converting its tokenized code into instructions executable in a particular real or virtual processor, into memory allocations for variables, and so on. Front-end 411 and back-end 416 sections of modern compilers perform many additional functions, such as code optimization, that do not concern the present invention and can be implemented in any known manner. The purpose of two-stage compilers with separate front and back phases joined by a common intermediate language is to allow the marketing of compilers for M different source languages and N different processors by developing only M+N versions, rather than M·N different variations. The present invention can easily be implemented to a single-phase or other type of known compiler architecture.

[0041] A set of attribute providers 430 provide the functionality for processing source program 220 in accordance with the attribute tags. For example, attribute provider 431 can be an executable program performing operations associated with the [window] attribute 221. Providers 432 and 433 handle other attributes. Attribute providers 430 are completely external to compiler 410. They are stored separately in a memory such as RAM 22 or disk 32, FIG. 1. They can be added or deleted from system 400 separately from the compiler itself, and separately from each other. In fact, different providers or groups of providers can be sold or downloaded in order to supply different language extensions for different problem domains, such as writing ActiveX controls or COM+ objects. The providers themselves can take the form of d11 or exe modules, and/or data or other files. In fact, if different compilers support the same interfaces, the same attribute provider can be used with multiple compilers, thus saving even more time and effort.

[0042] An individual provider such as 433 can be conceptually divided into several functional parts. An input module 434 receives a particular attribute from a tag and parses it for any parameters, values, properties, or other specifications. The compiler need not be written in any special way, as long as the compiler and attribute provider agree on the interface they will use to communicate. For instance, the compiler states that it will call methods M1, M2, and M3 and provide data in the form D1 and D2., while the attribute provider states that it will call methods M4 and M5, passing data in the form D3.

[0043] Operation module 435 determines which modifications must be made in order to implement the attribute in the source program, and identifies locations in the source program where code such as 213, FIG. 2A, is to be injected, or where other operations are to be carried out. Injected code usually comprises statements or other language elements added to the program at one or more locations. However, this term also includes existing source code that is changed or deleted, and other modifications. For example, attributes of COM+ or MIDL objects do not inject code, but rather effect the creation of meta-data that is not directly related to the compilation process. Injected code can be stored in the provider as templates 436 or in some other internal form. Output module 437 communicates the changes back to the compiler. There are many way that these changes can be actually effected in the application program. The attribute provider might write injected code to an external source file (not shown) that the compiler then consumes. The attribute provider might send the code directly to the compiler as a stream of bytes that the compiler processes. Another alternative is to have the attribute provider directly manipulate internal compiler structures, creating the necessary symbols, parse-trees, and other entities on the fly.

[0044] Interface 440 added to front end 411 ties attribute providers 330 into otherwise conventional compiler 310. As front end 411 encounters attribute tags in the input stream of source program 220, it changes its state designation 413 and saves the tags on an attribute list 441. This list also identifies the addresses or other locations of each attribute provider 431-433. In the environment of the Microsoft Windows95® operating system, the interface acquires the attribute-provider locations from the registry database. Interface 440 then continually communicates the compiler state 413 to all active providers 430. When any of them detects a point at which it desires to inject code or to perform any other operation, it signals converter 412 via interface 440. When a provider is called, it operates upon the application program in one of the ways mentioned above. Attribute providers 430 can effect their operations directly upon any file or structure in the compiler or outside of it. However, having an attribute provider instruct the compiler (e.g., converter 412) to perform the operations gives a measure of security; the compiler can reject or modify any request that would leave it in a bad internal state 413 or would otherwise compromise its proper functioning.

[0045] Interface 440 also produces a change log 442 containing a listing of all provider calls and arguments passed to the compiler. Internal compiler components 440 thus allow external attribute providers to reach into the internal states or structure of the compiler. An attribute in a source program can be analogized to a function whose arguments are the current and future states of the compiler. Furthermore, this approach far exceeds the functionality of a conventional preprocessor, which merely analyzes tokens without any knowledge of the higher structure of a program.

[0046] FIG. 5 shows a method 500 for using attributes according to the invention. In block 510, a programmer constructs a source-code program 220 such as shown in the boldface lines of source listing 300, FIG. 3. The program is a sequence of statements having syntactic elements in a particular programming language such as C++, and also includes attribute tags as described above.

[0047] Block 520 loads a C++ compiler modified according to the invention. Blocks 530 separately load a number of attribute providers under the control of a user, another program, or in any other manner. It should be especially noted that the providers can be loaded at any time grelative to the loading of the compiler or of any other component. It is even possible for the compiler to call them from storage or from a remote server as the associated attribute tags are encountered in a source program. Blocks 530 can also register the attribute-provider program names and locations in a database such as the registry of the Microsoft Windows operating system. Block 540 reads the source program into the compiler front end 411, FIG. 4.

[0048] Blocks 550 convert or translate the statements of source program 220 in compiler 410. Block 551 detects each syntactic element of the source program. Block 552 sets an appropriate compiler state for that element. If block 553 determines that the current element is a conventional construct in the chosen language, then block 554 converts that element into a token in the intermediate language of the compiler. Block 555 places the token at an appropriate node of parse tree 412 if the element is to produce object-code instructions, or produces an entry in symbol table 413 if the element is declarative in nature. These and possibly additional functions represent conventional compiler operations.

[0049] However, if block 553 determines that the current element is an attribute having the syntax of an attribute tag, then block 556 asks whether that particular attribute name is already in attribute list 442, FIG. 4. If not, block 557 creates a new list entry for it, and block 558 finds the location or other identifier by which the external attribute-provider program can be accessed. Finally, block 559 calls the provider, transmitting any parameters or other data accompanying the name of the attribute in the tag construct. Control then returns to block 551 for processing the next syntactic element. The steps need not be carried out in any particular order. For instance, the compiler can find and list the tags during a conventional parse phase, and then process them during a later semantic-analysis phase.

[0050] Blocks 560 describe the action of an attribute provider called by block 559. If more than one attribute provider is active at any time during compilation, then a separate copy of blocks 560 executes concurrently for each one. Further, each provider is reentrant, because the same attribute can be called with different parameters, etc., within the same scope. Block 561 receives the attribute data from block 559, and parses any parameters or other data that affect the operation of the provider.

[0051] Block 562 detects the occurrence of designated events within compiler blocks 550. In this embodiment, the events are the states 413, FIG. 4, that are set in block 552. Dashed line 563 indicates this communication between compiler and providers. As mentioned above, a compiler writer can decide which internal states of the compiler to expose to the providers as “stable states” where the compiler is willing to accept inputs from the providers. Block 562 examines a currently active attribute and the current compiler state to determine whether to do nothing or to initiate a specific one of the operations 565. For example, while processing a class, the compiler encounters an attribute tag [foo] requiring the introduction of a global function. The appropriate attribute provider is made aware of this tag's occurrence, but the compiler is at that moment building a class definition. The provider therefore waits until the compiler reaches another state, indicating a global scope. At that later point, block 562 senses the appropriate state, and a block 565 requests the compiler to add the particular function. One of the events detected in block 562 can signify the end of an attribute scope in the source-code program. In that case, line 564 exits the provider program. A subsequent instance of the same attribute, of course, will reexecute the provider.

[0052] Multiple blocks 565 represent different operations that the attribute provider performs for different events that might occur within the compiler, and for different parameters transmitted with the attribute. Among these operations are injection of statements or other program elements, possibly employing templates such as 436, FIG. 4, and modifying or deleting code. Other operations include adding new classes, methods and variables, or modifying existing ones. Modification can include renaming and extending an object or construct. The only practical limitation on the form of operation concerns how much of its internal state the compiler can or desires to expose via interface 440, FIG. 4. That is, the compiler should still be able to protect itself against corruption by an attribute provider, and simplification of the interface might dictate omission of some capabilities. Dashed lines 566 represent the passing of code, state information, instructions, or other data back to the compiler for modifying the input program as described above. Control then returns on line 567 to block 562 for the detection of further events, until exit event 564 occurs.

[0053] When the source program 220 has been completely converted, block 570 produces log 442 if desired. Blocks 580 complete the transformation of the tokenized intermediate form in blocks 412 and 413, FIG. 4, into object-code program 420. Block 581 signifies the conversion to machine-language instructions, memory allocations, and other data required for the target real or virtual processor. Block 582 outputs the object code in whatever form is desired, such as a binary file or a printed dump. Again, method 500 supposes a two-stage compiler, but any compiler architecture or organization can be employed instead.

[0054] The foregoing illustrates the structures and methods for practicing the invention in a form presently preferred by the inventors. Other embodiments also fall within the scope of the appended claims. In particular, conventional means other than those specifically shown can be employed, and the orders of operations can be carried out in different temporal sequences.