Melody Composer and Arranger

This invention generally relates to improvements in computer based multimedia systems and more particularly to a system and method for automatically creating music.

Automatic creation of music by a computer is a brand new field that has only recently come of age. The popular "Band in the Box" by PG Music is an example of computer based music generation directed to the generation of a musical accompaniment (without melody) from the knowledge of a song's chord structure. U.S. Pat. No. 4,399,731 discloses a method and system for generating simple melodies and rhythms for music education. The computer selects notes and rhythms randomly but constrained by specific musical rules to provide some degree of music. This technique is called algorithmic composition and is effective in creating very "novel" music due to a high degree of randomness.

U.S. Pat. No. 4,483,230 discloses a method for generating simple musical melodies for use as an alarm in a watch. The melody is initially defined by a user's control of musical pitch by varying the amount of light reaching the watch. The melody is saved in the watch's memory for subsequent playback as an alarm. The patent requires human intervention for defining a melody.

U.S. Pat. No. 4,708,046 discloses a method for generating simple musical accompaniments for use in an electronic musical keyboard. The accompaniment is derived from pre-stored forms with a degree of randomness that is triggered by the performer's selection of bass notes. The lowest pitch determines the key of the accompaniment and the selection of notes determines the chordal structure. The randomness allows the arrangement to have some variation in playback. Thus, this patent only provides an accompaniment to a person's performance on a musical keyboard.

Accordingly, it is a primary object of the present invention to provide a system and method for automatically generating an entire musical arrangement including melody and accompaniment on a computer.

These and other objects of the present invention are accomplished by combining predetermined, short musical phrases modified by selection of random parameters to produce a data stream that can be used to drive, for example, a synthesizer and generate music.

FIG. 1a is a block diagram of a personal computer system in accordance with the subject invention;

FIG. 1b is a block diagram of an audio capture and playback apparatus in accordance with the subject invention;

FIG. 2 illustrates a MIDI note generation process in accordance with the subject invention;

FIG. 3 is a data structure in accordance with the subject invention;

FIG. 4 is a flowchart of the music generation logic in accordance with the subject invention; and

FIG. 5 is a flowchart of the music generation logic in accordance with the subject invention.

The invention is preferably practiced in a representative hardware environment as depicted in FIG. 1a, which illustrates a typical hardware configuration of a workstation in accordance with the subject invention having a central processing unit 1, such as a conventional microprocessor, and a number of other units interconnected via a system bus 2. The workstation shown in FIG. 1a includes a Random Access Memory (RAM) 4, Read Only Memory (ROM) 6, an I/O adapter 8 for connecting peripheral devices such as disk units 9 or a MIDI synthesizer to the bus, a user interface adapter 11 for connecting a keyboard 14, a mouse 15, a speaker 17 and/or other user interface devices to the bus, a communication adapter 10 for connecting the workstation to a data processing network or an external music synthesizer and a display adapter 12 for connecting the bus to a display device 13.

Sound processing must be done on an auxiliary processor. A likely choice for this task is to use a Digital Signal Processor (DSP) in the audio subsystem of the computer as set forth in FIG. 1b. The figure includes some of the Technical Information that accompanies the M-Audio Capture and Playback Adapter announced and shipped on Sep. 18, 1990 by IBM. Our invention enhances the original audio capability that accompanied the card.

Referring to FIG. 1b, the I/O Bus 19 is a Micro Channel or PC I/O bus which allows the audio subsystem to communicate to a PS/2 or other PC computer. Using the I/O bus, the host computer passes information to the audio subsystem employing a command register 20, status register 30, address high byte counter 40, address low byte counter 50, data high byte bidirectional latch 60, and a data low byte bidirectional latch 70.

The host command and host status registers are used by the host to issue commands and monitor the status of the audio subsystem. The address and data latches are used by the host to access the shared memory 80 which is an 8K.times.16 bit fast static RAM on the audio subsystem. The shared memory 80 is the means for communication between the host (personal computer/PS/2) and the Digital Signal Processor (DSP) 90. This memory is shared in the sense that both the host computer and the DSP 90 can access it.

A memory arbiter, part of the control logic 100, prevents the host and the DSP from accessing the memory at the same time. The shared memory 80 can be divided so that part of the information is logic used to control the DSP 90. The DSP 90 has its own control registers 110 and status registers 120 for issuing commands and monitoring the status of other parts of the audio subsystem.

The audio subsystem contains another block of RAM referred to as the sample memory 130. The sample memory 130 is 2K.times.16 bits static RAM which the DSP uses for outgoing sample signals to be played and incoming sample signals of digitized audio for transfer to the host computer for storage. The Digital to Analog Converter (DAC) 140 and the Analog to Digital Converter (ADC) 150 are interfaces between the digital world of the host computer and the audio subsystem and the analog world of sound. The DAC 140 gets digital samples from the sample memory 130, converts these samples to analog signals, and gives these signals to the analog output section 160. The analog output section 160 conditions and sends the signals to the output connectors for transmission via speakers or headsets to the ears of a listener. The DAC 140 is multiplexed to give continuous operations to both outputs.

The ADC 150 is the counterpart of the DAC 140. The ADC 150 gets analog signals from the analog input section (which received these signals from the input connectors (microphone, stereo player, mixer. . . )), converts these analog signals to digital samples, and stores them in the sample memory 130. The control logic 100 is a block of logic which among other tasks issues interrupts to the host computer after a DSP interrupt request, controls the input selection switch, and issues read, write, and enable strobes to the various latches and the Sample and Shared Memory.

For an overview of what the audio subsystem is doing, let's consider how an analog signal is sampled and stored. The host computer informs the DSP 90 through the I/O Bus 19 that the audio adapter should digitize an analog signal. The DSP 90 uses its control registers 110 to enable the ADC 150. The ADC 150 digitizes the incoming signal and places the samples in the sample memory 130. The DSP 90 gets the samples from the sample memory 130 and transfers them to the shared memory 80. The DSP 90 then informs the host computer via the I/O bus 19 that digital samples are ready for the host to read. The host gets these samples over the I/O bus 19 and stores them it the host computer RAM or disk.

Many other events are occurring behind the scenes. The control logic 100 prevents the host computer and the DSP 90 from accessing the shared memory 80 at the same time. The control logic 100 also prevents the DSP 90 and the DAC 140 from accessing the sample memory 130 at the same time, controls the sampling of the analog signal, and performs other functions. The scenario described above is a continuous operation. While the host computer is reading digital samples from the shared memory 80, the DAC 140 is putting new data in the sample memory 130, and the DSP 90 is transferring data from the sample memory 130 to the shared memory 80.

Playing back the digitized audio works in generally the same way. The host computer informs the DSP 90 that the audio subsystem should play back digitized data. In the subject invention, the host computer gets code for controlling the DSP 90 and digital audio samples from its memory or disk and transfers them to the shared memory 80 through the I/O bus 19. The DSP 90, under the control of the code, takes the samples, converts the samples to integer representations of logarithmically scaled values under the control of the code, and places them in the sample memory 130. The DSP 90 then activates the DAC 140 which converts the digitized samples into audio signals. The audio play circuitry conditions the audio signals and places them on the output connectors. The playing back is also a continuous operation.

During continuous record and playback, while the DAC 140 and ADC 150 are both operating, the DSP 90 transfers samples back and forth between sample and shared memory, and the host computer transfers samples back and forth over the I/O bus 19. Thus, the audio subsystem has the ability to play and record different sounds simultaneously. The reason that the host computer cannot access the sample memory 130 directly, rather than having the DSP 90 transfer the digitized data, is that the DSP 90 is processing the data before storing it in the sample memory 130. One aspect of the DSP processing is to convert the linear, integer representations of the sound information into logarithmically scaled, integer representation of the sound information for input to the DAC 140 for conversion into a true analog sound signal.

The invention is a method and system for a computer based multimedia system. Music must be available in various styles to satisfy the tastes of a targeted audience. For example, a kiosk in a business mall may use a multimedia system to advertise specific products and need background music as part of the presentation. Thus, an invention, such as ours, which provides a generalized approach to creating original music in a computer has broad appeal.

A computer based multimedia music system may be realized in waveform and Music Instrument Digital Interface (MIDI) form. Waveform is an audio sampling process whereby analog audio is converted into a digital representation that is stored within a computer memory or disk. For playback, the digital data is converted back into an analog audio form that is a close representation of the original signal. Waveform requires a large amount of information to accurately represent audio which makes it a less efficient medium for a computer to employ for the creation of original music.

MIDI is a music encoding process that conforms to a widely accepted standard. MIDI data represents musical events such as the occurrence of a specific musical note realized by a specific musical sound (e.g. piano, horn or drum). The MIDI data is transformed into an audio signal via a MIDI controlled synthesizer located internally in the computer or externally connected via a communication link. MIDI data is very compact and easily modified. Thus, MIDI data is employed by the subject invention.

The invention performs a random selection and manipulation of shoft, musical phrases that are processed to generate a specific MIDI sequence that is input to a MIDI synthesizer and output to an audio speaker. Since the music is randomly generated, there is no correlation to existing music and each composition is unique. By employing appropriate musical structure constraints, the resulting music appears as a cohesive composition rather than a series of random audio tones.

A work of music created by the subject invention is divided into the following characteristic parameters. Voicing refers to the selection of musical sounds for an arrangement. Style refers to the form of a musical arrangement. Melody refers to a sequence of musical notes representing a theme of the arrangement. Tempo refers to a rate of playback of an arrangement. Key refers to the overall pitch of an arrangement.

Another list of parameters govern the generation of the MIDI data input to a MIDI synthesizer as set forth in FIG. 2. Voice.sub.-- Lead 200 is a random selection of MIDI data representative of a melody voice selection (e.g. piano, electric piano or strings) that is used to control the synthesizer realization of the lead melody instrument.

Voice.sub.-- Second 204 is a random selection of MIDI data representing the melody voice selection (e.g. piano, electric piano, horn or flute) that is used to control the synthesizer realization of the secondary melody instrument. Voice.sub.-- Second 204 must be different from Voice.sub.-- Lead 200.

Voice.sub.-- Accompaniment 210 is a random selection of MIDI data representing the accompaniment voice selection (e.g. piano, electric piano, strings) that is used to control the synthesizer realization of the accompaniment instrument. Voice.sub.-- Accompaniment 210 must be different from Voice.sub.-- Lead 200 or Voice.sub.-- Second 204.

Voice.sub.-- Bass 220 is a random selection of MIDI data representing the bass voice selection (e.g. acoustic bass, electric bass or fretless bass) that is used to control the synthesizer realization of the bass instrument. Style.sub.-- Type 240 is a random selection of musical style types (e.g. country, light rock or latin). This selection strongly affects the perception of the realized music and may be limited to match the tastes of the targeted audience. Style.sub.-- Type 240 affects the generation of MIDI note data for all instrument realizations. Style.sub.-- Form 241 is a random selection of musical forms (e.g. ABA, ABAB, ABAC; major key or minor key) that determine the overall structure of the composition. For example, the element "A" may represent the primary melody as played by the Lead Voice, "B" a chorus as played by the Secondary Voice, and "C" an ending as played by both the Lead and the Secondary Voices. Style.sub.-- Form 241 affects the generation of MIDI note data for all instrument realizations.

Melody.sub.-- Segment 205 is a random selection of MIDI note data representing the principal notes of an arrangement. Multiple Melody.sub.-- Segments are used in sequence to produce an arrangement. Tempo.sub.-- Rate 260 is a random selection of MIDI data representing the tempo of an arrangement (e.g. 60 beats per minute) that is used to control the rate at which the MIDI data is sent to the synthesizer. Note.sub.-- Transpose 230 is a random selection of a number used to offset all MIDI note data sent to the synthesizer to raise or lower the overall musical key (i.e. pitch) of the composition.

The invention flow is provided via FIG. 2 and executes as follows. All random parameters are selected for a given arrangement using a random number generator. Then, a MIDI voice selection data is generated to initialize the MIDI synthesizer with the appropriate voices for the realization of lead melody instruments, secondary melody instruments, accompaniment instruments, bass instruments and percussion instruments. The Lead and Secondary Instrument's MIDI data is generated from a selected sequence of Melody.sub.-- Segment MIDI note data 205 modified with the selected Style.sub.-- Type 240 and Style.sub.-- Form 241. The Bass 220, Accompaniment 210 and Percussion Instrument's MIDI data is generated from the selected Style.sub.-- Type 240 and Style.sub.-- Form 241. Then, the MIDI note data for all voices except percussion is modified by Note.sub.-- Transpose 230 to select the desired musical key and is transmitted to the MIDI synthesizer at the Tempo.sub.-- Rate 260 to realize the music.

The heart of the invention is the data structure set forth in FIG. 3. The compositional.sub.-- selection 300 stores the type of composition the particular information in the data structure refers to whether it be voice, rhythm or chords. If the particular selection is voice, then the voice.sub.-- matrix 310 will preserve the particular type of instrument used for voice in the musical composition. If the particular selection is rhythm, then rhythm.sub.-- matrix 320 will save the style and tempo of the musical composition. Finally, if the particular selection is chords, then chordal.sub.-- matrix 360 will keep the chord structure of the musical composition.

Regardless of the compositional selection, the following information is also obtained for a particular composition. Melodic.sub.-- Matrix 360 stores the musical half tones of a unit of music in the composition. Midi.sub.-- data 350 selects the instrument voice. Midi.sub.-- data 340 selects the musical note of the composition. The use of the data structure is illustrated in the flow charts which appear in FIGS. 4 and 5.

FIGS. 4 and 5 are flow charts of the detailed logic in accordance with the subject invention. Function block 400 performs initialization of the musical composition at system startup. A user is queried to determine the paricular musical requirements that are necessary. Normal processing commences at decision block 410 where a test is performed to determine if any MIDI data is ready to be transmitted. The MIDI data resides in SONG--BUFFER and is sent to a music synthesizer based on performance timing parameters stored in the system data structure. If there is data, then it is transmitted in function block 420 to a MIDI synthesizer.

A second test is performed at decision block 430 to determine if the song buffer is almost empty. If the buffer is not empty, then control passes to FIG. 5 at label 585. If it is, the a random seed is generated at function block 440 to assure that each musical composition is unique. Then, function block 450 randomly selects the lead melody instrument sound, the MIDI data corresponding to the lead melody instrument is loaded into the song buffer at function block 460 and the synthesized instrument sound for the second melody is selected at function block 470. A third test is performed at decision block 480 to insure that a different synthesized instrument is selected for the second melody part. If the same instrument was selected, then control branches back to function block 470 to select another instrument. If not, then control passes via label 490 to FIG. 5.

FIG. 5 processing commences with function block 500 where the MIDI data corresponding to the second melody part is loaded into the song buffer and the synthesized instrument sound for the accompaniment is selected at function block 510. Then a fourth test is performed at decision block 520 to assure that a different synthesized sound is selected for accompaniment. If not, then control passes to function block 510 to select another instrument for accompaniment. If a different instrument was selected, then at function block 530 the MIDI data to select the accompaniment music is loaded into the song buffer. At function block 540, the bass instrument is selected and its corresponding MIDI information is loaded into the song buffer at function block 550. Then, a specific style, form and tempo for a composition are selected at function block 560; a specific transpose and melody pattern are selected in function block 570 and finally, at function block 580, MIDI data to play the arrangement is loaded into the song buffer.

The following pseudo code illustrates the algorithmic technique for creating electronic music in computer-based multimedia systems.

While the invention has been described in terms of a preferred embodiment in a specific system environment, those skilled in the art recognize that the invention can be practiced, with modification, in other and different hardware and software environments within the spirit and scope of the appended claims.