Method of Measuring the Types of Motion and Configuration of Biological and Non-Biological Objects

The present invention relates to a method of measuring the types of motion and the configuration of biological and non-biological objects, particularly their motility, position in space, distancing behavior, reversal of direction and shoaling behavior, with the objects being observed and the observations being evaluated, and to uses of such a method.

The known measuring methods do not permit the detection of such behavior in a single measuring process. The prior art methods of measuring individual behavior are insufficient and expensive. An abstract and bibliography of such methods are provided in: Spieser, "Methoden der Toxizitatsprufung an Fischen"[Methods of Testing Toxicity in Fish], published by DFG, Boldt-Verlag, Boppard, 1980. The different methods referred to in this publication possess a variety of drawbacks: either the fish are impaired (e.g. by implantation of magnets); or the measuring system is too inaccurate or very complicated, and includes many electric eyes tending to blind the animals, or the method is too inaccurate, as in the optical/optoelectronic method disclosed by Petry, Motilimat GETRA.RTM., or is unreliable, as in the case of measurement with standing ultrasound waves, (biopulse.RTM.), or requires too much personnel, i.e. observers who operate time sum/frequency counters. In all of these cases, not much more than motility can be measured and this only with errors. Components of shoaling behavior can be measured only in a complicated strip cylinder apparatus or, for individual fish, in a Nakamura apparatus. Both are not practicable. Further, there exists no measuring process for fright reactions.

It is an object of the present invention to provide a sensitive and practicable testing method permitting, for example, environmental toxicology examinations of biological objects subjected to small doses which do not influence the objects, and also being used as a monitoring test.

A method for measuring the type of motion and configuration of objects by observation of the objects and evaluation of the observation results, including the steps of: producing data representative of an image of the objects in a selected plane at successive time intervals; deriving, from the data representations for each time interval a representation of only one pair of coordinate values in the plane for each object; producing, from the coordinate values, representations of motion paths for the objects during a plurality of the time intervals; and producing, from the motion path representations, representations of the path traversed by each object, and the average height position, the period of dwell and number of reversals of direction of motion of each object during the plurality of time intervals.

The simultaneous surveillance of several test objects in the same environment, e.g. an aquarium, has in the past not been possible, or would at most have been possible at considerable expense by a complete analysis of continuous images by means of a large computer. But this is impractical for economic reasons. Only the solution provided by the present invention with data reduction (on-line generation of a single coordinate pair per object and image) in conjunction with detection of object sizes down to the size of a raster dot in a video raster and making the raster coarser in adaptation to the test objects for storage of the coordinates and subsequent reconstruction of the paths of movement from these data, eliminates these expenditures. For the detection of the distancing behavior of the objects, e.g. fish, the invention provides a novel calculation system which takes into account the error produced by two-dimensional projection and which makes it easy to compare the measured results from objects of different groups, e.g. if a group is divided into a plurality of different sub-groups. The reaction to fright stimuli can here be made objective for the first time.

Thus, the present invention permits for the first time, economical and additionally simultaneous observation and documentation of a plurality of behavior patterns of many individual objects. It can be used in an economical manner in environmental toxicology examinations for long-term as well as short-term tests and around the clock for monitoring waters, etc.

FIG. 1 is a block diagram of an embodiment of a measuring device for carrying out the invention. FIGS. 2a-7d are tables setting forth measuring results obtained with the measuring device of FIG. 1.

Versions of the method capable of meeting various requirements will be described for individual applications. An extremely flexible, substantially freely programmable type of measuring device is shown in FIG. 1 to include at least one video camera 1, 1', 1" for each of a plurality of measuring chambers 2, 2', 2", an image processor 3 including object detection 4, signal processing 5, raster coarsening 6 and addressing 7, a data computer 8 including two drive devices and memories 14, a data display screen 9 equipped with a key field or terminal 16, an image monitor 10 and a printer 11. Alternately, use can be made of a plurality of cameras which can be interrogated in succession, automatically according to a programmed time schedule or manually. Embodiments including monitoring functions have no drive devices and are equipped with a substantially fixed program, very fast calculating system which makes its evaluations already during recording, and with an alarm system 12 or an interface 13 for appropriate regulating systems. For use in research, this high speed version can also be made to be freely programmable. The versions for use in "toxicological examinations of fish" additionally include a stimulus generator 15, 15', 15" for each measuring chamber, controlled selectively by the terminal 16 or by computer 8 for defined fright stimuli. The computer 8 can be an IBM or compatible microcomputer, e.g. TANDON at 30. The image-processor 3 can be built up on an extension card for this device.

In the method according to the invention, in contrast to all conventional image analysis methods, measurements are taken with the full picture, or frame, resolution by one or a plurality of video cameras 1 to 1", but the information content of the frames is reduced to the essential facts in the on-line process in that for each object to be detected per frame only one coordinate pair Xa, Ya; Xb, Yb; to Xn, Yn is stored. In most cases, the quantity of data is further reduced in that the frame raster 6 is made coarser by an integral multiple. Insofar as permitted by the problems to be solved, the quantity of data can be reduced even further by calling up a smaller number of frames per unit time, e.g. recording only every third or every fifth video frame.

The objects are generally detected on the basis of brightness contrast compared with their immediate environment. Objects which are brighter than some points in the image background are recognized while the background itself is not detected. Undesirable darker or brighter points and even points of the same brightness with vague outlines are not detected, depending on the setting of the device. The sensitivity of the device can be adjusted. Sharply contoured, stationary, undesirable objects may automatically be cut out by prior measurement. Usually, only a single coordinate point is cut out per object, with the remaining image surface covered by the respective object remaining sensitive. It is also possible to make additional eliminations or to cancel eliminations manually from terminal 16 by means of a cursor provided in monitor 10.

On monitor 10, the following remains visible during the measurement: identification of the sensitive, usually rectangular, picture section which assures precisely defined picture limitation, the continuous video picture to be examined, all eliminated coordinates, all objects presently "detected" by the device and the position of their coordinate points. The "true" video image may also be omitted or displayed on a separate monitor.

Fom the image data obtained from the detected objects, a subsequent computing process, or a parallel process in the high speed version, reconstructs the sequence of movement of the objects on the basis of detected characteristics of the moving objects, e.g. their location, their momentary direction of movement and their velocity. This simultaneously produces association of the coordinates Xn, Yn for the continuous images with the individual objects and also the elimination of additional undesirable objects, such as, for example, the reflections of the objects to be measured at the side walls of measuring chambers 2-2".

From the thus determined paths of movement, a field projection of the path traversed per unit time (here abbreviated as "motility"), the average height position and period of dwell and the number of reversals of direction of each object are determined. These, values are sorted according to the magnitude of the motility value which reveals more easily any possible correlations between the various measured values. The individual values can either be stored on a floppy disk together with their averages or can be printed out as a whole.

During the calculation, the coordinate sets Xn, Yn are "cleaned up", i.e. the coordinates of all objects detected to be "permanently present" are stored once more. This is done to avoid errors in the calculation of the configuration of the objects with respect to one another. In order to even further reduce the quantities of superfluous data, the program may be set up so that, if desired, not every frame but, for example, only every second frame or every tenth frame is stored a second time.

In a subsequent run routine with the thus reduced sets of data, the distancing behavior of the objects is calculated. A subprogram permits detection of possible grouping of objects, their number and average distances from the calculated centers of gravity of the groups and determination of the overall center of gravity of the objects.

To solve the problem produced by the fact that objects disposed precisely at the boundary between two sets of coordinates are associated once with the one and once with the other coordinate set, with a motility value quickly being calculated from this seeming movement (even higher motility values for stationary objects can be produced if an elongate object is disposed almost parallel to and at the borderline of a row of coordinates), a calculating mode has been developed which prevents the generation of such erroneous data.

Special data sets are provided to monitor the device. In these data sets, the type of movement of the objects are precisely given and they serve to check certain individual functions of the device.

It has been a difficult task to assure that the association of the objects with the coordinates Xn, Yn of the continuous images occurs with relative reliability, i.e. the frequency of objects being mixed up when they meet one another or when one crosses the other's path behind it is kept as low as possible. For this purpose, data sets have been generated in which encounters between two objects occur in different ways, e.g. one passes the other, or two generated objects encounter one another in the same swimming direction, with one swimming faster than the other. The swimming velocity and the duration of the encounters were varied. Similar data sets simulate encounters in opposite directions with identical motilities. There are also encounters at an angle of 90.degree. and at 45.degree., with the paths of the two objects crossing in such a manner that the coordinates of the two meeting objects meet at a point, i.e. one of the objects disappears for the duration of one or two frames.

Motility was checked with data sets in which the objects traversed defined paths per unit time. Other data sets exist which simulate and check turning movement Data from live fish were also used for this purpose.

A further possibility of checking the operation of the device is the so-called "view mode". In this mode, the data set of a preceding picture, or frame, is made visible on data screen 9 during a calculation in the form of association letters for each individual object. That means that it is possible to observe the movements of the objects and the association of the coordinates Xn, Yn with individual objects in slow motion directly on data screen 9. Thus it is possible to monitor functions of the device even if there is very complex movement, i.e. whole groups of moving objects. Such monitoring takes place during calculation of the data and involves only a very slight loss of time.

It is possible to provide a complete output of all coordinates and their manner of association. However, because of the large amount of time required for observation on screen 9 or the amount of material involved in a printout 11, this is done only for test purposes. Initially, all coordinates Xn, Yn existing in a frame and not yet associated are listed. Then appear the associated coordinates of the respectively preceding frame and of the new frame provided with the association letters assigned to each object during the entire calculation, all lined up in a row. Added to this is the momentary direction of movement expressed in the form of an angular value and the motility value calculated for each object from the path traversed between the preceding frame and the current frame. While the view mode shows only an approximation of the functions of the device, error functions can be here checked in every detail.

The calculation of the distancing behavior is made in a manner not known before. In contrast to conventional methods, in which usually the distances of the individual objects from a swarm center of gravity or the distances from to the respectively next object are examined, the distancing behavior is included in the calculation in a manner which appears to be more suitable: this procedure considers the fact that if only one camera 1, l' or 1" is employed, only the field projection of a shoal of fish is processed. The error created by shortened paths in the calculation of the distances from the projected image would become too large, i.e. data would be simulated which were not measured at all. However the decisive value for the distancing behavior in schools of fish is the immediate distance from the nearest neighbor. Therefore a field projection of these distances, is nevertheless used as the basis of the calculation process.

Computer 8 of the measuring device calculates four concentric squares around each fish (object) at a given mutual edge spacing. Then it counts separately for each fish the adjacent fishes which are now enclosed in each square. These data are arranged by size, frame for frame. For continuous fields, the numbers of neighbors are summed up, beginning always with the fish which has the most neighbors, then the fish which has the second most neighbors, etc. Identification of the fish is intentionally not made. It is known that, in a shoal of fish, those which are at the edge of the shoal continuously exchange their positions with fish at the interior of the shoal. However, it is an easily understood fact that fish in the middle of a school have more neighbors than those at the edge of a school.

By making the calculation with four square sizes, four averaged curves are produced over the entire measuring period which very well represent the distancing behavior of the fish. These curves can easily be compared with those of other groups of fish of any distribution. If several groups form in one aquarium while only one group forms in another aquarium holding the same number of fish, a comparison between these aquariums is not possible with conventional calculating modes. With the method according to the invention, however, it is very easy to make statistical comparisons of the curves.

A calculation of turning movement is made on the basis of the swimming direction. Selectively, one can sum up the times the 90.degree. limits or the 180.degree. limits are exceeded in the direction of swimming. If the fish are observed from the top by camera 1-1" it is recommended to select the 90.degree. mode, since a preferred swimming direction cannot be expected. If the fish are observed from the side, it is advisable to select the 180.degree. mode so that only movement in the horizontal plane is recorded since movement in the vertical plane represents another motion quality and it cannot be expected that the horizontal and vertical movements have the same distribution.

The, number of reversals of direction are put out separately for each individual fish and additionally an average is made for all fish, with of course the most important statistical values also being calculated.

Detection of turning movement can be observed in the "view mode" on data screen 9 during the calculation in that the letters representing the association of coordinates Xn, Yn with the individual fish are converted from upper case letters to lower case letters and vice versa with every turning movement. Additionally, special data sets have been created for the turning movement with which these individual functions of the device can also be checked.

Calculation of motility is made for every fish on the basis of the path traversed per unit time. The values are put out in such a manner that different measuring periods can also be compared with one another without conversion in that all values refer to one second, regardless of which frame frequency was selected for recording. The variations in the motility data for each individual fish shows whether a fish was uniformly active over the entire measuring period or whether it made larger pauses and had periods of greater activity in between. When the motility values of all fish are summed at the end of each measuring period (sample) the spread in values indicates the variability between the fish over the entire measuring period. That means that variations in activity during the measuring period are not incorporated in this value, only the differences between the motility values of the individual fishes per sample.

Data output can be effected selectively via data screen 9, printer 11 or to a diskette. Motility is illustrated numerically and graphically on data screen 9 and on printer 11. The curves representing distancing behavior may selectively illustrated graphically plus numerically or only numerically. In these graphs, the range of variation of the data is also indicated. Additionally, it is possible to compare the differences between the respective four curves identifying the distancing behavior in two of the measuring chambers 2-2". This is done once by subtraction of the four curve pairs and then--even with larger numbers of measuring chambers to be compared--by averaging and putting out the standard deviation of the averaged data. To save calculating time, it is possible to reduce the cleaned-up data sets for the calculation during the calculation of motility data in that, for example, only every tenth frame is recorded. The frequency of the frames to be recorded can be put in via the key field 16. If motility is not very high, a frame sequence reduced in this manner is fully sufficient to correctly display the distance relationships corresponding to actual conditions.

An additional input mode makes it possible to automatically calculate entire sequences of measurements. The data files are then called up successively, are evaluated and the calculated data are stored on a diskette. Later, these data can be observed on screen 10 or can be printed out. For versions operating with several cameras 1-1", programmable automatic read-out of the individual components at various times is provided where reference is always made to the fixed cut-out data for each camera 1-1" at the start of the measuring series.

For testing of chemicals, plant protection agents and drugs, the behavior of fish, water fleas, microorganisms such as protozoa (paramecium), and algae (euglena, phacus, oscillatoria, etc.) as well as of birds, rats and mice can be analyzed at relatively low cost.

The values measured for protozoa and lower plants are motility and phototaxis.

The values measured for water fleas are motility and swimming height.

The values measured for fish are: for every animal, motility, average swimming height, number of reversals of direction and sholing or schooling behavior as well as the time sequences for the changes of these measured values after application of a defined fright stimulus.

The motility and social behavior of rats and mice can be economically monitored in a screening test. The results are more informative with respect to "threshold effects" than the cruel regulation LD 50.

For birds, values of interest are, in addition to motility (also in a day-night rhythm), the association of the animals with one another, the frequency of their interaction and their respective locations.

The diurnal movement of the leaves of the scarlet runner (phaseolus) are observed in the lapsed time mode and analyzed. The behavior of untreated plants is compared with those subjected to noxa in an air-conditioned chamber or to whom noxa are applied through the soil or the water. A comparison between the treated and the untreated control group is always done after corresponding time intervals. Shifts in time of the endogenous diurnal periodicity, as they have been proven to exist as a result of damage by noxa, and changes in the motion amplitude serve as criteria for ecologically damaging biological effects. In the past, such examinations were very complicated. Each individual plant had to be connected by means of a thread with a cymograph. The graphs recorded by the cymograph had to be carefully measured by hand and evaluated. The present invention now permits observation of a larger number of plants simultaneously without any further costs for apparatus. Since many plants have a diurnal motion rhythm, this method can be used for many ground and air contaminant toxicological examinations. The values measured are here, inter alia, the amplitude of the motion of the leaf tips over time, measured upon the occurrence of the respective motion maxima and the duration of movement in both directions. The measurements can be made in continuous light as well as upon a change of illumination intensity since the measuring device has a very broad range of operation.

In the past, such monitoring methods have sometimes been used for fish with unsuitable test parameters. The method according to the invention here offers economical, practicable possibilities also for the use of other organisms. A comparison is made of the behavior of biological objects with desired values or with the behavior of a control group. If limit values for motion standards of test animals or plants are exceeded or not reached, an alarm signal is given. These monitoring methods can also be used for the early detection of damage to the environment.

An example for the use of the invention in animal breeding and medicine is the determination of the motility of spermatozoa and their density.

An example for the use in industry and physiology is the measurement of the flow behavior of particles in capillaries with selective detection of various directions of movement.

In behavioral research, the invention permits a description of the manner of movement of animals, their association with one another and identification of certain types of movement. Behavior in the biosphere, interactions between animals and diurnal processes can also be measured.

The following rules apply for the numeric tabulations shown in FIGS. 2-7:

The division of the measuring duration, e.g. 270 seconds, into measuring sections (samples) permits a glimpse of the reproducibility of the values.

The time interval from the beginning of the evaluation of a frame to the next frame was set at 0.1 s. Each sample is thus composed of 900 frames.

Nr.=the identification letter of the respective fish, valid only within the sample. The samples are identified by letters attached to the file name (E80A, E80B, E80C). Fish N in sample B is not identical with fish N in sample A.

SH=swimming height of the individual animal, averaged during the sample. The frame is subdivided into eight identical height zones. The lowermost zone has the number 8.

SHD=standard deviation of the swimming height during sample time, but expressed in a raster range of 32 height levels.

PPR=percentage of the presence of an object during the duration of a sample. During the time the paths of two objects intersect, position coordinates are generated only for one object.

TURN=number of changes in direction of swimming from the right to the left or from the left to the right for the respective animal during the duration of an individual measurement (sample).

MOTILITY=path traversed (field projection), expressed in field limits passed per second with the total image being rastered into 60.times.32 fields.

DERIVAT=a measure for changes in motility during the measuring period, with reference to the individual animal.

DIAGRAM=graphic illustration of motility.

TURNING=number of turns made by all animals (mean value) per second. To the right of it: its standard deviation.

Motility was averaged correspondingly. To the right of it: its standard deviation.

OFFSET=distance between a fish and the edge of the innermost concentric square calculated around it, expressed in coordinate field values for an image raster of 60.times.32 fields.

DISTANCE=distance between the concentric squares in the above sense.

PLANE 0=innermost one of the concentric squares.

PLANE 3=outermost one of the concentric squares.

PLANE 1,2=intermediate concentric squares.

AVG=average number of neighboring fish within the square.

SDEV=standard deviation for the above.

SEM=standard error of the mean.

DIFF.=difference between the respective values of the two curves to be compared.

FIGS. 2a to 2c show three successive behavior measurements of 90 seconds each for zebrafish (Brachydanio rerio Hamilton-Buchanan) in a relatively undisturbed state.

FIGS. 3a to 3c show three 90-second behavior measurements for the same fish after the addition of live water fleas, which they hunt. The greater motility and larger number of reversals of direction compared with the measurements in the rest state (FIGS. 2a to 2c) are notable.

FIGS. 4a to 4c show behavior measurements after the addition of more live water fleas. The fish are still similarly active as in the measurements according to FIGS. 3a to 3c. However, all of them are located closer to the bottom which can be read from the swimming heights (SH).

FIGS. 5a to 5d show the distancing behavior of the same fish in a relatively quiet, undisted state during the measurement of FIG. 2a.

FIGS. 6a to 6d show the distancing behavior of these fish after being fed twice with live water fleas during the measurement according to FIGS. 4. Within set limits, the fish have more neighbors compared to the conditions shown in FIGS. 5a to 5d. Thus the fish are present in greater density.

FIGS. 7a to 7d show a comparison of the curves of FIGS. 5a to 5d and 6a to 6d by averaging and computing the standard deviations and by subtraction.

The present disclosure relates to the subject matter disclosed in Federal Republic of Germany Application No. P 35 43 515.1 of Dec. 10th, 1985, the entire specification of which is incorporated herein by reference.

It will be understood that the above description of the present invention is susceptible to various modifications, changes and adaptations, and the same are intended to be comprehended within the meaning and range of equivalents of the apended claims.