DROWSINESS DETERMINATION DEVICE AND DROWSINESS DETERMINATION METHOD

[0001] The present disclosure relates to a drowsiness determination device and a drowsiness determination method.

[0002] There is a drowsiness determination device that determines whether or not a driver is drowsy.

[0003] As such a drowsiness determination device, for example, Patent Literature 1 discloses a device including a drowsiness sensor.

[0004] The drowsiness sensor senses drowsiness of the driver on the basis of heartbeat intervals obtained from the electrocardiographic waveform of the driver.

[0005] Patent Literature 1: JP 2019-13737 A

[0006] The heartbeat intervals are generally longer when the driver is drowsy than when the driver is not drowsy. However, even when the driver is not drowsy, the heartbeat intervals vary depending on the activity status of the autonomic nerve. For example, the driver relaxes more as a certain period elapses after the start of driving than when the driver starts driving the vehicle, so that the activity situation of the autonomic nerve changes, and the heartbeat interval may become longer even if the driver has not become drowsy.

[0007] In the device disclosed in Patent Literature 1, the drowsiness sensor senses the drowsiness of the driver only on the basis of the heartbeat intervals. Thus, there is a problem that the drowsiness sensor erroneously detects drowsiness when the driver is not drowsy.

[0008] The present disclosure has been made to solve the above problems, and an object of the present disclosure is to obtain a drowsiness determination device that can reduce false detection of drowsiness more than the device disclosed in Patent Literature 1.

[0009] A drowsiness determination device according to the present disclosure includes: an image data acquiring unit to acquire image data indicating a face image of a driver from a camera that captures a face of the driver; an alertness level estimating unit to estimate an alertness level indicating a degree of alertness of the driver on the basis of the image data acquired by the image data acquiring unit; a waveform information acquiring unit to acquire waveform information indicating an electrocardiographic waveform of the driver from a sensor that detects an electrocardiographic waveform of the driver; and a heartbeat feature amount calculating unit to calculate a feature amount of a heartbeat of the driver from the electrocardiographic waveform indicated by the waveform information acquired by the waveform information acquiring unit. Further, the drowsiness determination device further includes: a drowsiness presence-absence determination unit to determine presence or absence of drowsiness of the driver on the basis of the alertness level estimated by the alertness level estimating unit and the feature amount of the heartbeat calculated by the heartbeat feature amount calculating unit.

[0010] According to the present disclosure, it is possible to reduce false detection of drowsiness more than the device disclosed in Patent Literature 1.

[0021] Hereinafter, in order to describe the present disclosure in more detail, modes for carrying out the present disclosure will be described with reference to the accompanying drawings.

[0022] FIG. 1 is a configuration diagram illustrating a drowsiness determination device 3 according to a first embodiment.

[0023] FIG. 2 is a hardware configuration diagram illustrating hardware of the drowsiness determination device 3 according to the first embodiment.

[0024] In FIG. 1, a camera 1 is installed, for example, on the instrument panel of a vehicle, the windshield of the vehicle, or the ceiling of the vehicle.

[0025] The camera 1 is implemented by, for example, one or more visible light cameras, one or more infrared cameras, or a video camera. In a case where the camera 1 is implemented by an infrared camera, a light source that emits infrared rays for imaging may be provided in an area including the driver's face. The light source is implemented by, for example, a light emitting diode (LED).

[0026] The camera 1 captures the face of the driver and outputs image data indicating a face image of the driver to the drowsiness determination device 3.

[0027] A sensor 2 is implemented by, for example, an electrocardiographic sensor that detects an electrocardiographic waveform of the driver in a state of being in contact with the driver, or an electrocardiographic sensor that detects an electrocardiographic waveform of the driver in a state of not being in contact with the driver.

[0028] The sensor 2 outputs waveform information indicating an electrocardiographic waveform of the driver to the drowsiness determination device 3.

[0029] The sensor 2 is not limited to one implemented by an electrocardiographic sensor, and may be implemented by an infrared camera. In a case where the sensor 2 is implemented by an infrared camera, the sensor 2 detects an electrocardiographic waveform of the driver on the basis of luminance of a face surface that changes with a heartbeat of the driver.

[0030] The drowsiness determination device 3 includes an image data acquiring unit 11, an alertness level estimating unit 12, a waveform information acquiring unit 13, a heartbeat feature amount calculating unit 14, and a drowsiness presence-absence determination unit 15.

[0031] The drowsiness determination device 3 determines whether the driver is drowsy on the basis of the image data output from the camera 1 and the waveform information output from the sensor 2.

[0032] The image data acquiring unit 11 is implemented by, for example, an image data acquiring circuit 21 illustrated in FIG. 2.

[0033] The image data acquiring unit 11 acquires the image data indicating the face image of the driver from the camera 1.

[0034] The image data acquiring unit 11 outputs the image data to the alertness level estimating unit 12.

[0035] The alertness level estimating unit 12 is implemented by, for example, an alertness level estimating circuit 22 illustrated in FIG. 2.

[0036] The alertness level estimating unit 12 includes a facial feature amount calculating unit 12a and an alertness level estimation processing unit 12b.

[0037] The alertness level estimating unit 12 acquires the image data from the image data acquiring unit 11.

[0038] The alertness level estimating unit 12 estimates an alertness level indicating the degree of alertness of the driver on the basis of the image data. The alertness level is an apparent alertness level of the driver estimated from the face image of the driver.

[0039] The alertness level estimating unit 12 outputs the alertness level of the driver to the drowsiness presence-absence determination unit 15.

[0040] The facial feature amount calculating unit 12a acquires image data from the image data acquiring unit 11.

[0041] The facial feature amount calculating unit 12a detects the face of the driver from the image data and calculates the feature amount of the face.

[0042] The facial feature amount calculating unit 12a calculates a normalization coefficient of the facial feature amount on the basis of a plurality of pieces of image data acquired by the image data acquiring unit 11.

[0043] The plurality of pieces of image data is a plurality of pieces of image data at different times of capturing by the camera 1, and includes the latest image data output from the image data acquiring unit 11 and past image data output before the latest image data from the image data acquiring unit 11. For example, when the number of pieces of data of the plurality of pieces of image data is N, the plurality of pieces of image data includes (N−1) pieces of past image data in addition to the latest image data. The N pieces of image data are time-series data, and N is an integer equal to or more than 2.

[0044] The facial feature amount calculating unit 12a normalizes the feature amount of the face using the normalization coefficient.

[0045] The facial feature amount calculating unit 12a outputs the feature amount of the face after normalization to the alertness level estimation processing unit 12b.

[0046] The alertness level estimation processing unit 12b acquires the feature amount of the face after normalization from the facial feature amount calculating unit 12a.

[0047] The alertness level estimation processing unit 12b estimates the alertness level of the driver on the basis of the feature amount of the face after normalization.

[0048] The alertness level estimation processing unit 12b outputs the alertness level of the driver to the drowsiness presence-absence determination unit 15.

[0049] The waveform information acquiring unit 13 is implemented by, for example, a waveform information acquiring circuit 23 illustrated in FIG. 2.

[0050] The waveform information acquiring unit 13 acquires, from the sensor 2, waveform information indicating an electrocardiographic waveform of the driver.

[0051] The waveform information acquiring unit 13 outputs the waveform information to the heartbeat feature amount calculating unit 14.

[0052] The heartbeat feature amount calculating unit 14 is implemented by, for example, a heartbeat feature amount calculating circuit 24 illustrated in FIG. 2.

[0053] The heartbeat feature amount calculating unit 14 includes a heartbeat interval specifying unit 14a and a heartbeat feature amount calculation processing unit 14b.

[0054] The heartbeat feature amount calculating unit 14 acquires the waveform information from the waveform information acquiring unit 13.

[0055] The heartbeat feature amount calculating unit 14 calculates the feature amount of the heartbeat of the driver from the electrocardiographic waveform indicated by the waveform information.

[0056] The heartbeat feature amount calculating unit 14 outputs the feature amount of the heartbeat to the drowsiness presence-absence determination unit 15.

[0057] The heartbeat interval specifying unit 14a acquires the waveform information from the waveform information acquiring unit 13.

[0058] The heartbeat interval specifying unit 14a specifies a heartbeat interval of the driver on the basis of the electrocardiographic waveform indicated by the waveform information.

[0059] The heartbeat interval specifying unit 14a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 14b.

[0060] The heartbeat feature amount calculation processing unit 14b acquires a heartbeat interval of the driver from the heartbeat interval specifying unit 14a.

[0061] The heartbeat feature amount calculation processing unit 14b calculates the feature amount of the heartbeat from the heartbeat interval.

[0062] The heartbeat feature amount calculation processing unit 14b calculates a normalization coefficient of the feature amount of the heartbeat on the basis of the plurality of pieces of waveform information acquired by the waveform information acquiring unit 13.

[0063] The plurality of pieces of waveform information is a plurality of pieces of waveform information at different times of detection by the sensor 2, and includes the latest waveform information output from the waveform information acquiring unit 13 and past waveform information output before the latest waveform information from the waveform information acquiring unit 13. For example, when the number of pieces of data of the plurality of pieces of waveform information is M, the plurality of pieces of waveform information includes (M−1) pieces of past waveform information in addition to the latest waveform information. The M pieces of waveform information are time-series data, and M is an integer equal to or more than 2.

[0064] The heartbeat feature amount calculation processing unit 14b normalizes the feature amount of the heartbeat using the normalization coefficient.

[0065] The heartbeat feature amount calculation processing unit 14b outputs the feature amount of the heartbeat after normalization to the drowsiness presence-absence determination unit 15.

[0066] The drowsiness presence-absence determination unit 15 is implemented by, for example, a drowsiness presence-absence determination circuit 25 illustrated in FIG. 2.

[0067] The drowsiness presence-absence determination unit 15 determines the presence or absence of drowsiness of the driver on the basis of the alertness level estimated by the alertness level estimating unit 12 and the feature amount of the heartbeat calculated by the heartbeat feature amount calculating unit 14.

[0068] Specifically, the drowsiness presence-absence determination unit 15 acquires the alertness level of the driver from the alertness level estimation processing unit 12b, and acquires the feature amount of the heartbeat after normalization from the heartbeat feature amount calculation processing unit 14b.

[0069] Then, the drowsiness presence-absence determination unit 15 determines the presence or absence of drowsiness of the driver on the basis of the alertness level of the driver and the feature amount of the heartbeat after normalization.

[0070] In FIG. 1, it is assumed that each of the image data acquiring unit 11, the alertness level estimating unit 12, the waveform information acquiring unit 13, the heartbeat feature amount calculating unit 14, and the drowsiness presence-absence determination unit 15, which are components of the drowsiness determination device 3, is implemented by dedicated hardware as illustrated in FIG. 2. That is, it is assumed that the drowsiness determination device 3 is implemented by the image data acquiring circuit 21, the alertness level estimating circuit 22, the waveform information acquiring circuit 23, the heartbeat feature amount calculating circuit 24, and the drowsiness presence-absence determination circuit 25.

[0071] Further, each of the image data acquiring circuit 21, the alertness level estimating circuit 22, the waveform information acquiring circuit 23, the heartbeat feature amount calculating circuit 24, and the drowsiness presence-absence determination circuit 25 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof.

[0072] The components of the drowsiness determination device 3 are not limited to those implemented by dedicated hardware, and the drowsiness determination device 3 may be implemented by software, firmware, or a combination of software and firmware.

[0073] The software or firmware is stored in a memory of a computer as a program. The computer means hardware that executes a program, and corresponds to, for example, a central processing unit (CPU), a graphics processing unit (GPU), a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a processor, or a digital signal processor (DSP).

[0074] FIG. 3 is a hardware configuration diagram of a computer in a case where the drowsiness determination device 3 is implemented by software, firmware, or the like.

[0075] In a case where the drowsiness determination device 3 is implemented by software, firmware, or the like, a program for causing a computer to execute each processing procedure in the image data acquiring unit 11, the alertness level estimating unit 12, the waveform information acquiring unit 13, the heartbeat feature amount calculating unit 14, and the drowsiness presence-absence determination unit 15 is stored in a memory 31. Then, a processor 32 of the computer executes the program stored in the memory 31.

[0076] Further, FIG. 2 illustrates an example in which each of the components of the drowsiness determination device 3 is implemented by dedicated hardware, and FIG. 3 illustrates an example in which the drowsiness determination device 3 is implemented by software, firmware, or the like. However, this is merely an example, and some components in the drowsiness determination device 3 may be implemented by dedicated hardware, and the remaining components may be implemented by software, firmware, or the like.

[0077] Next, the operation of the drowsiness determination device 3 illustrated in FIG. 1 will be described.

[0078] FIG. 4 is a flowchart illustrating a drowsiness determination method that is a processing procedure of the drowsiness determination device 3.

[0079] For example, the camera 1 starts capturing the face of a driver from a time point when the driver gets into the vehicle and the driver starts the engine. However, the capturing start time point of the camera 1 is not limited to the time point when the driver starts the engine, and may be, for example, a time point when the driver gets into the vehicle.

[0080] The camera 1 outputs image data indicating the face image of the driver to the image data acquiring unit 11 of the drowsiness determination device 3. The face image indicated by the image data may be a moving image or a plurality of still images intermittently captured.

[0081] The sensor 2 detects an electrocardiographic waveform of the driver.

[0082] The sensor 2 outputs the waveform information indicating the electrocardiographic waveform of the driver to the waveform information acquiring unit 13 of the drowsiness determination device 3.

[0083] The image data acquiring unit 11 acquires the image data indicating the face image of the driver from the camera 1 (step ST1 in FIG. 4).

[0084] The image data acquiring unit 11 outputs the image data to the alertness level estimating unit 12.

[0085] The alertness level estimating unit 12 acquires the image data from the image data acquiring unit 11.

[0086] The alertness level estimating unit 12 estimates the alertness level of the driver on the basis of the image data (step ST2 in FIG. 4).

[0087] The alertness level estimating unit 12 outputs the alertness level of the driver to the drowsiness presence-absence determination unit 15.

[0088] Hereinafter, the estimation processing of the alertness level by the alertness level estimating unit 12 will be specifically described.

[0089] The facial feature amount calculating unit 12a acquires the image data from the image data acquiring unit 11.

[0090] The facial feature amount calculating unit 12a detects the face of the driver from the face image of the driver indicated by the image data.

[0091] Specifically, the facial feature amount calculating unit 12a can detect the face of the driver by using, for example, a Haar-Like detector to which a machine learning algorithm called adaptive boosting (AdaBoost) or an algorithm called cascade is applied. The Haar-Like detector is a detector that handles Haar-Like feature amounts. The Haar-Like feature amount is a feature amount obtained from brightness differences of a plurality of local regions.

[0092] The facial feature amount calculating unit 12a calculates the detected facial feature amount.

[0093] Specifically, the facial feature amount calculating unit 12a detects a face part from the driver's face. The face part is, for example, an eye of the driver or the mouth of the driver.

[0094] Next, the facial feature amount calculating unit 12a calculates, for example, the eye opening degree of the driver's eyes or the opening amount of the driver's mouth.

[0095] The eye opening degree of the eye is calculated from, for example, a distance L1 between the upper eyelid and the lower eyelid in the vertical direction of the driver when the driver is not drowsy and a current distance L2 between the upper eyelid and the lower eyelid in the vertical direction of the driver as expressed in the following equation (1). When the driver is seated on the driver's seat and looking forward in the vehicle, the up-down direction of the upper eyelid and the lower eyelid is substantially the same as the vertical direction.

[0096] As illustrated in FIG. 5A, the distance L1 is a distance at a position where the distance between the upper eyelid and the lower eyelid is maximized when the driver is not drowsy. The distance L1 is calculated, for example, from an eye of the driver, which is a face part detected until a certain time elapses after the driver gets into the vehicle. The distance L1 is not limited to the distance calculated from the driver's eye detected until a certain time elapses after the driver gets into the vehicle, and may be stored in an internal memory of the facial feature amount calculating unit 12a or may be provided from the outside of the drowsiness determination device 3, for example.

[0097] As illustrated in FIG. 5B, the distance L2 is calculated from the driver's eye, which is a face part detected from the face image indicated by the latest image data acquired from the image data acquiring unit 11.

[0098] FIG. 5A is an explanatory diagram illustrating the distance L1 between the upper eyelid and the lower eyelid when the driver is not drowsy.

[0099] FIG. 5B is an explanatory diagram illustrating the current distance L2 between the upper eyelid and the lower eyelid of the driver.

$\begin{array}{cc}\mathrm{Eye}\mathrm{opening}\mathrm{degree}\mathrm{of}\mathrm{eye}=\left(L2/L1\right)\times 100& \left(1\right)\end{array}$

[0100] Here, the eye opening degree of the eye is calculated by Expression (1). However, this is merely an example, and the eye opening degree of the eye may be obtained by normalizing the distance L1 in the vertical direction between the upper eyelid and the lower eyelid by a face part such as the nose, the mouth, or an ear.

[0101] The opening amount of the mouth is, for example, a distance in the vertical direction between the upper lip and the lower lip at a position where the distance between the upper lip and the lower lip is maximum.

[0102] The facial feature amount calculating unit 12a calculates a facial feature amount effective for estimating the alertness level of the driver on the basis of, for example, the eye opening degree of the eyes or the opening amount of the mouth.

[0103] Examples of such a feature amount of the face include a ratio of a time during which the eyes are closed within a certain period, the number of blinks in a certain period, and the number of yawns in a certain period.

[0104] The ratio of the time during which the eye is closed within a certain period is obtained by dividing the time during which the degree of eye opening becomes 0% within the certain period by the time of the certain period.

[0105] The number of blinks in a certain period is obtained from the number of times the eye opening degree reached 0% within the certain period.

[0106] Yawning is detected on the basis of a change in the opening amount of the mouth.

[0107] The facial feature amount calculating unit 12a normalizes the facial feature amount in order to absorb individual differences in the facial feature amounts.

[0108] Specifically, for example, the facial feature amount calculating unit 12a accumulates the facial feature amounts calculated on the basis of each of N pieces of image data including the latest image data and the past image data, and calculates an average value of the accumulated feature amounts or a percentile value of the accumulated feature amounts. Then, the facial feature amount calculating unit 12a sets the calculated average value or the calculated percentile value as a normalization coefficient.

[0109] The facial feature amount calculating unit 12a subtracts a normalization coefficient from the facial feature amount calculated on the basis of the latest image data or divides the facial feature amount calculated on the basis of the latest image data by the normalization coefficient, thereby normalizing the facial feature amount.

[0110] The facial feature amount calculating unit 12a outputs the feature amount of the face after normalization to the alertness level estimation processing unit 12b.

[0111] In the drowsiness determination device 3 illustrated in FIG. 1, the facial feature amount calculating unit 12a accumulates the facial feature amount calculated on the basis of each of the N pieces of image data including the latest image data and the past image data. However, this is merely an example, and the facial feature amount calculating unit 12a may accumulate the facial feature amount calculated on the basis of each of N pieces of image data including only past image data.

[0112] Furthermore, the feature amount of the face calculated on the basis of each of the N pieces of image data may be a feature amount of the face calculated during a period from when the driver gets into the vehicle until a certain time elapses.

[0113] The alertness level estimation processing unit 12b acquires the feature amount of the face after normalization from the facial feature amount calculating unit 12a.

[0114] The alertness level estimation processing unit 12b estimates the alertness level of the driver on the basis of the feature amount of the face after normalization.

[0115] Specifically, the alertness level estimation processing unit 12b gives the feature amount of the face after normalization to a learning model using a general algorithm such as random forest or logistic regression, and acquires the alertness level of the driver from the learning model.

[0116] At the time of learning, the learning model is given the feature amount of the face after normalization and teacher data indicating the alertness level of the driver, and learns the alertness level of the driver.

[0117] When the feature amount of the face after normalization is given from the alertness level estimation processing unit 12b at the time of inference, the learning model outputs the alertness level of the driver to the alertness level estimation processing unit 12b. The alertness level of the driver is, for example, data of 0 to 1.

[0118] The alertness level estimation processing unit 12b outputs the alertness level of the driver to the drowsiness presence-absence determination unit 15.

[0119] The waveform information acquiring unit 13 acquires, from the sensor 2, the waveform information indicating the electrocardiographic waveform of the driver (step ST3 in FIG. 4).

[0120] The waveform information acquiring unit 13 outputs the waveform information to the heartbeat feature amount calculating unit 14.

[0121] The heartbeat feature amount calculating unit 14 acquires the waveform information from the waveform information acquiring unit 13.

[0122] The heartbeat feature amount calculating unit 14 calculates the feature amount of the heartbeat of the driver from the electrocardiographic waveform indicated by the waveform information (step ST4 in FIG. 4).

[0123] The heartbeat feature amount calculating unit 14 outputs the feature amount of the heartbeat to the drowsiness presence-absence determination unit 15.

[0124] Hereinafter, calculation processing of the heartbeat feature amount by the heartbeat feature amount calculating unit 14 will be specifically described.

[0125] The heartbeat interval specifying unit 14a acquires the waveform information from the waveform information acquiring unit 13.

[0126] The heartbeat interval specifying unit 14a specifies a heartbeat interval of the driver on the basis of the electrocardiographic waveform indicated by the waveform information.

[0127] The heartbeat interval is the time between two adjacent heartbeats, as illustrated in FIG. 6.

[0128] FIG. 6 is an explanatory diagram illustrating an example of a heartbeat interval of the driver.

[0129] The heartbeat interval specifying unit 14a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 14b.

[0130] The heartbeat feature amount calculation processing unit 14b acquires the heartbeat interval of the driver from the heartbeat interval specifying unit 14a.

[0131] The heartbeat feature amount calculation processing unit 14b calculates the feature amount of the heartbeat from the heartbeat interval.

[0132] Examples of the feature amount of the heartbeat include the following feature amounts (1) to (6).

• • [0133] (1) Average value of heartbeat intervals of the driver within a certain period
  • [0134] (2) Standard deviation within a certain period of the heartbeat interval of the driver
  • [0135] (3) Average value of driver's heartbeats within a certain period
  • [0136] (4) Square root of the average value within a certain period of squares of difference between two heartbeat intervals adjacent in the time direction
  • [0137] (5) Number of times a difference between two heartbeat intervals adjacent in the time direction becomes larger than a threshold within a certain period

[0138] (6) Ratio at which difference between two heartbeat intervals adjacent in time direction becomes larger than threshold within certain period

[0139] The heartbeat feature amount calculation processing unit 14b normalizes the feature amount of the heartbeat in order to absorb the individual differences in the feature amounts of the heartbeat.

[0140] Specifically, the heartbeat feature amount calculation processing unit 14b acquires, for example, M pieces of waveform information including the latest waveform information and the past waveform information from the waveform information acquiring unit 13 via the heartbeat interval specifying unit 14a.

[0141] The heartbeat feature amount calculation processing unit 14b accumulates the feature amount of the heartbeat calculated on the basis of each of the M pieces of waveform information, and calculates an average value of the accumulated feature amounts or a percentile value of the accumulated feature amounts. Then, the heartbeat feature amount calculation processing unit 14b sets the calculated average value or the calculated percentile value as a normalization coefficient.

[0142] The heartbeat feature amount calculation processing unit 14b normalizes the feature amount of the heartbeat by subtracting a normalization coefficient from the feature amount of the heartbeat calculated on the basis of the latest waveform information or dividing the feature amount of the heartbeat calculated on the basis of the latest waveform information by the normalization coefficient.

[0143] The heartbeat feature amount calculation processing unit 14b outputs the feature amount of the heartbeat after normalization to the drowsiness presence-absence determination unit 15.

[0144] In the drowsiness determination device 3 illustrated in FIG. 1, the heartbeat feature amount calculation processing unit 14b accumulates the feature amount of the heartbeat calculated on the basis of each of the M pieces of waveform information including the latest waveform information and the past waveform information. However, this is merely an example, and the heartbeat feature amount calculation processing unit 14b may accumulate the feature amount of the heartbeat calculated on the basis of each of the M pieces of waveform information including only the past waveform information.

[0145] Furthermore, the feature amount of the heartbeat calculated on the basis of each of the M pieces of waveform information may be a feature amount of the heartbeat calculated during a period from when the driver gets into the vehicle until a certain time elapses.

[0146] The drowsiness presence-absence determination unit 15 acquires the alertness level of the driver from the alertness level estimation processing unit 12b, and acquires the feature amount of the heartbeat after normalization from the heartbeat feature amount calculation processing unit 14b.

[0147] The drowsiness presence-absence determination unit 15 determines the presence or absence of drowsiness of the driver on the basis of the alertness level of the driver and the feature amount of the heartbeat after normalization (step ST5 in FIG. 4).

[0148] Specifically, for example, the drowsiness presence-absence determination unit 15 gives the alertness level of the driver and the feature amount of the heartbeat after normalization to a learning model using a general algorithm of random forest or logistic regression, and acquires the determination result of the presence or absence of drowsiness from the learning model.

[0149] At the time of learning, the learning model is given the alertness level of the driver, the feature amount of the heartbeat after normalization, and the teacher data indicating the presence or absence of drowsiness of the driver, and learns the presence or absence of drowsiness. The teacher data is, for example, data “1” when the driver is not drowsy, and data “0” when the driver is drowsy.

[0150] When the alertness level of the driver and the feature amount of the heartbeat after normalization are given from the drowsiness presence-absence determination unit 15 at the time of inference, the learning model outputs the determination result of the presence or absence of drowsiness to the drowsiness presence-absence determination unit 15.

[0151] The drowsiness presence-absence determination unit 15 outputs an determination result of presence/absence of drowsiness to, for example, a driver monitoring system (not illustrated).

[0152] In the first embodiment described above, the drowsiness determination device 3 includes the image data acquiring unit 11 to acquire image data indicating a face image of a driver from the camera 1 that captures the face of the driver, the alertness level estimating unit 12 to estimate an alertness level indicating the degree of alertness of the driver on the basis of the image data acquired by the image data acquiring unit 11, the waveform information acquiring unit 13 to acquire waveform information indicating an electrocardiographic waveform of the driver from the sensor 2 that detects the electrocardiographic waveform of the driver, and the heartbeat feature amount calculating unit 14 to calculate a feature amount of a heartbeat of the driver from the electrocardiographic waveform indicated by the waveform information acquired by the waveform information acquiring unit 13. Further, the drowsiness determination device 3 includes the drowsiness presence-absence determination unit 15 to determine the presence or absence of drowsiness of the driver on the basis of the alertness level estimated by the alertness level estimating unit 12 and the feature amount of the heartbeat calculated by the heartbeat feature amount calculating unit 14. Therefore, the drowsiness determination device 3 can reduce false detection of drowsiness more than the device disclosed in Patent Literature 1.

[0153] In the drowsiness determination device 3 illustrated in FIG. 1, the alertness level estimation processing unit 12b gives the feature amount of the face after normalization to the learning model, and acquires the alertness level of the driver from the learning model. However, this is merely an example, and if the feature amount of the face after normalization is, for example, a proportion of the time during which the eyes are closed within a certain period, the alertness level estimation processing unit 12b may give a proportion of the time during which the eyes are closed within the certain period to a function that returns a larger alertness level as the proportion of the time during which the eyes are closed within the certain period is smaller, and acquire the alertness level of the driver from the function.

[0154] In addition, if the feature amount of the face after normalization is, for example, the number of blinks in a certain period, the alertness level estimation processing unit 12b may give the number of blinks in the certain period to a function that returns a larger alertness level as the number of blinks in the certain period is smaller, and acquire the alertness level of the driver from the function.

[0155] Furthermore, if the feature amount of the face after normalization is, for example, the number of yawns in a certain period, the alertness level estimation processing unit 12b may give the number of yawns in the certain period to a function that returns a larger alertness level as the number of yawns in the certain period is smaller, and acquire the alertness level of the driver from the function.

[0156] In the drowsiness determination device 3 illustrated in FIG. 1, the drowsiness presence-absence determination unit 15 gives the alertness level of the driver and the feature amount of the heartbeat after normalization to the learning model, and acquires the determination result of the presence or absence of drowsiness from the learning model. However, this is merely an example, and the drowsiness presence-absence determination unit 15 may determine that the driver is drowsy, for example, when the alertness level of the driver is smaller than a first determination threshold and the feature amount of the heartbeat after normalization is larger than a second determination threshold, and determine that the driver is not drowsy otherwise.

[0157] Each of the first determination threshold and the second determination threshold may be stored in an internal memory of the drowsiness presence-absence determination unit 15 or may be provided from the outside of the drowsiness determination device 3.

[0158] In addition, by accepting a change in each of the first determination threshold and the second determination threshold, the drowsiness presence-absence determination unit 15 may determine the presence or absence of drowsiness with emphasis on the alertness level of the driver rather than the feature amount of the heartbeat after normalization, or may determine the presence or absence of drowsiness with emphasis on the feature amount of the heartbeat after normalization rather than the alertness level of the driver.

[0159] In the drowsiness determination device 3 illustrated in FIG. 1, the facial feature amount calculating unit 12a outputs the feature amount of the face after normalization to the alertness level estimation processing unit 12b. When the driver in the vehicle is always the same person and there is no need to absorb the individual differences in the feature amounts of the face, the facial feature amount calculating unit 12a may output the feature amount of the face that is not normalized to the alertness level estimation processing unit 12b.

[0160] Further, in the drowsiness determination device 3 illustrated in FIG. 1, the heartbeat feature amount calculation processing unit 14b outputs the feature amount of the heartbeat after normalization to the drowsiness presence-absence determination unit 15. When the driver who gets into the vehicle is always the same person and it is not necessary to absorb the individual differences in the feature amounts of the heartbeat, the heartbeat feature amount calculation processing unit 14b may output the feature amount of the heartbeat that is not normalized to the drowsiness presence-absence determination unit 15.

[0161] In a second embodiment, a drowsiness determination device 3 in which a heartbeat feature amount calculating unit 16 calculates a normalization coefficient of a heartbeat feature amount on the basis of a plurality of pieces of waveform information only when an alertness level estimated by an alertness level estimating unit 12 is equal to or more than a threshold will be described.

[0162] FIG. 7 is a configuration diagram illustrating the drowsiness determination device 3 according to the second embodiment. In FIG. 7, the same reference numerals as those in FIG. 1 denote the same or corresponding parts, and thus description thereof is omitted.

[0163] FIG. 8 is a hardware configuration diagram illustrating hardware of the drowsiness determination device 3 according to the second embodiment. In FIG. 8, the same reference numerals as those in FIG. 2 denote the same or corresponding parts, and thus description thereof is omitted.

[0164] The drowsiness determination device 3 illustrated in FIG. 7 includes an image data acquiring unit 11, the alertness level estimating unit 12, a waveform information acquiring unit 13, the heartbeat feature amount calculating unit 16, and a drowsiness presence-absence determination unit 15.

[0165] The heartbeat feature amount calculating unit 16 is implemented by, for example, a heartbeat feature amount calculating circuit 26 illustrated in FIG. 8.

[0166] The heartbeat feature amount calculating unit 16 includes a heartbeat interval specifying unit 16a and a heartbeat feature amount calculation processing unit 16b.

[0167] The heartbeat feature amount calculating unit 16 acquires waveform information from the waveform information acquiring unit 13, and acquires an alertness level of the driver from the alertness level estimating unit 12.

[0168] The heartbeat feature amount calculating unit 16 calculates the feature amount of the heartbeat of the driver from an electrocardiographic waveform indicated by the waveform information.

[0169] When the alertness level of the driver is equal to or more than a threshold, the heartbeat feature amount calculating unit 16 calculates a normalization coefficient of the feature amount of the heartbeat on the basis of the plurality of pieces of waveform information, and normalizes the feature amount of the heartbeat using the normalization coefficient.

[0170] When the alertness level of the driver is less than the threshold, the heartbeat feature amount calculating unit 16 normalizes the feature amount of the heartbeat using the normalization coefficient when the driver is not drowsy without calculating the normalization coefficient of the feature amount of the heartbeat.

[0171] The heartbeat feature amount calculating unit 16 outputs the feature amount of the heartbeat after normalization to the drowsiness presence-absence determination unit 15.

[0172] The heartbeat interval specifying unit 16a acquires the waveform information from the waveform information acquiring unit 13.

[0173] The heartbeat interval specifying unit 16a specifies a heartbeat interval of the driver on the basis of the electrocardiographic waveform indicated by the waveform information.

[0174] The heartbeat interval specifying unit 16a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 16b.

[0175] The heartbeat feature amount calculation processing unit 16b acquires the heartbeat interval of the driver from the heartbeat interval specifying unit 16a, and acquires the alertness level of the driver from the alertness level estimating unit 12.

[0176] The heartbeat feature amount calculation processing unit 16b calculates the feature amount of the heartbeat from the heartbeat interval.

[0177] When the alertness level of the driver is equal to or more than the threshold, the heartbeat feature amount calculation processing unit 16b calculates a normalization coefficient of the feature amount of the heartbeat on the basis of the plurality of pieces of waveform information acquired by the waveform information acquiring unit 13. The threshold may be stored in an internal memory of the heartbeat feature amount calculation processing unit 16b or may be given from the outside of the drowsiness determination device 3. The plurality of pieces of waveform information is a plurality of pieces of waveform information at different times of detection by the sensor 2.

[0178] The heartbeat feature amount calculation processing unit 16b normalizes the feature amount of the heartbeat using the normalization coefficient.

[0179] When the alertness level of the driver is less than the threshold, the heartbeat feature amount calculation processing unit 16b normalizes the feature amount of the heartbeat using the normalization coefficient when the driver is not drowsy without calculating the normalization coefficient of the feature amount of the heartbeat.

[0180] The heartbeat feature amount calculation processing unit 16b outputs the feature amount of the heartbeat after normalization to the drowsiness presence-absence determination unit 15.

[0181] In FIG. 7, it is assumed that each of the image data acquiring unit 11, the alertness level estimating unit 12, the waveform information acquiring unit 13, the heartbeat feature amount calculating unit 16, and the drowsiness presence-absence determination unit 15, which are components of the drowsiness determination device 3, is implemented by dedicated hardware as illustrated in FIG. 8. That is, it is assumed that the drowsiness determination device 3 is implemented by an image data acquiring circuit 21, an alertness level estimating circuit 22, a waveform information acquiring circuit 23, a heartbeat feature amount calculating circuit 26, and a drowsiness presence-absence determination circuit 25.

[0182] Further, each of the image data acquiring circuit 21, the alertness level estimating circuit 22, the waveform information acquiring circuit 23, the heartbeat feature amount calculating circuit 26, and the drowsiness presence-absence determination circuit 25 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof.

[0183] The components of the drowsiness determination device 3 are not limited to those implemented by dedicated hardware, and the drowsiness determination device 3 may be implemented by software, firmware, or a combination of software and firmware.

[0184] In a case where the drowsiness determination device 3 is implemented by software, firmware, or the like, a program for causing a computer to execute each processing procedure in the image data acquiring unit 11, the alertness level estimating unit 12, the waveform information acquiring unit 13, the heartbeat feature amount calculating unit 16, and the drowsiness presence-absence determination unit 15 is stored in the memory 31 illustrated in FIG. 3. Then, the processor 32 illustrated in FIG. 3 executes the program stored in the memory 31.

[0185] Further, FIG. 8 illustrates an example in which each of the components of the drowsiness determination device 3 is implemented by dedicated hardware, and FIG. 3 illustrates an example in which the drowsiness determination device 3 is implemented by software, firmware, or the like. However, this is merely an example, and some components in the drowsiness determination device 3 may be implemented by dedicated hardware, and the remaining components may be implemented by software, firmware, or the like.

[0186] Next, the operation of the drowsiness determination device 3 illustrated in FIG. 7 will be described. The drowsiness determination device 3 is similar to the drowsiness determination device 3 illustrated in FIG. 1 except for the heartbeat feature amount calculating unit 16. Thus, only the operation of the heartbeat feature amount calculating unit 16 will be described here.

[0187] The heartbeat interval specifying unit 16a acquires waveform information from the waveform information acquiring unit 13.

[0188] Similarly to the heartbeat interval specifying unit 14a illustrated in FIG. 1, the heartbeat interval specifying unit 16a specifies the heartbeat interval of the driver on the basis of the electrocardiographic waveform indicated by the waveform information.

[0189] The heartbeat interval specifying unit 16a outputs the heartbeat interval of the driver to the heartbeat feature amount calculation processing unit 16b.

[0190] The heartbeat feature amount calculation processing unit 16b acquires the heartbeat interval of the driver from the heartbeat interval specifying unit 16a, and acquires the alertness level of the driver from the alertness level estimating unit 12.

[0191] Similarly to the heartbeat feature amount calculation processing unit 14b illustrated in FIG. 1, the heartbeat feature amount calculation processing unit 16b calculates the feature amount of the heartbeat from the heartbeat interval.

[0192] The heartbeat feature amount calculation processing unit 16b acquires, for example, M pieces of waveform information including the latest waveform information and the past waveform information from the waveform information acquiring unit 13 via the heartbeat interval specifying unit 16a.

[0193] The heartbeat feature amount calculation processing unit 16b compares the alertness level of the driver with the threshold.

[0194] When the alertness level of the driver is equal to or more than the threshold, the heartbeat feature amount calculation processing unit 16b calculates a normalization coefficient of the feature amount of the heartbeat on the basis of the M pieces of waveform information, similarly to the heartbeat feature amount calculation processing unit 14b illustrated in FIG. 1.

[0195] Similarly to the heartbeat feature amount calculation processing unit 14b illustrated in FIG. 1, the heartbeat feature amount calculation processing unit 16b normalizes the feature amount of the heartbeat using the normalization coefficient.

[0196] When the alertness level of the driver is less than the threshold, the heartbeat feature amount calculation processing unit 16b normalizes the feature amount of the heartbeat using the normalization coefficient when the driver is not drowsy without calculating the normalization coefficient of the feature amount of the heartbeat. The normalization coefficient when the driver is not drowsy may be stored in the internal memory of the heartbeat feature amount calculation processing unit 16b or may be given from the outside of the drowsiness determination device 3.

[0197] Furthermore, the normalization coefficient when the driver is not drowsy may be, for example, a normalization coefficient calculated by the heartbeat feature amount calculation processing unit 16b until a certain time elapses from the time point when the driver gets into the vehicle.

[0198] In a case where the normalization coefficient of the feature amount of the heartbeat is calculated on the basis of the waveform information when the driver is drowsy, the feature amount of the heartbeat after normalization does not reflect the intensity of drowsiness much. As a result, even if the driver is drowsy, the determination result indicating that the driver is drowsy may not be obtained from the drowsiness presence-absence determination unit 15. Therefore, the heartbeat feature amount calculation processing unit 16b normalizes the feature amount of the heartbeat using the normalization coefficient when the driver is not drowsy.

[0199] The heartbeat feature amount calculation processing unit 16b outputs the feature amount of the heartbeat after normalization to the drowsiness presence-absence determination unit 15.

[0200] In the second embodiment described above, when the alertness level estimated by the alertness level estimating unit 12 is equal to or more than the threshold, the heartbeat feature amount calculating unit 16 calculates a normalization coefficient of the feature amount of the heartbeat on the basis of the plurality of pieces of waveform information, and normalizes the feature amount of the heartbeat using the normalization coefficient. The drowsiness determination device 3 illustrated in FIG. 7 is configured in such a manner that when the alertness level estimated by the alertness level estimating unit 12 is less than the threshold, the heartbeat feature amount calculating unit 16 normalizes the feature amount of the heartbeat using the normalization coefficient when the driver is not drowsy without calculating the normalization coefficient of the feature amount of the heartbeat. Therefore, the drowsiness determination device 3 illustrated in FIG. 7 can enhance the drowsiness sensing accuracy more than drowsiness determination device 3 illustrated in FIG. 1.

[0201] In a third embodiment, a drowsiness determination device 3 will be described in which, if an alertness level estimated by an alertness level estimating unit 12 is less than a threshold, a drowsiness presence-absence determination unit 17 determines the presence or absence of drowsiness of the driver on the basis of the alertness level.

[0202] FIG. 9 is a configuration diagram illustrating the drowsiness determination device 3 according to the third embodiment. In FIG. 9, the same reference numerals as those in FIGS. 1 and 7 denote the same or corresponding parts, and thus description thereof is omitted.

[0203] FIG. 10 is a hardware configuration diagram illustrating hardware of the drowsiness determination device 3 according to the third embodiment. In FIG. 10, the same reference numerals as those in FIGS. 2 and 8 denote the same or corresponding parts, and thus description thereof is omitted.

[0204] The drowsiness determination device 3 illustrated in FIG. 9 includes an image data acquiring unit 11, an alertness level estimating unit 12, a waveform information acquiring unit 13, a heartbeat feature amount calculating unit 14, and a drowsiness presence-absence determination unit 17.

[0205] The drowsiness presence-absence determination unit 17 is implemented by, for example, a drowsiness presence-absence determination circuit 27 illustrated in FIG. 10.

[0206] The drowsiness presence-absence determination unit 17 acquires an alertness level of a driver from an alertness level estimation processing unit 12b, and acquires a feature amount of the heartbeat after normalization from a heartbeat feature amount calculation processing unit 14b.

[0207] When the alertness level of the driver is equal to or more than the threshold, the drowsiness presence-absence determination unit 17 determines the presence or absence of drowsiness of the driver on the basis of the alertness level of the driver and the feature amount of the heartbeat after normalization, similarly to the drowsiness presence-absence determination unit 15 illustrated in FIG. 1. The threshold may be stored in an internal memory of the drowsiness presence-absence determination unit 17 or may be given from the outside of the drowsiness determination device 3.

[0208] When the alertness level of the driver is less than the threshold, the drowsiness presence-absence determination unit 17 determines the presence or absence of drowsiness of the driver on the basis of the alertness level of the driver.

[0209] In FIG. 9, it is assumed that each of the image data acquiring unit 11, the alertness level estimating unit 12, the waveform information acquiring unit 13, the heartbeat feature amount calculating unit 14, and the drowsiness presence-absence determination unit 17, which are components of the drowsiness determination device 3, is implemented by dedicated hardware as illustrated in FIG. 10. That is, it is assumed that the drowsiness determination device 3 is implemented by an image data acquiring circuit 21, an alertness level estimating circuit 22, a waveform information acquiring circuit 23, a heartbeat feature amount calculating circuit 24, and a drowsiness presence-absence determination circuit 27.

[0210] Further, each of the image data acquiring circuit 21, the alertness level estimating circuit 22, the waveform information acquiring circuit 23, the heartbeat feature amount calculating circuit 24, and the drowsiness presence-absence determination circuit 27 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof.

[0211] The components of the drowsiness determination device 3 are not limited to those implemented by dedicated hardware, and the drowsiness determination device 3 may be implemented by software, firmware, or a combination of software and firmware.

[0212] In a case where the drowsiness determination device 3 is implemented by software, firmware, or the like, a program for causing a computer to execute each processing procedure in the image data acquiring unit 11, the alertness level estimating unit 12, the waveform information acquiring unit 13, the heartbeat feature amount calculating unit 14, and the drowsiness presence-absence determination unit 17 is stored in the memory 31 illustrated in FIG. 3. Then, the processor 32 illustrated in FIG. 3 executes the program stored in the memory 31.

[0213] Further, FIG. 10 illustrates an example in which each of the components of the drowsiness determination device 3 is implemented by dedicated hardware, and FIG. 3 illustrates an example in which the drowsiness determination device 3 is implemented by software, firmware, or the like. However, this is merely an example, and some components in the drowsiness determination device 3 may be implemented by dedicated hardware, and the remaining components may be implemented by software, firmware, or the like.

[0214] Next, the operation of the drowsiness determination device 3 illustrated in FIG. 9 will be described. The drowsiness determination device 3 is similar to the drowsiness determination device 3 illustrated in FIG. 1 except for the drowsiness presence-absence determination unit 17. Thus, only the operation of the drowsiness presence-absence determination unit 17 will be described here.

[0215] The drowsiness presence-absence determination unit 17 acquires the alertness level of the driver from the alertness level estimation processing unit 12b, and acquires the feature amount of the heartbeat after normalization from the heartbeat feature amount calculation processing unit 14b.

[0216] The drowsiness presence-absence determination unit 17 compares the alertness level of the driver with a threshold.

[0217] When the alertness level of the driver is equal to or more than the threshold, the drowsiness presence-absence determination unit 17 determines the presence or absence of drowsiness of the driver on the basis of the alertness level of the driver and the feature amount of the heartbeat after normalization, similarly to the drowsiness presence-absence determination unit 15 illustrated in FIG. 1.

[0218] When the alertness level of the driver is less than the threshold, the drowsiness presence-absence determination unit 17 determines the presence or absence of drowsiness of the driver on the basis of the alertness level of the driver.

[0219] Specifically, if the alertness level of the driver is equal to or more than a set value, the drowsiness presence-absence determination unit 17 determines that the driver is drowsy.

[0220] If the alertness level of the driver is less than the set value, the drowsiness presence-absence determination unit 17 determines that the driver is not drowsy. The set value may be stored in an internal memory of the drowsiness presence-absence determination unit 17 or may be given from the outside of the drowsiness determination device 3.

[0221] The drowsiness presence-absence determination unit 17 outputs an determination result of presence/absence of drowsiness to, for example, a driver monitoring system (not illustrated).

[0222] In the third embodiment described above, if the alertness level estimated by the alertness level estimating unit 12 is equal to or more than the threshold, the drowsiness presence-absence determination unit 17 determines the presence or absence of drowsiness of the driver on the basis of the alertness level and the feature amount of the heartbeat after normalization output from the heartbeat feature amount calculating unit 14. The drowsiness determination device 3 illustrated in FIG. 9 is configured in such a manner that, when the alertness level estimated by the alertness level estimating unit 12 is less than the threshold, the drowsiness presence-absence determination unit 17 determines the presence or absence of drowsiness of the driver on the basis of the alertness level. Therefore, the drowsiness determination device 3 illustrated in FIG. 9 can improve drowsiness sensing accuracy more than the drowsiness determination device 3 illustrated in FIG. 1.

[0223] Note that, in the present disclosure, free combinations of the embodiments, modifications of any components of the embodiments, or omissions of any components in the embodiments are possible.

[0224] The present disclosure is suitable for a drowsiness determination device and a drowsiness determination method.

[0225] 1: camera, 2: sensor, 3: drowsiness determination device, 11: image data acquiring unit, 12: alertness level estimating unit, 12a: facial feature amount calculating unit, 12b: alertness level estimation processing unit, 13: waveform information acquiring unit, 14: heartbeat feature amount calculating unit, 14a: heartbeat interval specifying unit, 14b: heartbeat feature amount calculation processing unit, 15: drowsiness presence-absence determination unit, 16: heartbeat feature amount calculating unit, 16a: heartbeat interval specifying unit, 16b: heartbeat feature amount calculation processing unit, 17: drowsiness presence-absence determination unit, 21: image data acquiring circuit, 22: alertness level estimating circuit, 23: waveform information acquiring circuit, 24: heartbeat feature amount calculating circuit, 25: drowsiness presence-absence determination circuit, 26: heartbeat feature amount calculating circuit, 27: drowsiness presence-absence determination circuit, 31: memory, 32: processor