SYSTEM AND METHOD FOR PROVIDING INTERACTION WITH DRIVER OF VEHICLE DEPENDING ON SITUATION

Calling a driver interaction driver based on customized to setting the extent to which the control module to a traveling condition of a vehicle capable of road and thereby improves the safety. Also in the embodiment according to Figure 1 shows a system for executing the operating in accordance with a situation of one driver (10) block representing are disclosed. System (10) vehicle (15) the operator of the motor (14) of (13) sensing and vision module monitoring performance of work (12) executing the one or more servers (11) having a predetermined wavelength. Of a driver module to a indicating whether the driver or is away a but, other of the States also disclosed. For example, in the embodiment in another, of a driver (13) comprising a driver approaches feelings can be. Sensing and vision module (12) vehicle (15) interacts with one or more sensors of monitors a status of other. Monitoring a driver operating a sensor at least one driver - oriented digital camera (16) having a predetermined wavelength. Camera (16) is a visible light Image-camera (16) may be; alternatively camera (16) is infrared camera (16) may be disclosed. Other types of camera also disclosed. In addition user can be other sensors monitoring. For example, attract the biometric sensors (biometric sensor) can be worn by, e.g. smart micro (smartwatch) (17) can be integrated, driver (14) can be physiologically sensing data. The, sensor light volume variation waveform (photoplethysmogram) of a driver ("PPG wave") smart micro record (17) that is integrated into the pulse oximeter (pulse oximeter) implementation being. In another types of sensors also disclosed.

Server (11) is associated with a specific moment in time traveling situation of a set of environment sensors determining situation module (18) further executable module. This situation situation graph (19) which appears as the, this situation the graph situation module (18) is stored in a backup storage media (20) can be stored in addition to the. Situation graph (19) includes a semantic graph (19) are disclosed (also referred to as "semantic graph (19)" are referred to the description of the situation the graph below). Driver (140) in addition to the state of, semantic graph (19) is, speed of vehicle or engine per minute of be repeated, in front of any object, vehicle (15) travels road state, and vehicle for road lane markings on a roadway and maintain a position (this may be determined using a vehicle sensor) whether such as vehicle (15) state (21) but reflect, pivotably in addition different types of vehicle condition information. In addition, storage device (20) is a specific city or region such as spatial data regarding the position of the driver (22) but also the geographic location of the pivotably include different types. Spatial data (22) between the position in restaurant, shop and other points of interest comprising data relating to can be. Spatial data (22) such as a cellular network is Internet or internetwork (Internetwork) (23) is accessible via one or more web page (21) from situation module (18) can be obtained by. Situation module (18) is, in connection with the vehicle a built-in GPS can be achieved and inter-network (23) connect to a vehicle (15) to be transmitted via the embedded wireless transceiver receives a present location of the vehicle, identifying points of interest can be located adjacent the driver. Situation module (18) points of interest semantic graph (19) can be included.

In additional in the embodiment, vehicle (15) sensor, for example, camera or the other, which the driver need not to be concentrated attention and output in the high load; slightly tilting the determined driver attention must be allowed in order additional load normal load; or driver variable eddy or leisure in times of low load such as lamp driver load (not shown) can be measure. The blink of an eye velocity load (blinking rate) (this can be determined as described below) or evaluating, leaping of the eye pupil (pupil dilation) (saccade) or extension of measuring but may be measured, other evaluation method driver load also disclosed. The sensors by includes a wireless transceiver data from the sensors are sent to the semantic graph (19) can be integrated. Semantic graph (19) is registered user 8 December 2015 roberts (Roberts) a cavity such as a call status American patent number 9,208,439 disclosure as driver's condition (13), vehicle condition (21), position information (22) or other information can be generated from. Semantic graph (19) other method generating also disclosed.

Server (11) includes, collecting driver information, based on the collected information, e.g., examination of a driver characteristics of learning a driver current and past tube such as personal data module (24) but further execution, also other characteristic and driver, this characteristic is, as is further described below with reference to the driver also 5 profile (25) represented. Simply, personal data module includes inter-network (23) to the server through the (11) can be retrieved by a driver related web content (26) extracting data items from (23) to each other. Web contents (26) is social networking Bulletin (social networking post) by the driver or driver's social network connection, driver having a subscription in a RSS feed (feed), one or more servers (27) carried on driver's social network profile can include information such as feed but, other types of web contents (26) also disclosed.

The extracted data item storage device (20) may be stored in a number of other main (28) (grouped into categories can be a distressing main number) and, personal data module (24) by, but compared, comparison of different types also disclosed. Based on the comparison, each data item associated with layer (28) main number is identified substrate. Identified based on main number, data item classification parameter space can be uniform. In particular, in one in the embodiment, each representation vector data items identified can be generated from the main number. Such vectors can be hierarchical classification in document represents a major number, unique to the content of the data item to a point define a high-dimensional vector space. Vector of data item can be weighted based on an old degree (age), data items are weighted more recent than vector. Weighted vectors describing techniques driver in an examination that the operator's current and past tube profile (25) functioning as coupled to a single vector. Field vector hierarchical (28) associated with a main number included in numerical value into the slide groove. In additional in the embodiment, personal data module (24) comprises a plurality of user as are vectors combining population prior distribution (population prior) (29) can be formed.

Pre-distribution (29) includes a cluster formation (clustering) and supervision created through other techniques such as free learning technology but also disclosed. For example, the prior distribution (29) is based on the similarity (25) can be used for joining a recommendation can be configured through the cooperation filtering rules. Similarly, population prior distribution profile (25) and other data such as driver age in driver (13) can be constructed based on associated with other information. Another method generating population prior distribution also disclosed. Specific driver profile (25) nearest to the population distribution as is further described under pre driver profile (25) for imparting motion to a recommendation can be used instead.

Semantic graph (19) and a driver (14) profile (25) (or closest prior distribution) together with the (11) executed by the recommender (30) by, personal characteristic of a driver (e.g., examination of tube), of a driver, including current status and the status of the vehicle driver position a single vector space "calling" vector (31) merged into a substrate. In addition, the recommender (30) a driver approaches (14) can be taken allows (32) to a list of accesses. The operation may include driver (14) and other operational can be practiced in a specific interactive pattern. For example, the operation may include driver (14) made by social networking connection of social networking bulletins for main number such as driver (14) dialogue the; and returns (14) whether desired magnetic gear presenting new stories, whether recipient want have heard about the point of particular interest around driver (14) can be ask the. List (32) allows for another also disclosed. Allows for implement can be recommended for parameterized recommendation allows (33) disposed thereon for generating.

Recommendation allows (33) for generating, recommender (30) a driver approaches (14) connected to the recent social network, recent new adjust the web contents (26) associated with an item of data representative of information from recent extracting current driver, parameterized operation effected as far as possible the contents which are extracted as follows. This operation (33) is calling vector (31) further based on its isomorphs. The, e.g., allows for (32) when the driver a conversation, vector (31) and a cooking (cooking) of interest on the display driver, and a social networking Bulletin cooking current information extracted, generating operation be a social networking is extracted contents distribution it will be a great disaster.

Characteristic operating in a high-dimensional vector space generated recommendation allows each represented by vectors. Vector space comprises a main number layers representation into the slide groove. For example, China/fusion and anti-bent when talking about points of interest, "and anti-fusion cooking (Asian Fusion Cuisine)" and "Chinese (Chinese Cuisine)" may have a value higher in an techniques. Similarly, when a portion of the content intended for user to play induces sleepiness higher than in "sleep" may have. In addition the homepage is made when such content is friend related content item is a close friend is not 0 "friend" near friend is in contrast to prior 1 may have value.

Further, the resulting operations (32) be implemented at least some of the operation before triggering condition satisfied (34) can be associated with, the hereinafter described more respect. For example, driver (14) that the operator's cognitive load operation including a dialogue the conversation safely performing a trigger by the electrodes and the implemented until may not disclosed. Similarly, point of particular interest for interaction is be be triggered by peripheral points of interest.

The recommender (30) characterization of the vector used to analyze the generated operation imparts a ranking them. The plurality of rankings can be performed. For example, characterization vectors are driver profile (25) can be compared with the parent density. For example, in one in the embodiment, the value of the characterization vector within slots in, driver profile (25) and then multiplied by the values in vector in a corresponding slot, so that the item number encoded value aggregating in the score. The recommender (30) compares the scores on different vectors, recommendation based allows for comparison (33) for vector ranking the substrate. In additional in the embodiment, the specific operation performed before the operation whether novelty (novelty) and indicate how recently performed previously (recency) can be imparted to the fresh operations according to the ranking. In another in the embodiment, a plurality of ranking can be imparted using a number of techniques, coupled and differentially weighted result. The weights can be optimized by machine learning.

Such a specific ranking vector operation having two upper 2 score vector (33) recommender for being executed (30) recommended by 2000. In another in the embodiment, other ranking of different operation (33) also can be used in addition. And the various data item, the disclosure to a cavity such as a robot (Roberts) status American patent application number 2015/0142785 21 May 2015, 5 April 2008 and bellows tee (Belotti) such as "signature tee mobile leisure guide through an activity-based advantageous recommendation (Activity provided based serendipitous recommendations with the Magitti mobile leisure guide)" (CHI2008) can be performed as to disclosure.

Recommended operation (33) server (11) implemented by an operating module (35) embodied by. Operation module (35) the driver (14) involved in natural language dialogue on natural language component (35) having a predetermined wavelength. The interaction is described 2015/0293904 disclosure to cavity 15 October 2015 status American patent application disclosure number can be the application management. The recommender (30) is, recommended operation (33) and, in various forms such as shape of object serialized JSON, operation module (35) can be [...] to number, the object can, operation (33) in addition to delivered, driver (14) the current state of (13), and operation can be required to implement any information. The, number presents information associated participating points (the current shift range and away a in the vicinity of the points of interest) in a trigger to take a variety of actions (34); semantic graph (19) retrieved from the center module to a driver (since vary between embodiment information with information on updated) situation information; driver (14) profile (25) personal data from, and recommended operation (33) can be used to implement a social networking update, and sports and entertainment such as extracted web contents (26) can be a. Operation module includes driver's condition (13) presents a triggering condition by analyzing information and other number (34) generated by combine triggering condition (34) associated with recommendation operation when it is time to each other.

Natural language component (35) is a bi-directional interactive conversation can be prompt and assisted. The components by interactive prompts (36) such as social networking Bulletin is read game can call one of the predefined pattern. Identifying selecting music such as other conversation operation is simpler interaction module with a number [...] substrate. For example, is less than the design makes a melody (upbeat tune) prevent when recommending, recommender (30) (X) (ConfirmTune) is [...] publication request can, wherein X a driver approaches recommended by preference known and the current state of the melody are disclosed. To the request by natural language component (36) for the driver and a driver approaches an FAQ number [...] approval or rejection can be loose music number.

Operation module (36) for performing operation recommended internetwork (23) located vehicles through driver interface (37) through driver (14) interacts with the substrate. In one in the embodiment, driver interface (37) may be software components is, travel or navigation computer, the vehicle rearview mirror monitor and with other components (15) can be integrated using an onboard computer system. In additional in the embodiment, driver interface (37) has a (10) in an area not used by other vehicle components dedicated to portion can be hardware components. Driver interface (37) the interface forwarding the voice shape such as a ring-shaped display animation varying opacity but, other visual interface express disclosed. Operation module (35) to the involved if it is determined that a specific interactive, operation module (35) the driver (14) tell the full text by the TTS (text provided to a non-speech) transformation agent interface (37), e.g., every matter tendency [chu main, [pe ring tone (Burlington) number in a communication [cu yarn (Nuance Communications, Inc.) high pressure liquid coolant residing behind a nuance nuance (Nuance (registered trademark) using commercially available software such as TTS be full text transformations. Subject received voice are disclosed. Voice vehicle (15) that is integrated into the speaker (not shown) transferred through. Driver (14) the response of the driver interface (13) connected to one or more microphones and picked up through driver (14) is fitted at its interaction can be used. Interface (37) an appropriate voice - full text conversion software audio - full text conversion driver prior to performing other necessary for performing processing and applying basic threshold voice, a conversation analysis reported for full text or other take a variety of actions a natural language component (36) and transmits it to. Interface (37) is in addition to flash to a user for user knows which can be light further comprises another component take a variety of actions such as can be. Driver interface (37) other elements pivotably.

As aforementioned, server (11) comprises a basic body to the disclosure in the embodiment comprise a plurality of modules for performing specification. The source code written in a programming language of the existing method of these modules can be implemented as a computer program or procedures, as object or byte code executed by the central processing device for ball number encoded. Alternatively, integrated circuit or read only memory component in addition these modules is installed inside can be embodied as hardware, each server has a specialized computer can be operated. For example, when implemented in the hardware module, specific hardware described above and other computer cannot use a specialized to perform communication. Additionally module read only memory component for storing the read only memory curved to other the computer performing a specialized computer described above cannot be performed. Source code and object and byte code various implementations of example floppy disk, hard drive, digital video disk (DVD), random access memory (RAM), read only memory (ROM) and similar storage medium such as a computer-readable storage medium can be maintained. Other types of module and module function other physical hardware component capable pivotably. For example, server (11) input/output port, network interface and non-volatile storage device such as a programmable computing device it can be found in other components, other components are also disclosed. Server (11) on the storage device (20) is part of a server or dedicated computing environment can be clouds.

According to take the driver driver things tube examination operation setting module to a driver can be helps to maintain operation device. Figure 2 shows a driver information and stores operation also one in the embodiment according to method (40) representing flow are disclosed. Method (40) can be implemented using the system of Figure 1. As is further described below with reference to fig. 3, the present operating environment information (step 41) determined. As is further described below with reference to driver 6 also take one or more actions are recommended. One or more recommended during operation is being performed, method (40) is completed at a time t5. (step 43). With reference to 1 also as aforementioned, the method recommended operation is the driver or the driver intervention is still another operation can be. With reference to 1 also as aforementioned, recommended items can be associated with at least some of the trigger, this recommended operation are performed when associated with trigger is recognized. Some operation which may be performed when not associated with trigger can be recommended.

Various components can be driver situation. Figure 3 shows a situation of a driver in the embodiment of Figure 2 method according one also used determining routine (50) representing flow are disclosed. One or more sensors such as a camera of a driver contained vehicle determined using (step 51) but, as is further described below with reference to the 4 also, other method determining situation also disclosed. The state of the vehicle determined (step 52) of sensor via. Retrieving information from the Internet such as, driver applies the spatial information is obtained (step 53) position. Of a driver, and the status of the vehicle a semantic graph represented by spatial information (step 54). Operation is further described with reference to the user's profile including auditory canal 6 under driver characteristics as determined (step 55); in another in the embodiment, the prior distribution of hypothetical profiles can be generated (not shown). Driver characteristics, driver's condition, the vehicle condition including information on current calling that characterize the vector is also 5 as is further described below with reference to the semantic graph driver profile (or otherwise subsidized closest prior distribution) (step 56) generated by incorporating, routine (50) is completed at a time t5.

Figure 4 shows a one in the embodiment of Figure 3 according to driver's condition monitoring routine also used in routine (60) representing flow are disclosed. Initially, vehicle for images captured by the camera front, left and right face detector by course (course) (front - right - left) concurrently posture estimation is carried out (step 61). For frames during which the detected as the front facial pose, and at the same time face indicia such as applying a Kalman filter technique is detected and tracked set of robust features but, also disclosed (step 62) other techniques. Is characterized eye, nose, and mouth face (face pat) such as fat relative geographic location other features but also disclosed. As is further described with reference to also 5 (step 63), driver that other characteristics used to estimate where to make a driver based on the particular characteristics of a driver in a common direction the precision capable estimation is carried out. In one in the embodiment, two different sight estimation is performed with respect to the direction 8 but, in other different numbers of direction in the embodiment also disclosed. As a result of estimation precision capable of a driver for measuring the extent of dibasic acids such as another route and the vehicle speed driver coupled to the context information (step 64). The blink of an eye and eye velocity such as camera are estimated using a cross-section of an eye metric (step 65) percent. Document [E. Murphy-a Chutorian, M. Trivedi, "Head Pose Estimation in Computer Vision: A Survey", IEEE TPAMI, Vol. 31, no. 4, pp. 607 - 626, 2009 and E. Murphy-a Chutorian, A. Doshi, M. M. Trivedi, "Head Pose Estimation for Driver Assistance Systems: A Robust Algorithm and Experimental Evaluation", IEEE ITSC, 2007] described techniques, such as estimated frequency of operation and sag by nod head operation but, other techniques also disclosed (step 66). Document [B. Lee and W. Chung, "Driver alertness monitoring using fusion of facial features and bio-a signals", IEEE Sensors Journal, Vol. 12, No. 7, pp. 2416 - 2422] as described, worn by the PPG waveform detection unit driver obtained high peak - to - peak statistics and power spectral density such as PPG waveform features but is calculated, calculating the feature also disclosed (step 67) other method. Calculated PPG waveform, the movement head from an estimated eye metric and estimated driver back another time and other related information such as a driver to travel in just such as another situation information (the information can be processed from the database and graphs may be included and, driver vehicle sensors determined based on or social networking Bulletin by analyzing can be determined but, other method for determining this information also disclosed) [...] substrate (step 68) degree of sensed steering wheel comes into engagement with a number. Selectively, the estimated metric indicator number may be displayed or inter-network connection with number 3 (step 69) can be applied to through hole, routine (60) is completed at a time t5. In another in the embodiment, output signal of the vehicle sensor gesturing or surrounding environment can be analyzing data for other driver.

Estimating a driver the direction can be help determine how the average driver. Figure 5 shows a one in the embodiment of Figure 4 also according to routine (60) capable to perform the estimation precision used in routine (70) indicating flow are disclosed. The distance the driver have an one or more of the video is recorded have an labeled (step 71). The video can be stored in a storage device. Each training video features identified substrate (step 72) the face of the driver. Step (62) acquired in the face of the driver (step 73) compared with the face feature of video training features, precise voice according capable estimation is carried out. (Coherency) idea (consistency) occupied by frames to obtain consistency and over to the predetermined smoothing is performed and the eyes relative to the estimation (step 74), routine (70) is completed at a time t5.

Figure 6 shows a one in the embodiment of Figure 2 method also according to (40) driver used in determining a characteristic routine (80) representing flow are disclosed. A driver data items relating to, driver tube examination, social networking connection including driver belongs is the driver or the social network user model is another website hosting web contents are extracted from such user is generated (step 81) contents. The extracted data item is compared with main number of layers, layer cap in each data items are associated with the main number is identified (step 82). The identified data items using main number (step 83) can be classification in uniform parameter space. With reference to 1 also as aforementioned, in one in the embodiment, the identification classification comprising vector representing a main number can be. For each data item of data item vector weighted based on an old degree and, than older data item (current driver characteristics) vector (step 84) imparts a latest data item further more weight. Weighted vector such as examination of a driver tube current and past properties of a driver profile function (step 85) coupled to a single vector. Selectively, along with the personal data by representing the profile vector (step 86) can be pre-population, routine (80) is completed at a time t5.

Driver operation of a driver is executed against the database driver things according can be recommendations. Figure 7 shows a one in the embodiment of Figure 2 method also according to (40) driver to be used in take various routine (90) indicating flow are disclosed. Allows users (step 91) being maintained. Recent social networking bulletins driver such as driver's social networking connection present information extracted from a web, selectively semantic graph (19) (step 92) in indexed. The extracted information current, potential object and a recommendation based on the calling vector (step 93) allows is generated. The ranking is given to the substrate (step 94) allows for recommendations. As also with reference to the aforementioned 1, wherein at least one of the ranking process from a plurality of method. As above-described with reference to also 1, based on the ranking for one or more action is selected (step 95) routine (90) is completed at a time t5.