This paper proposes a stochastic model for human driving behavior using a double layer Hidden Markov Model (HMM) with continuous observations. In the proposed model, gas pedal’s position, steering wheel’s angle, velocity and angular velocity of the vehicle is used and recorded in every 100 msec for recognizing driving task. Data acquisition is done during a simulated driving task, after that data is segmented and clustered into 9 different cases. The lower-layer with one-dimensional continuous HMM is used for recognizing translational acceleration of a vehicle. The upper-layer with one dimensional continuous HMM is used for recognizing angular velocity of a vehicle. For recognizing a user’s behavior, we used sliding windows with size of 10 samples and length of sequence with size of 30 samples. We apply a Kalman filter to reduce noise. After the filtering, the data was processed by sorting into three groups for a pedal, a steering wheel, and speed. We used two main features, which are angular velocity and the translational acceleration, in order to present driving behavior. We constructed a driving simulator based on Logitech G27 Racing platform to evaluate the proposed method. Using the developed driving simulator, some experiments were conducted for comparing the accuracy rate of the proposed method with that of the conventional method. Futhermore, we compared the average learning time of the proposed method with that of the conventional method because Learning time becomes also an important factor for investigating the performance of stochastic models. The experimental results confirm the accuracy of the proposed approach by revealing recognition accuracy around 96%-97%. Furthermore, the proposed method decreases learning time around 40%.