This paper focuses on analyzing the factors of knowing, centering on Sishu jizhu, and deriving the general meaning that satisfies all factors. The fundamental elements of knowing can be classified into four categories. First, it is the intrinsic factor that insights into the human nature of the mind by inferring the fundamentals of behaviors that conform to the five relationships through intellectual talent. Second, it is an intrinsic factor that internally understands the use of the five relationships while reading Confucian classics, the object of knowing. Third, it is the extrinsic factor that penetrates through the act to know whenever there is a doubt about why the five relationships should be practiced. Fourth, it is the extrinsic factor to practice the result of knowing the five relationships. The meaning of knowing that satisfies all of these factors is to realize the human nature of the mind through intellectual talent, to tide the five relationships with the body while boldly trying to read Confucian classics to achieve the Tao of self-cultivation and governing people.