The ubiquity of handheld devices provides straightforward access to the Internet and Social networking. The quick and easy updates from social networks help users in many situations like natural disasters, man-made disasters, etc. In such situations, individuals share information with the people in their network without checking the veracity of posts, which leads to the issue of rumor diffusion in a social network. Detection of rumor and source identification plays a vital role to control the diffusion of misinformation in a social network and also a good research domain in social network analysis. Source detection of such misinformation is often interesting and challenging task due to the fast diffusion of information and dynamic evolution of the social network. Accurate and quick detection of the rumor source is a very important and useful task in many application domains like source of disease in an epidemic model, start of virus spread, source of information or rumor in a social network. Most of the existing reviews which focused on source detection relate to various application domains and network perspective. But as per the need of current social networking usage and its influence on the society, it is a crucial and important topic to review the source detection approaches in the social network. The objective of this paper is to study and analyze the source detection approaches of rumor or misinformation in a social network. As an outcome of the literature study, we present the pictorial taxonomy of factors to be considered for the source detection approach and the classification of current source detection approaches in the social network. The focus has been given to various state-of-the-art source detection approaches of rumor or misinformation and comparison between approaches in social networks. This paper also focused on research challenges in current source detection approaches, public datasets and future research directions.