A Survey on Automatic Image Annotation Methods

  • Farsad Zamani Department of Computer Science, Isfahan (Khorasgan) Branch, Islamic Azad University, Isfahan, Iran
  • Arash Hemmati
Keywords: Image Annotation, Content Based Image Retrieval, Semantic Gap


In the modern world a huge amount of data is being produced every second and a considerable percentage of them are images that need to be processed and analyzed. One of the critical challenges in this aspect is image recovery. The process of image recovery should be done automatically by the machines which is the process of recognition of images concepts and assigning homological labels to them. In order to discover the hidden concepts in the images, one should achieve high level concepts using the low-level features, which is a difficult task. A variety of techniques are proposed to solve this problem that usually use combination of different algorithms. In this paper we review and compare various popular and modern image annotation techniques.


[1] A. Kumar, S. Dyer, J. Kim, C. Li, P. H. Leong, M. Fulham, et al., "Adapting content-based image retrieval techniques for the semantic annotation of medical images" Computerized Medical Imaging and Graphics, vol. 49, pp. 37-45, 2016.
[2] J.Zhang, W.Hu. "Effective multi-modal multi-label learning for automatic image annotation." In Fuzzy Systems and Knowledge Discovery (FSKD), 2012 9th International Conference on, pp. 1216-1220. IEEE, 2012.
[3] M. S. I. Abdulhammed, "Automatic image annotation" CU Theses, 2012.
[4] D.Zhang, MM.Islam, G. Lu. "A review on automatic image annotation techniques." Pattern Recognition 45, no. 1 (2012): 346-362.
[5] MP.Gangan, R. Karthi. "Automatic Image Annotation by Classification Using Mpeg-7 Features." International Journal of Scientific and Research Publications2, no. 9 (2012).
[6] A. Tazaree, A.-M. Eftekhari-Moghadam, and S. Sajjadi-Ghaem-Maghami, "A semantic image classifier based on hierarchical fuzzy association rule mining" Multimedia tools and applications, vol. 69, pp. 921-949, 2014
[7] L. Wenyin, S. T. Dumais, Y. Sun, H. Zhang, M. Czerwinski, and B. A. Field, "Semi-Automatic Image Annotation" in Interact, 2001, pp. 326-333.
[8] M. Grubinger, “Analysis and evaluation of visual information system performance” Victoria University, Ph.D thesis, 2007
[9] Y.Verma, and C.V Jawahar, “Image annotation by propagating labels from semantic neighbourhoods. ” International Journal of Computer Vision, 121(1), pp.126-148, 2017.
[10] T.Uricchio, L.Ballan, L.Seidenari and A.Del Bimbo, “Automatic image annotation via label transfer in the semantic space”. Pattern Recognition, 71, pp.144-157, 2017.
[11] W. Yao, C. O. Dumitru, O. Loffeld, and M. Datcu, "Semi-supervised hierarchical clustering for semantic SAR image annotation" IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 9, pp. 1993-2008, 2016.
[12] Jiayu, T. "Automatic Image Annotation and Object Detection." PhDthesis, University of Southampton, United Kingdom(2008).
[13] Y. Liu, D. Zhang, G. Lu, and W.-Y. Ma, "A survey of content-based image retrieval with high-level semantics" Pattern recognition, vol. 40, pp. 262-282, 2007.
[14] Alghamdi, Raniah A., Mounira Taileb, and Mohammad Ameen. "A new multimodal fusion method based on association rules mining for image retrieval." In Electrotechnical Conference (MELECON), 2014 17th IEEE Mediterranean, pp. 493-499. IEEE, 2014.
[15] C. Jin and S. W. Jin, "Automatic image annotation using feature selection based on improving quantum particle swarm optimization" Signal Processing, vol. 109, pp.172-181, 2015.
[16] V. Maihami and F. Yaghmaee, "Fuzzy Neighbor Voting for Automatic Image Annotation" Journal of Electrical and Computer Engineering Innovations, vol. 4, pp. 1-8, 2016.
[17] Abubacker, Nirase Fathima, Azreen Azman, Shyamala Doraisamy, Masrah Azrifah Azmi Murad, Mohamed Eltahir Makki Elmanna, and Rekha Saravanan. "Correlation-based feature selection for association rule mining in semantic annotation of mammographic medical images." In Asia Information Retrieval Symposium, pp. 482-493. Springer, Cham, 2014.
[18] Z. Li, L. Li, K. Yan, and C. Zhang, "Automatic image annotation using fuzzy association rules and decision tree" Multimedia Systems, pp. 1-12, 2016.
[19] Popescu, Andreea, Bogdan Popescu, Marius Brezovan, and Eugen Ganea. "Image semantic annotation using fuzzy decision trees." In Computer Science and Information Systems (FedCSIS), 2013 Federated Conference on, pp. 597-601. IEEE, 2013.
[20] P. Duygulu, K. Barnard, J. F. de Freitas, and D. A. Forsyth, "Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary" in European conference on computer vision, 2002, pp. 97-112.
[21] A. Hanbury, "A survey of methods for image annotation" Journal of Visual Languages & Computing, vol. 19, pp. 617-627, 2008.
[22] A. Makadia, V. Pavlovic, and S. Kumar, "Baselines for image annotation" International Journal of Computer Vision, vol. 90, pp. 88-105, 2010.
[23] J. Ashley, R. Barber, M. Flickner, J. Hafner, D. Lee, W. Niblack, et al., "Automatic and semi-automatic methods for image annotation and retrieval in QBIC" 1995.
[24] S. Dasiopoulou, C. Doulaverakis, V. Mezaris, I. Kompatslaris, and M. Strintzis, "An ontology-based framework for semantic image analysis and retrieval" Semantic-based visual information retrieval, pp. 208-228, 2007.
[25] Z. Hua, X.-J. Wang, Q. Liu, and H. Lu, "Semantic knowledge extraction and annotation for web images" in Proceedings of the 13th annual ACM international conference on Multimedia, 2005, pp. 467-470.
[26] C.-H. Hoi and M. R. Lyu, "Web image learning for searching semantic concepts in image databases" in Proceedings of the 13th international World Wide Web conference on Alternate track papers & posters, 2004, pp. 406-407.
[27] C.-F. Tsai and C. Hung, "Automatically annotating images with keywords: A review of image annotation systems" Recent Patents on Computer Science, vol. 1, pp. 55-68, 2008.
[28] Z. Li, Z. Shi, W. Zhao, Z. Li, and Z. Tang, "Learning semantic concepts from image database with hybrid generative/discriminative approach" Engineering Applications of Artificial Intelligence, vol. 26, pp. 2143-2152, 2013.
[29] S. Manochitra, E. Janice, and S. Suganya, "Hybrid based Semantic Image Annotation using SVM and DT" International Journal of Computer Applications, vol. 65, 2013.
[30] Q.Cheng, Q.Zhang, P.Fu, C.Tu. and S.Li, “A survey and analysis on automatic image annotation.” Pattern Recognition, 79, pp.242-259, 2018.
[31] T. Sumathi and M. Hemalatha, "An innovative hybrid hierarchical model for automatic image annotation" Global Trends in Information Systems and Software Applications, pp. 718-726, 2012.
[32] X. Ke, S. Li, and D. Cao, "A two-level model for automatic image annotation" Multimedia Tools and Applications, vol. 61, pp. 195-212, 2012.
[33] M. P. Patil and S. R. Kolhe, "Automatic Image Annotation Using Decision Trees and Rough Sets," IJCSA, vol. 11, pp. 38-49, 2014.
[34] J. Jeon,V. Lavrenko and R. Manmatha,"Automatic image annotation and retrieval using cross-media relevance models"in Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, 2003, pp.119-126.
[35] V. Lavrenko, R. Manmatha, and J. Jeon, "A model for learning the semantics of pictures" in Advances in neural information processing systems, 2004, pp. 553-560
[36] S. Feng, R. Manmatha, and V. Lavrenko, "Multiple bernoulli relevance models for image and video annotation" in Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, 2004, pp. II-II.
[37] L.-y. RU, S.-p. MA, and J. LU, "Boosting-based Automatic Linguistic Indexing of Pictures" Journal of Image and Graphics, vol. 4, p. 006, 2006.
[38] G. Carneiro, A. B. Chan, P. J. Moreno, and N. Vasconcelos, "Supervised learning of semantic classes for image annotation and retrieval" IEEE transactions on pattern analysis and machine intelligence, vol. 29, pp. 394-410, 2007.
[39] F. Monay and D. Gatica-Perez, "Modeling semantic aspects for cross-media image indexing" IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, 2007.
[40] Nasierding, Gulisong, Grigorios Tsoumakas, and Abbas Z. Kouzani. "Clustering based multi-label classification for image annotation and retrieval" In Systems, Man and Cybernetics, 2009. SMC 2009. IEEE International Conference on, pp. 4514-4519. IEEE, 2009.
[41] J. Verbeek, M. Guillaumin, T. Mensink, and C. Schmid, "Image annotation with tagprop on the mirflickr set" in Proceedings of the international conference on Multimedia information retrieval, 2010, pp. 537-546.