Empirical Evaluation of Deep Learning Approaches for Landmark Detection in Fish Bioimages

[en] In this paper we perform an empirical evaluation of variants of deep learning methods to automatically localize anatomical landmarks in bioimages of fishes acquired using different imaging modalities (microscopy and radiography). We compare two methodologies namely heatmap based regression and multivariate direct regression, and evaluate them in combination with several Convolutional Neural Network (CNN) architectures. Heatmap based regression approaches employ Gaussian or Exponential heatmap generation functions combined with CNNs to output the heatmaps corresponding to landmark locations whereas direct regression approaches output directly the (x, y) coordinates corresponding to landmark locations. In our experiments, we use two microscopy datasets of Zebrafish and Medaka fish and one radiography dataset of gilthead Seabream. On our three datasets, the heatmap approach with Exponential function and U-Net architecture performs better. Datasets and open-source code for training and prediction are made available to ease future landmark detection research and bioimaging applications.

Disciplines :

Computer science
Human health sciences: Multidisciplinary, general & others

Author, co-author :

Kumar, Navdeep ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science

Claudia Di Biagio; University of Rome Tor Vergata > Biology

Zachary Dellacqua; University of Rome Tor Vergata > Biology

Raman, Ratish ; Université de Liège - ULiège > GIGA ; Université de Liège - ULiège > Département des sciences de la vie ; Université de Liège - ULiège > GIGA > GIGA I3 - Laboratory for Organogenesis and Regeneration ; Université de Liège - ULiège > Département des sciences de la vie > GIGA-R : Biologie et génétique moléculaire

Arianna Martini; CREA - Council for Agricultural Research and Economics Research Centre for Animal Production and Aquaculture

Clara Boglione; University of Rome Tor Vergata > Biology

Muller, Marc ; Université de Liège - ULiège > GIGA > GIGA I3 - Laboratory for Organogenesis and Regeneration

Geurts, Pierre ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science

Marée, Raphaël ; Université de Liège - ULiège > Montefiore Institute of Electrical Engineering and Computer Science

Language :

English

Title :

Empirical Evaluation of Deep Learning Approaches for Landmark Detection in Fish Bioimages

Publication date :

14 February 2023

Event name :

Bio Image Computing Workshop in European Conference on Computer Vision 2022

Event place :

Tel Aviv, Israel

Event date :

23-27 October

Audience :

International

Main work title :

Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science

Publisher :

Springer, Cham, Unknown/unspecified

ISBN/EAN :

978-3-031-25069-9

Peer reviewed :

Peer reviewed

Development Goals :

2. Zero hunger
3. Good health and well-being
9. Industry, innovation and infrastructure

European Projects :

H2020 - 766347 - BioMedaqu - Aquaculture meets Biomedicine: Innovation in Skeletal Health research.

Funders :

EC - European Commission
EU - European Union

Data Set :

Seabream Radiograph
Zebrafish Microscopy
Medaka Microscopy

Available on ORBi :

since 31 August 2022

Statistics

Number of views

234 (38 by ULiège)

Number of downloads

171 (18 by ULiège)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Abadi, M., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/, software available from tensorflow.org
Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: new benchmark and state of the art analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3686–3693 (2014)
Aubert, B., Vazquez, C., Cresson, T., Parent, S., De Guise, J.: Automatic spine and pelvis detection in frontal x-rays using deep neural networks for patch displacement learning. In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 1426–1429. IEEE (2016)
Bookstein, F.L.: Combining the tools of geometric morphometrics. In: Marcus, L.F., Corti, M., Loy, A., Naylor, G.J.P., Slice, D.E. (eds) Advances in Morphometrics. NATO ASI Series, vol 284, pp. 131–151. Springer, Boston (1996). https://doi. org/10.1007/978-1-4757-9083-2 12(1996)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. Ieee (2009)
Edwards, C.A., Goyal, A., Rusheen, A.E., Kouzani, A.Z., Lee, K.H.: Deepnavnet: automated landmark localization for neuronavigation. Front. Neurosci. 15, 730 (2021)
Fragkoulis, S., Printzi, A., Geladakis, G., Katribouzas, N., Koumoundouros, G.: Recovery of haemal lordosis in gilthead seabream (sparus aurata l.). Sci. Rep. 9(1), 1–11 (2019)
Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.: MS-Celeb-1M: a dataset and benchmark for large-scale face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 87–102. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9 6
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Huang, W., Yang, C., Hou, T.: Spine landmark localization with combining of heatmap regression and direct coordinate regression. arXiv preprint arXiv:2007.05355 (2020)
Ibragimov, B., Vrtovec, T.: Landmark-based statistical shape representations. In: Statistical Shape and Deformation Analysis, pp. 89–113. Elsevier (2017)
Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., Schiele, B.: DeeperCut: a deeper, stronger, and faster multi-person pose estimation model. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 34–50. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4 3
Jarque, S., Rubio-Brotons, M., Ibarra, J., Ordoñez, V., Dyballa, S., Miñana, R., Terriente, J.: Morphometric analysis of developing zebrafish embryos allows predicting teratogenicity modes of action in higher vertebrates. Reprod. Toxicol. 96, 337–348 (2020)
Khabarlak, K., Koriashkina, L.: Fast facial landmark detection and applications: A survey. arXiv preprint arXiv:2101.10808 (2021)
Lai, H., Xiao, S., Pan, Y., Cui, Z., Feng, J., Xu, C., Yin, J., Yan, S.: Deep recurrent regression for facial landmark detection. IEEE Trans. Circuits Syst. Video Technol. 28(5), 1144–1157 (2016)
Lee, H., Park, M., Kim, J.: Cephalometric landmark detection in dental x-ray images using convolutional neural networks. In: Medical Imaging 2017: Computer-Aided Diagnosis, vol. 10134, p. 101341W. International Society for Optics and Photonics (2017)
Lindner, C., Cootes, T.F.: Fully automatic cephalometric evaluation using random forest regression-voting. In: IEEE International Symposium on Biomedical Imaging (ISBI) 2015-Grand Challenges in Dental X-ray Image Analysis-Automated Detection and Analysis for Diagnosis in Cephalometric X-ray Image (2015)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)
Loy, B.A., Boglione, C., Cataudella, S.: Geometric morphometrics and morpho-anatomy: a combined tool in the study of sea bream (sparus aurata, sparidae) shape. J. Appl. Ichthyol. 15(3), 104–110 (1999)
Mahpod, S., Das, R., Maiorana, E., Keller, Y., Campisi, P.: Facial landmarks localization using cascaded neural networks. Comput. Vis. Image Underst. 205, 103171 (2021)
Marée, R., Rollus, L., Stévens, B., Hoyoux, R., Louppe, G., Vandaele, R., Begon, J.M., Kainz, P., Geurts, P., Wehenkel, L.: Collaborative analysis of multi-gigapixel imaging data using cytomine. Bioinformatics 32(9), 1395–1401 (2016)
Mathis, A., Mamidanna, P., Cury, K.M., Abe, T., Murthy, V.N., Mathis, M.W., Bethge, M.: Deeplabcut: markerless pose estimation of user-defined body parts with deep learning. Nat. Neurosci. 21(9), 1281–1289 (2018)
Mohseni, H., Kasaei, S.: Automatic localization of cephalometric landmarks. In: 2007 IEEE International Symposium on Signal Processing and Information Technology, pp. 396–401. IEEE (2007)
Park, J.H., Hwang, H.W., Moon, J.H., Yu, Y., Kim, H., Her, S.B., Srinivasan, G., Aljanabi, M.N.A., Donatelli, R.E., Lee, S.J.: Automated identification of cephalometric landmarks: Part 1-comparisons between the latest deep-learning methods yolov3 and ssd. Angle Orthod. 89(6), 903–909 (2019)
Payer, C., Štern, D., Bischof, H., Urschler, M.: Regressing heatmaps for multiple landmark localization using CNNs. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 230–238. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8 27
Payer, C., Štern, D., Bischof, H., Urschler, M.: Integrating spatial configuration into heatmap regression based CNNs for landmark localization. Med. Image Anal. 54, 207–219 (2019)
Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2015)
Riegler, G., Urschler, M., Ruther, M., Bischof, H., Stern, D.: Anatomical landmark detection in medical applications driven by synthetic data. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 12–16 (2015)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, U-net: Convolutional networks for biomedical image segmentation (2015). https://doi.org/10.1007/978-3-319-24574-4 28
Samet, N., Akbas, E.: Hprnet: hierarchical point regression for whole-body human pose estimation. Image Vis. Comput. 115, 104285 (2021)
Song, Y., Qiao, X., Iwamoto, Y., Chen, Y.W.: Automatic cephalometric landmark detection on x-ray images using a deep-learning method. Appl. Sci. 10(7), 2547 (2020)
Stern, O., Marée, R., Aceto, J., Jeanray, N., Muller, M., Wehenkel, L., Geurts, P.: Automatic localization of interest points in zebrafish images with tree-based methods. In: Loog, M., Wessels, L., Reinders, M.J.T., de Ridder, D. (eds.) PRIB 2011. LNCS, vol. 7036, pp. 179–190. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24855-9 16
Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693–5703 (2019)
Torosdagli, N., Liberton, D.K., Verma, P., Sincan, M., Lee, J.S., Bagci, U.: Deep geodesic learning for segmentation and anatomical landmarking. IEEE Trans. Med. Imaging 38(4), 919–931 (2018)
Vandaele, R., Aceto, J., Muller, M., Peronnet, F., Debat, V., Wang, C.W., Huang, C.T., Jodogne, S., Martinive, P., Geurts, P., et al.: Landmark detection in 2d bioimages for geometric morphometrics: a multi-resolution tree-based approach. Sci. Rep. 8(1), 1–13 (2018)
Verhaegen, Y., Adriaens, D., De Wolf, T., Dhert, P., Sorgeloos, P.: Deformities in larval gilthead sea bream (sparus aurata): a qualitative and quantitative analysis using geometric morphometrics. Aquaculture 268(1–4), 156–168 (2007)
Weinhardt, V., Shkarin, R., Wernet, T., Wittbrodt, J., Baumbach, T., Loosli, F.: Quantitative morphometric analysis of adult teleost fish by x-ray computed tomography. Sci. Rep. 8(1), 1–12 (2018)
Xu, Z., Li, B., Yuan, Y., Geng, M.: Anchorface: an anchor-based facial landmark detector across large poses. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 3092–3100 (2021)
Yeh, Y.C., Weng, C.H., Huang, Y.J., Fu, C.J., Tsai, T.T., Yeh, C.Y.: Deep learning approach for automatic landmark detection and alignment analysis in whole-spine lateral radiographs. Sci. Rep. 11(1), 1–15 (2021)

Name	Provider / Domaine	Expiration	Description
JSESSIONID	Oracle Corporation www.uliege.be	Session	General purpose platform session cookie, used by sites written in JSP. Usually used to maintain an anonymous user session by the server.
CookieScriptConsent	CookieScript .uliege.be	1 year	This cookie is used by Cookie-Script.com service to remember visitor cookie consent preferences. It is necessary for Cookie-Script.com cookie banner to work properly.

Name	Provider / Domaine	Expiration	Description
_pk_id	InnoCraft Ltd .uliege.be	1 year	Used to store a few details about the user such as the unique visitor ID
_pk_ses	InnoCraft Ltd .uliege.be	30 minutes	Short lived cookies used to temporarily store data for the visit
_pk_ref	InnoCraft Ltd .uliege.be	6 months	Used to store the attribution information, the referrer initially used to visit the website