Objective:

2

To develop a multimodal deep-learning model for classifying Chinese medicine constitution

i.e.

the balanced and unbalanced constitutions

based on inspection of tongue and face images

pulse waves from palpation

and health information from a total of 540 subjects.

Methods:

2

This study data consisted of tongue and face images

pulse waves obtained by palpation

and health information

including personal information

life habits

medical history

and current symptoms

from 540 subjects (202 males and 338 females). Convolutional neural networks

recurrent neural networks

and fully connected neural networks were used to extract deep features from the data. Feature fusion and decision fusion models were constructed for the multimodal data.

Results:

2

The optimal models for tongue and face images

pulse waves and health information were ResNet18

Gate Recurrent Unit

and entity embedding

respectively. Feature fusion was superior to decision fusion. The multimodal analysis revealed that multimodal data compensated for the loss of information from a single mode

resulting in improved classification performance.

Conclusions:

2

Multimodal data fusion can supplement single model information and improve classification performance. Our research underscores the effectiveness of multimodal deep learning technology to identify body constitution for modernizing and improving the intelligent application of Chinese medicine.