LLM-Based Empathy Scoring for Psychological Counseling Dialogues Using Multi-Dimensional Client Feedback

Automatic assessment of counseling-dialogue quality requires models that connect long, role-structured interactions with the experience reported by clients after a session. This study presents a multi-dimensional empathy-scoring evaluation on KokoroChat, a Japanese psychological-counseling dialogue corpus released with 20 client-feedback dimensions on a 0-5 scale. The task is formulated as multi-output regression: given a complete counselor-client dialogue, predict the full feedback profile covering feeling heard, respect, insight, hope, conversational flow, acceptance, affirmation, questioning, summarization, clarification, goal identification, actionable suggestions, and encouragement. The evaluation uses a chronological split, with 2020-2022 dialogues for training, 2023 dialogues for validation, and 2024 dialogues for testing. Ten systems are compared: a global-mean baseline, structure-and-topic ridge regression, four hashed Japanese character n-gram ridge models, two compact self-attention encoders, a rubric-calibrated judge, and a linear ensemble. The rubric-calibrated judge obtains the lowest test MAE of 0.845 and RMSE of 1.056, with mean Pearson correlation of 0.449 and a bootstrap 95% confidence interval of 0.823-0.871 for overall MAE. Client-side text alone is numerically almost tied at MAE 0.846 and yields the highest mean correlation, indicating that client language contains strong evidence about perceived session quality. Topic-level analysis shows lower error for spouse-family and workplace sessions and higher error for small or heterogeneous groups. The results support detailed, reproducible evaluation of counseling quality through feedback profiles rather than a single empathy score.