We have developed a Japanese extension of a multimodal LLM benchmark. After carefully examining the existing benchmark, MMMU, we translated 24 culturally-neutral subjects into Japanese and created 4 new culturally-dependent subjects, resulting in a total of 1,320 questions (1,118 images). On our website, we also conducted benchmarking of major multimodal LLMs.
https://mmmu-japanese-benchmark.github.io/JMMMU/
https://mmmu-japanese-benchmark.github.io/JMMMU/