VisionTeamWorkshopsPublications

JST DFG ANR Trilateral AI Research Learning Cyclotron

Vision

The aim of this project is to augment and amplify human intelligence with artificial intelligence. In particular, we propose and develop Learning Cyclotron (LeCycl), a framework that monitors and actuates learning processes. The figure above shows an overview of LeCycl. In this framework, learners acquire knowledge with the following three steps: Perceive (e.g., reading a textbook), Master (e.g., solving an exercise), and Transfer (e.g., explaining to other learners). Behaviors of learners on each step are measured by sensors. From the sensor signals, we estimate cognitive and affective states, including attention, interest, concentration, mental workload, and self-confidence. The estimated states will be utilized to select how to nudge learners. As the same as a cyclotron accelerates charged particles by a magnetic field, LeCycl circulates knowledge between learners.

Team

🇯🇵 Japan

  • Koichi Kise, Osaka Metropolitan University
  • Motoi Iwata, Osaka Metropolitan University
  • Andrew Vargo, Osaka Metropolitan University

🇩🇪 Germany

  • Andreas Dengel, German Research Center for Artificial Intelligence (DFKI)
  • Jochen Kuhn, Ludwig-Maximilians-Universität Munich
  • Shoya Ishimaru, Osaka Metropolitan University
  • Nicolas Großmann, German Research Center for Artificial Intelligence (DFKI)
  • Jayasankar Santhosh, University of Kaiserslautern-Landau
  • Ko Watanabe, University of Kaiserslautern-Landau

🇫🇷 France

  • Laurence Devillers, Centre National de la Recherche Scientifique (CNRS)
  • Ioana Vasilescu, Centre National de la Recherche Scientifique (CNRS)
  • Olivier Augereau, European Center for Virtual Reality (CERV)
  • Pajak Serge, Paris-Saclay University
  • Fabrice Le Guel, Paris-Saclay University

Workshops

We organized the following workshops to discuss the future of LeCycl.

Publications

Journal Papers

  1. 🇩🇪 Jayasankar Santhosh, David Dzsotjan and Shoya Ishimaru. “Multimodal Assessment of Interest Levels in Reading: Integrating Eye-Tracking and Physiological Sensing”. IEEE Access, pp. 93994–94008, 2023.

  2. 🇩🇪 Ko Watanabe, Tanuja Sathyanarayana, Andreas Dengel and Shoya Ishimaru. “EnGauge: Engagement Gauge of Meeting Participants Estimated by Facial Expression and Deep Neural Network”. IEEE Access, pp. 52886–52898, 2023.

  3. 🇯🇵 Hannah R. Nolasco, Andrew Vargo, Niklas Bohley, Christian Brinkhaus and Koichi Kise. “Examining Participant Adherence with Wearables in an In-the-Wild Setting”. Sensors 23 (14), p. 6479, 2023.

  4. 🇯🇵 Andrew Vargo, Benjamin Tag, Chris Blakely and Koichi Kise. “Poverty Traps in Online Knowledge-Based Peer-Production Communities”. Informatics (10), vol. 3 p. 61, 2023.

  5. 🇯🇵 Junki Oshiba, Motoi Iwata and Koichi Kise. “Face Image Generation of Anime Characters Using an Advanced First Order Motion Model with Facial Landmarks”. IEICE Transactions on Information and Systems, E106-D, vol. 9, pp. 22–30, 2022.

  6. 🇯🇵 Md. Rabiul Islam, Andrew W. Vargo, Motoi Iwata, Masakazu Iwamura and Koichi Kise. “Exploring Sensor Modalities to Capture User Behaviors for Reading Detection”. IEICE Transactions on Information and Systems, E105-D, vol. 9, pp. 1629–1633, 2022.

  7. 🇩🇪 Chenhao Chen, Yutaka Arakawa, Ko Watanabe and Shoya Ishimaru. “Quantitative Evaluation System for Online Meetings Based on Multimodal Microbehavior Analysis”. Sensors and Materials 34 (8), pp. 3017–3027, 2022.

  8. 🇯🇵 🇩🇪 Andrew Vargo, Shoya Ishimaru, Md. Rabiul Islam, Benjamin Tag and Koichi Kise. “Obtaining Labels for In-the-Wild Studies: Using Visual Cues and Recall”. IEEE Pervasive Computing 21 (1), pp. 9–17, 2022.

  9. 🇩🇪 Ko Watanabe, Yusuke Soneda, Yuki Matsuda, Yugo Nakamura, Yutaka Arakawa, Andreas Dengel and Shoya Ishimaru. “DisCaaS: Micro Behavior Analysis on Discussion by Camera as a Sensor”. Sensors 21 (17), p. 5719, 2021.

Book Chapters

  1. 🇩🇪 🇫🇷 Andreas Dengel, Laurence Devillers and Laura Maria Schaal. “Augmented Human and Human-Machine Co-evolution: Efficiency and Ethics”. Reflections on Artificial Intelligence for Humanity, pp. 203–227, 2021.

International Conference Papers

  1. 🇯🇵 🇩🇪 🇫🇷 Andrew Vargo, Benjamin Tag, Mathilde Hutin, Victoria Abou-Khalil, Shoya Ishimaru, Olivier Augereau, Tilman Dingler, Motoi Iwata, Koichi Kise, Laurence Devillers and Andreas Dengel. “Intelligence Augmentation: Future Directions and Ethical Implications in HCI”. Proc. INTERACT 2023, pp. 644–649, 2023.

  2. 🇯🇵 Peter Neigel, Andrew Vargo, Yusuke Komatsu, Chris Blakely and Koichi Kise. “Exploring Users’ Ability to Choose a Proper Fit in Smart-Rings: A Year-Long “In the Wild” Study”. Proc. INTERACT 2023, pp. 486–490, 2023.

  3. 🇯🇵 Hannah R. Nolasco, Andrew Vargo, Yusuke Komatsu, Motoi Iwata and Koichi Kise. “Perception Versus Reality: How User Self-reflections Compare to Actual Data”. Proc. INTERACT 2023, pp. 665–674, 2023.

  4. 🇩🇪 Ankur Bhatt, Ko Watanabe, Andreas Dengel and Shoya Ishimaru. “Appearance-Based Gaze Estimation with Deep Neural Networks: From Data Collection to Evaluation”. Proc. ABC 2023, 2023.

  5. 🇩🇪 Rashmi Alur Ramachandra, Jayasankar Santhosh, Andreas Dengel and Shoya Ishimaru. “Enhancing Stress Detection for Students: Exploring the Impact of Fine-Tuning and User-Specific Data Calibration in Deep Learning”. Proc. ABC 2023, 2023.

  6. 🇩🇪 Ko Watanabe, Andreas Dengel and Shoya Ishimaru. “Accelerating Knowledge Transfer by Sensing and Actuating Social-Cognitive States”. Proc. UbiComp/ISWC 2023 Adjunct, pp. 258–262, 2023.

  7. 🇯🇵 🇩🇪 Ko Watanabe, Seiya Tanaka, Andrew Vargo, Koichi Kise and Shoya Ishimaru. “TrackThink Camera: A Tool for Tracking Facial and Body Information while Web Browsing”. Proc. AIED 2023 Workshop, 2023.

  8. 🇯🇵 🇩🇪 Ko Watanabe, Hacer Kübra Kücük, Sarah Gonzales, Andrew Vargo, Koichi Kise and Shoya Ishimaru. “Combining the Knowledge of Experienced Programmers to Extract Useful Web Resources for Solving Programming Tasks”. Proc. Asian CHI 2023, pp. 22–27, 2023.

  9. 🇫🇷 Sofiya Kobylyanskaya. “Speech and Eye Tracking Features for L2 Acquisition: A Multimodal Experiment”. AIED 2022, pp. 47–52, 2022.

  10. 🇫🇷 Mathilde Hutin, Sofiya Kobylyanskaya and Laurence Devillers. “Nudges in technology-mediated knowledge transfer : Two experimental designs”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 267–273, 2022.

  11. 🇯🇵 Kanta Yamaoka, Andrew Vargo, Motoi Iwata and Koichi Kise. “Towards Guided-Breath as an Intervention Within Computer-Assisted Learning”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 136–137, 2022.

  12. 🇯🇵 Andrew Vargo, Kentaro Ohara, Takumi Nakai and Koichi Kise. “Just the Right Amount of Exercise: Aerobic Exercise and Memory”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 126–127, 2022.

  13. 🇯🇵 Andrew Vargo, Masayuki Fujimoto and Koichi Kise. “Sleep Quality Tracking and Alignment with a Simple In-the-Wild Task”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 128–129, 2022.

  14. 🇯🇵 Hannah R. Nolasco, Andrew Vargo and Koichi Kise. “Mixed-Methods Self-Tracking: Qualitative Styles of Quantifying the Self”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 287–291, 2022.

  15. 🇯🇵 Takumi Nakai, Andrew Vargo and Koichi Kise. “Attaching Microlearning to Existing Habits”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 284–286, 2022.

  16. 🇯🇵 🇩🇪 Kanta Yamaoka, Ko Watanabe, Koichi Kise, Andreas Dengel and Shoya Ishimaru. “Experience is the Best Teacher: Personalized Vocabulary Building Within the Context of Instagram Posts and Sentences from GPT-3”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 313–316, 2022.

  17. 🇯🇵 🇩🇪 Haruki Suzawa, Ko Watanabe, Masakazu Iwamura, Koichi Kise, Andreas Dengel and Shoya Ishimaru. “Supporting Smooth Interruption in a Video Conference by Dynamically Changing Background Music Depending on the Amount of Utterance”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 299–302, 2022.

  18. 🇯🇵 🇩🇪 🇫🇷 Andrew Vargo, Motoi Iwata, Mathilde Hutin, Sofiya Kobylyanskaya, Ioana Vasilescu, Olivier Augereau, Ko Watanabe, Shoya Ishimaru, Benjamin Tag, Tilman Dingler, Koichi Kise, Laurence Devillers and Andreas Dengel. “Learning Cyclotron: An Ecosystem of Knowledge Circulation”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 308–312, 2022.

  19. 🇩🇪 Akshay Palimar Pai, Jayasankar Santhosh and Shoya Ishimaru. “Real-Time Feedback on Reader's Engagement and Emotion Estimated by Eye-Tracking and Physiological Sensing”. Proc. UbiComp/ISWC 2022 Adjunct, pp. 97–98, 2022.

  20. 🇯🇵 Md. Rabiul Islam, Andrew W. Vargo, Motoi Iwata, Masakazu Iwamura and Koichi Kise. “Evaluating Contrastive Learning for Fine-grained Reading Detection”. Proc. IIAI-AAI 2022, pp. 430–435, 2022.

  21. 🇩🇪 Ko Watanabe, Yuki Matsuda, Yugo Nakamura, Yutaka Arakawa and Shoya Ishimaru. “How do Programmers Use the Internet? Discovering Domain Knowledge from Browsing and Coding Behaviors”. Proc. IEEE SmartData, pp. 605–610, 2022.

  22. 🇯🇵 Riku Higashimura, Andrew W. Vargo, Motoi Iwata and Koichi Kise. “Helping Mobile Learners Know Unknown Words through their Reading Behavior”. CHI 2022 Extended Abstracts, no. 249, 5 pages, 2022.

  23. 🇫🇷 Mohamed El Baha, Olivier Augereau, Sofiya Kobylyanskaya, Ioana Vasilescu and Laurence Devillers “Eye Got It: A System for Automatic Calculation of the Eye-Voice Span”. DAS 2022, pp. 713–725, 2022.

  24. 🇩🇪 Jihed Makhlouf, Yutaka Arakawa and Ko Watanabe. “A Privacy-aware Browser Extension to Track User Search Behavior for Programming Course Supplement”. Proc. MUSICAL 2021, pp. 783–796, 2021.

  25. 🇯🇵 Seiya Tanaka, Andrew W. Vargo, Motoi Iwata and Koichi Kise. “Confidence Estimation via Wrist Movement”. Proc. UbiComp/ISWC 2021 Adjunct, pp. 467–471, 2021

  26. 🇩🇪 David Dzsotjan, Kim Ludwig-Petsch, Sergey Mukhametov, Shoya Ishimaru, Stefan Küchemann and Jochen Kuhn. “The Predictive Power of Eye-Tracking Data in an Interactive AR Learning Environment”. Proc. UbiComp/ISWC 2021 Adjunct, pp. 76–77, 2021.

  27. 🇩🇪 Pramod Vadiraja, Jayasankar Santhosh, Hanane Moulay, Andreas Dengel and Shoya Ishimaru. “Effects of Counting Seconds in the Mind while Reading”. Proc. UbiComp/ISWC 2021 Adjunct, pp. 486–490, 2021.

  28. 🇩🇪 Prerna Garg, Jayasankar Santhosh, Andreas Dengel and Shoya Ishimaru. “Stress Detection by Machine Learning and Wearable Sensors”. Proc. IUI 2021, pp. 237–240, 2021.

  29. 🇩🇪 Pramod Vadiraja, Andreas Dengel and Shoya Ishimaru. “Text Summary Augmentation for Intelligent Reading Assistant”. Proc. AHs 2021, pp. 319–321, 2021.

  30. 🇯🇵 Jin Kato, Motoi Iwata and Koichi Kise. “Manga Vocabulometer, A new support system for extensive reading with Japanese mangatranslated into English”. Proc. ICPR 2020 (MANPU 2020), pp. 223–235, 2020.

Domestic Conference Papers

  1. 🇯🇵 髙池 太郎, 岩田 基 and 黄瀬 浩一. “音読音声と視線情報を用いた英語漫画中の未知語を含んだ吹き出し推定”. 第22回情報科学技術フォーラム講演論文集, 2023.

  2. 🇯🇵 内田 時央, Andrew Vargo and 黄瀬 浩一. “錯乱肢の生成による英語多肢選択問題の適応的難易度調整”. 第22回情報科学技術フォーラム講演論文集, 2023.

  3. 🇫🇷 Sofiya Kobylyanskaya, Ioana Vasilescu, Olivier Augereau and Laurence Devillers. “Vers la compréhension des difficultés de lecture en l2 à travers des paramètres acoustiques et de mouvement des yeux”. In : Proc. EIAH2023, 2023.

  4. 🇯🇵 浦上 太一 and 黄瀬 浩一. “バーチャルペット飼育型単語学習システムとその評価”. 第202回情報処理学会ヒューマンコンピュータインタラクション研究発表会資料, 2022.

  5. 🇯🇵 中島 悠太, 岩田 基 and 黄瀬 浩一. “text-to-imageモデルによる生成画像を用いた英単語学習の提案と画像で表現可能な英単語の調査”. 情報処理学会研究報告教育学習支援情報システム (CLE), 2022.

  6. 🇯🇵 岩下 雄一郎, 岩田 基, Andrew W. Vargo and 黄瀬 浩一. “英単語学習を長期的に継続するための適応型学習システムの開発”, 2022.

  7. 🇯🇵 大原健太郎, Andrew W. Vargo and 黄瀬浩一. “個人に適した有酸素運動の単語記憶課題への効果”. 情報処理学会第85回全国大会講演論文集, 2022.

  8. 🇯🇵 浦上 太一 and 黄瀬 浩一. “バーチャルペット飼育型単語学習システムの提案”. 情報処理学会第85回全国大会講演論文集, 2022.

  9. 🇯🇵 東村 理功, Andrew W. Vargo, 岩田 基 and 黄瀬 浩一. “モバイル学習者の読書行動に基づく未知単語推定”. 第21回情報科学技術フォーラム (FIT2022) 講演論文集, 2022.

  10. 🇯🇵 田中 勢也, Andrew W. Vargo and 黄瀬 浩一. “腕の動きとWebカメラを用いた英語問題の確信度推定”. 2022年電子情報通信学会総合大会 講演論文集, 2022.

  11. 🇯🇵 森瀧 瑞希, 湯浅 成章, Andrew W. Vargo and 黄瀬 浩一. “英語空所補充問題の自動生成”. 2022年電子情報通信学会総合大会 講演論文集, 2022.

  12. 🇯🇵 湯浅 成章, Andrew W. Vargo and 黄瀬 浩一. “個人に適した英語多肢選択問題の自動生成方法の提案”. 情報処理学会研究報告教育学習支援情報システム (CLE), 2022-CLE-36, vol. 5, pp. 1–8.

  13. 🇯🇵 中井 拓海, Andrew W. Vargo and 黄瀬 浩一. “習慣添付型マイクロラーニングの提案”. 情報処理学会研究報告教育学習支援情報システム (CLE), 2022-CLE-36, vol. 7, pp. 9–17, 2021

© 2023 LeCycl